Learning a craft

During my attendance at Dev8D last week, I used the time to start pulling together some ideas I’ve been knocking around for a few months now. Most of the projects I’ve helped set up or led over the last three years have involved working closely with developers working in higher education and ever since JISCPress, we’ve been growing a culture where student developers, either on bursaries, employed part-time or full-time recent graduates (i.e. their first ‘proper job’) are core contributors to the project. My point here is that through these projects I have been able to observe how young hackers learn their craft and make the transition from a formal education in Computer Science to learning on the job (i.e. apprenticeship).

DSC03598

DevXS http://devxs.org

I use the word ‘craft’ deliberately. I think our work is much more closely aligned with craftsmanship than with engineering but I was unsure how to articulate this until I read Pete McBreen’s Software Craftsmanship. McBreen argues that a focus on craftsmanship is to return to the roots of software development:

Good software developers have always understood that programming is a craft skill. Regardless of the amount of arcane and detailed technical knowledge that a person has, in the end, application development comes down to feel and experience.

McBreen distinguished software craftsmanship from software engineering and computer science, not as their opposites but as a different tradition “that happily coexists with and benefits from science and engineering.” He compares the software craftsman to the blacksmith, both of whom transcend science and engineering and benefit from improvements in their tools, materials and understanding. For McBreen, GNU/Linux is an example of software craftsmanship that thrives due to the dedication, skill and craft of the people who contribute to the development of the operating system.

‘Software engineering’ was born out of a so-called ‘software crisis’, identified at a NATO conference in 1968. It was determined that the way out of that crisis was to apply an engineering approach to large scale state of the art defence projects. Since that time, argues McBreen, “the needs of the US Department of Defence have dominated the conversation about software engineering.”

software engineering is not all that relevant to many projects. Software engineering was created to solve the problems of really large groups working on multiyear projects. Most modern software development, however, is done in relatively small teams.

Not surprisingly, the practise of software craftsmanship is often allied with agile development, which emphasises the human aspects of software development, the creative and variable processes involved in creating software that meets human needs. Both share a concern with quality and both encourage continual reflective critique of one’s work in order to improve the software and oneself. In his book on Crystal Clear, for example, Alistair Cockburn identifies ‘reflective improvement’ and ‘osmotic communication’ as two of the three minimum requirements for a Crystal Clear project. In practice, reflective improvement refers to workshops and end-of-iteration discussion about how the work is going and how it might be improved. Osmotic communication refers to how flows of information between developers should be unobstructed by locked doors, walls and corridors. It’s about sitting developers in the same or adjacent rooms so that they absorb information about the project with as few barriers as possible. Access to information, a safe environment where people can respectfully speak their minds and close collegial working are highly valued. At times, developers will program side-by-side or in pairs at one workstation, so as to review each others’ work. He refers to this as ‘peer code peering’.

Pair programming

Both reflective improvement and osmotic communication can be enhanced by the choice of tools developers use and the feedback they provide to her. Craftsmen in all trades rely on their tools to provide feedback – a joiner’s plane, a responsive power drill, etc. Many craftsmen will make their own tools to improve their responsiveness. Cockburn notes that one of the most important tools developers can use is an automated regression test suite, which allows the team to continuously test their work and provides feedback to each developer about the quality of their code. Feedback from a Continuous Integration (CI) suite of tools can be usefully presented by ‘information radiators’, dashboards which typically provide status information of servers, the number of use cases delivered and the number of tests passed. Although, Cockburn doesn’t use the term, I think that reflective improvement and osmotic communication refer to the ‘learning environment’ that the software craftsman creates so as to improve their understanding of their work and further develop their craft.

References to the importance of learning from others and from one’s work are made throughout McBreen’s book, as well as an entire chapter at the end of the book called ‘Perpetual Learning’. There, he outlines how to create a ‘learning environment’ by building a library of books for developers to read, as well as ensuring that they take time out each week to practice or learn something new. Like Cockburn, he emphasises the importance of workshops and a series of seminars where developers discuss their work. In addition, McBreen suggests that developers are encouraged to attend and present at conferences and write papers about their work as well as take on the role of instructor where they are able to do so. Craftsmen continually undertake self-directed learning, preferring non-proprietary, open source tools that are sustained by a community and made freely available to learn from, but more importantly, software craftsmen learn from each other, with master craftsmen mentoring journeymen and apprentices.

In a more recent book on software craftsmanship, Apprenticeship Patterns, Hoover and Oshineye also devote a chapter to ‘Perpetual Learning‘,  offering further practical advice to aspiring software craftsmen. They list the following:

  • Expand your bandwidth: Read books and articles, engage with your peers via conferences, social media and mailing lists, join user groups, study from open educational materials on the web
  • Practice, practice, practice: Take the time to practice your craft without interruptions, in an environment where you can feel comfortable making mistakes. i.e. ‘deliberate practice’ They borrow the term ‘code katas’, whereby programming exercises are repeated again and again until they become ingrained in the individual.
  • Breakable toys: Budget for failure by designing and building toy systems that are similar in toolset, but not in scope to the systems you build at work. i.e. build something personal to you, that you will learn from. Develop your own CMS or game that you can afford to break while learning.
  • Use the source:  Seek out other people’s code and read it.  Start with the applications and tools you use every day.
  • Reflect as you work: Become a reflective practitioner of software development. This involves regular introspection into how you are working.
  • Record what you learn: Keep a record of your journey in a journal, personal wiki, or blog. A chronological record of the lessons you have learned can provide inspiration to those you mentor, since it makes your journey explicit, but it can also give you a vital resource to draw upon.
  • Share what you learn: Early in your apprenticeship, develop the habit of regularly sharing the lessons you have learned. i.e. keep a blog, run workshop sessions, be part of a community of learners.
  • Create feedback loops: Create mechanisms for regularly gathering more or less objective external data about your performance. i.e. automated regression tests, information radiators, exams and certifications, pair programming, and asking your peers what they think.
  • Learn how you fail: Seek to identify the ways in which you tend to fail and try to resolve those that are worth fixing.

In his book, The Craftsman, Richard Sennett chooses to focus on open source software development as a modern form of craftsmanship (“the skill of making things well”). The value of Sennett’s book is its breadth of scope. While making no reference to McBreen’s earlier book, Sennett situates open source hacking and the development of GNU/Linux within the social and economic history of craftsmanship and our relationship with technology. For Sennett, Linux is a “public craft” and open source hackers are a “community of craftsmen.” In terms of learning this public craft, he compares ancient pottery making with open source hacking and finds that only the speed between problem finding and problem solving differentiates the two.  In programming, and especially open source programming, the velocity at which we can learn can be much greater than in traditional, material crafts. Our tools and the open, distributed nature of developer communities enhance our opportunities for learning.1

In The Nature and Art of Workmanship, David Pye does not make reference to software development but does develop a very useful framework for identifying and understanding craftsmanship, which complements much of what Sennett and McBreen describe. For Pye, craftsmanship is the workmanship of risk; that is, work that is constantly at risk of error in the process of creation. A simple example of this is writing with a pen. In contrast to the workmanship of risk (craftsmanship) is the workmanship of certainty; that is, workmanship where the quality is always pre-determined and is usually found in quantity production, and always in fully automated work. An example of this would be modern printing. The workmanship of certainty is most common in modern, industrial society, but has existed in some form for hundreds, if not thousands of years. All types of workmanship exist somewhere on the axis between risk and certainty and furthermore are subject to varying degrees of regulation or freedom.

Workmanship

Workmanship

What distinguishes the degree of risk or certainty for Pye is the extent to which the workman’s tools regulate his work. Pye argues that a pure form of workmanship of risk is hardly ever seen in any trade; for centuries people have developed tools to help regulate their work in some way (e.g. jigs) and guarantee some degree of certainty in the quality of their work. Regulation of work does not necessarily lead to certainty as some tools such as a lathe can be used in combination with a free hand, producing unique objects that are nevertheless regulated in some respects such as their size but not their form.

A simple way of asking whether it is workmanship of risk or certainty is to ask, “is the result predetermined and unalterable once production begins?”

In the drawing above, I have speculated that software craftsmanship as McBreen describes it, would be called workmanship of risk by Pye, with regulation introduced by tools such as Integrated Development Environments (IDE), Continuous Integration (CI) suites and the general operating system environment the developer is working with. Although regulated, like almost all workmanship of risk, the software craftsman often produces something bespoke to the users’ requirements, iteratively working towards the desired result through the writing and refactoring of code. Agile software development recognises that the result of most software development cannot be predetermined and that projects run best when they remain responsive to the users’ feedback. Software Engineering, on the other hand, aims to eliminate as much of the risk as possible and pre-determine the outcome of the design and programming effort, which both McBreen and Pyritz both identify as a form of Taylorism.

The Institute of Electrical and Electronics Engineers defines software engineering as “the application of a systematic, disciplined, quantifiable approach to development, operation, and maintenance of software”. Pyritz questions whether the Taylorist model of scientific management is even viable in software development, quoting Humphrey, who asked ‘Why don’t they practice what they preach?’

The general practices of industrial software engineers are poor by almost any measure”. Why? “The educational system does not provide graduates with the practical skills they will later need. . . . Few software organizations are willing or able to provide the remedial training their new engineers need. Today’s software organizations have few if any role models who consistently demonstrate effective work habits and disciplines.

In learning to become software craftsmen we need role models, too. Both McBreen and Sennett emphasise the importance of apprenticeship and slowly learning by doing with others. This is what JISC’s DevCSI offers to the sector, running regular hack days, DevXS and the annual Dev8D conference. This year over 250 developers attended Dev8D to learn from each other.

Learning from each other at Dev8D

Learning from each other at Dev8D

While I was at Dev8D, I issued an informal survey asking developers a number of questions about their working environment, how they learn their craft, the tools they use and an assessment of their skills. I shall provide a summary of the responses in another post [UPDATE: here it is], but I am encouraged to do further research in determining how developers (working in tertiary education) learn their craft and how opportunities for learning might be improved.

Another related project I started while at Dev8D was Hacking the University, a simple website intended to collect short interviews with developers working in universities. It is inspired by The Setup and I hope that over time, it will provide a record of the people working in this community and add to the recognition of the work they do, how they learn and how their working environment impacts upon their work and learning.

If you are a developer working in a university, please consider contributing to Hacking the University and telling others about your approach to your craft.

  1. Sennett’s observations also help us consider software craftsmanship together with learning theories such as cognitive apprenticeship, situated learning, and constructivism. []

Jailbreaking WordPress with Web hooks

As is often the case, I struggle at first glance to see the full implications of a new development in technology, which is why I so often rely on others to kick me up the arse before I get it.1

Where I ramble about WordPress as a learning tool for the web…

I first read about web hooks while looking at WordPress, XMPP and FriendFeed’s SUP and then again when writing about PubSubHubbub. Since then, Dave Winer’s RSSCloud has come along, too, so there’s now plenty of healthy competition in the world of real time web and WordPress is, predictably, a mainstream testing ground for all of it. Before I go on to clarify my understanding of the implications of web hooks+WordPress, I should note that my main interest here is not web hooks nor specifically the real time web, which is interesting but realistically, not something I’m going to pursue with fervour. My main interest is that WordPress is an interesting and opportunistic technology platform for users, administrators and developers, alike. Whoever you are, if you want to understand how the web works and how innovations become mainstream, WordPress provides a decent space for exercising that interest. I find it increasingly irritating to explain WordPress in terms of ‘blogging’. I’ve very little interest in WordPress as a blog. I tend to treat WordPress as I did Linux, ten years ago. Learning about GNU/Linux is a fascinating, addictive and engaging way to learn about Operating Systems and the role of server technology in the world we live in. Similarly, I have found that learning about WordPress and, perhaps more significantly, the ecosystem of plugins and themes2 is instructive in learning about the technologies of the web. I encourage anyone with an interest, to sign up to a cheap shared host such as Dreamhost, and use their one-click WordPress offering to set up your playground for learning about the web. The cost of a domain name and self-hosting WordPress need not exceed $9 or £7/month.3

… and back to web hooks

Within about 15 minutes of Tony tweeting about HookPress, I had watched the video, installed the plugin and sent a realtime tweet using web hooks from WordPress.

It’s pretty easy to get to grips with and if a repository of web hook scripts develops, even the non-programmers like me could make greater use of what web hooks offer.

Web hooks are user-defined callbacks over HTTP. They’re intended to, in a sense, “jailbreak” our web applications to become more extensible, customizable, and ultimately more useful. Conceptually, web applications only have a request-based “input” mechanism: web APIs. They lack an event-based output mechanism, and this is the role of web hooks. People talk about Unix pipes for the web, but they forget: pipes are based on standard input and standard output. Feeds are not a sufficient form of output for this, which is partly why Yahoo Pipes was not the game changer some people expected. Instead, we need adoption of a simple, real-time, event-driven mechanism, and web hooks seem to be the answer. Web hooks are bringing a new level of event-based programming to the web.

I think the use of the term ‘jailbreak’ is useful in understanding what HookPress brings to the WordPress ecosystem. WordPress is an application written in PHP and if you wish to develop a plugin or theme for WordPress you are required to use the PHP programming language. No bad thing but the HookPress plugin ‘jailbreaks’ the requirement to work with WordPress in PHP by turning WordPress’ hooks (‘actions’ and ‘filters’) into web hooks.

WordPress actions and filters, are basically inbuilt features that allow developers to ‘hook’ into WordPress with their plugins and themes. Here’s the official definition:

Hooks are provided by WordPress to allow your plugin to ‘hook into’ the rest of WordPress; that is, to call functions in your plugin at specific times, and thereby set your plugin in motion. There are two kinds of hooks:

  1. Actions: Actions are the hooks that the WordPress core launches at specific points during execution, or when specific events occur. Your plugin can specify that one or more of its PHP functions are executed at these points, using the Action API.
  2. Filters: Filters are the hooks that WordPress launches to modify text of various types before adding it to the database or sending it to the browser screen. Your plugin can specify that one or more of its PHP functions is executed to modify specific types of text at these times, using the Filter API.

So, if I understand all this correctly, what HookPress does is turn WordPress hooks into web hooks which post the output of the executed actions or filters to scripts written in other languages such as Python, Perl, Ruby and Javascript (they can be written in PHP, too) hosted elsewhere on the web.   In the example given in the HookPress video, the WordPress output of the action, ‘publish_post‘, along with two variables ‘post_title’ and ‘post_url’, was posted to a script hosted on scriptlets.org,  which performs the event of sending a tweet which includes the title and URL of the WordPress post that has just been published. All this happens as fast as the component parts of the web allows, i.e. in ‘real time’.

In other words, what is happening is that WordPress is posting data to a URL, where lies a script, which takes that data and creates an event which notifies another application. Because the scripts can be hosted elsewhere, on large cloud platforms such as Google’s AppEngine, the burden of processing events can be passed off to somewhere else. I see now, why web hooks are likened to Unix pipes, in that the “output of each process feeds directly as input to the next one” and so on. In the case of HookPress, the output of the ‘publish_post’ hook feeds directly as input to the scriptlet and the output of that feeds directly as input to the Twitter API which outputs to the twitter client.

Besides creating notifications from WordPress actions, the other thing that HookPress does (still with me on this ‘learning journey’ ??? I’ve been reading, writing and revising this blog post for hours now…), is extend the functionality of WordPress through the use of WordPress filters. Remember that filters in WordPress, modify text before sending it to the database and/or displaying it on your computer screen. The example in the video, shows the web hook simply reversing the text before it is rendered on the screen. ‘This is a test’ becomes ‘tset a si sihT’.

The output of the ‘the_content‘ filter has been posted to the web hook, which has reversed the order of the blog post content and returned it back to WordPress which renders the modified blog post.

Whereas the action web hooks are about providing event-driven notifications, the filter web hooks allow developers to extend the functionality of WordPress itself in PHP and other scripting languages.  In both cases, web hooks ‘jailbreak’ WordPress by turning it into a single process in a series of piped processes where web hooks create, modify and distribute data.

Finally, I’ll leave you with this presentation, which is all about web hooks.

In the presentation, there are two quotes which I found useful. One from Wikipedia which kind of summarises what HookPress is doing to WordPress:

“In computer programming, hooking is a technique used to alter or augment the behaviour of [a programme], often without having access to its source code.”

and another from Marc Prensky, which relates back to my point about using WordPress as a way to learn about web technologies in a broader sense. WordPress+HookPress is where programming for WordPress leaves the back room:

As programming becomes more important, it will leave the back room and become a key skill and attribute of our top intellectual and social classes, just as reading and writing did in the past.

  1. I am not ashamed to admit that I’m finding that my career is increasingly influenced by following the observations of Tony Hirst. Some people are so-called ‘thought-leaders’. I am not one of them and that is fine by me. I was talking to Richard Davis about this recently and, in mutual agreement, he quoted Mario Vargas Llosa, who wrote: “There are men whose only mission is to serve as intermediaries to others; one crosses them like bridges, and one goes further.” That’ll do me. []
  2. Note that themes are not necessarily a superficial makeover of a WordPress site. Like plugins, they have access to a rich and extensible set of functions. []
  3. I am thinking of taking the idea of WordPress as a window on web technology further and am tentatively planning on designing such a course with online journalism lecturer, Bernie Russell. It would be a boot camp for professional journalists wanting (needing…?) to understand the web as a public space and we would start with and keep returning to WordPress as a mainstream expression of various web technologies and standards. []