I co-teach Online Journalism for level three students with Bernie Russell and this week, Tony Hirst from the OU came to Lincoln to give his, now annual, data-driven journalism class. Bernie and I prep the students a few weeks beforehand and then Tony rolls in and packs as much into the class as he can, leaving me and Bernie to pick up the pieces 😉
We’re grateful for it.
If you’re a student struggling with the Wikipedia/Pipes/Google Maps exercise, here’s a working example that you can clone and work backwards through to understand how it works. It’s basically slide 6 of the presentation above.
UPDATE: What follows is broken because of changes to the Wikipedia source page structure, changes to Yahoo Pipes and changes to Google Docs. Trying to keep it working is a pain, so it will have to stay broken for now.
Note how I’ve fetched the CSV into Yahoo Pipes, defined the data I’m interested in, renamed two key attributes, renamed the title attribute to be ‘population’ and then used the location builder in a loop block to determine the geo-locations. Once that’s done, it runs in the Pipe like this:
Is this displaying correctly? I’ve found that embeds directly from Yahoo Pipes can be a bit flaky.
However, if you right click on the KML link and paste the KML link into the search box of Google maps, then you should see something like this:
Yes, that’s right. Dr. Tony Hirst was recently made a Senior Fellow of the University of Lincoln and a Visiting Senior Fellow in Networked Teaching and Learning.
I was very keen to propose that Tony be recognised by the University of Lincoln for his important work over the last few years, as evidenced on his very popular OUseful blog. I also worked with Tony on the JISCPress project in 2009 and hope that this formal relationship he now has with Lincoln will lead to more, mutually beneficial collaboration with the LNCD group in the future.
Tony's annotated Twitter follower network: A new form of scholarly metric.
We’ll be seeing Tony in Lincoln next month as he’s a Keynote speaker at DevXS, the national student developer conference we’re hosting and helping to organise . A further visit is also planned in March, where he’ll be talking at our ‘Thinking Aloud’ seminar series. Outside the Centre for Educational Research and Development, where I work, there’s a lot of interest in Tony’s work in the School of Computing and School of Journalism, too, following his recent meeting with LNCD and LiSC.
As is often the case, I struggle at first glance to see the full implications of a new development in technology, which is why I so often rely on others to kick me up the arse before I get it. ((I am not ashamed to admit that I’m finding that my career is increasingly influenced by following the observations of Tony Hirst. Some people are so-called ‘thought-leaders’. I am not one of them and that is fine by me. I was talking to Richard Davis about this recently and, in mutual agreement, he quoted Mario Vargas Llosa, who wrote: “There are men whose only mission is to serve as intermediaries to others; one crosses them like bridges, and one goes further.” That’ll do me.))
Where I ramble about WordPress as a learning tool for the web…
I first read about web hooks while looking at WordPress, XMPP and FriendFeed’s SUP and then again when writing about PubSubHubbub. Since then, Dave Winer’s RSSCloud has come along, too, so there’s now plenty of healthy competition in the world of real time web and WordPress is, predictably, a mainstream testing ground for all of it. Before I go on to clarify my understanding of the implications of web hooks+WordPress, I should note that my main interest here is not web hooks nor specifically the real time web, which is interesting but realistically, not something I’m going to pursue with fervour. My main interest is that WordPress is an interesting and opportunistic technology platform for users, administrators and developers, alike. Whoever you are, if you want to understand how the web works and how innovations become mainstream, WordPress provides a decent space for exercising that interest. I find it increasingly irritating to explain WordPress in terms of ‘blogging’. I’ve very little interest in WordPress as a blog. I tend to treat WordPress as I did Linux, ten years ago. Learning about GNU/Linux is a fascinating, addictive and engaging way to learn about Operating Systems and the role of server technology in the world we live in. Similarly, I have found that learning about WordPress and, perhaps more significantly, the ecosystem of plugins and themes ((Note that themes are not necessarily a superficial makeover of a WordPress site. Like plugins, they have access to a rich and extensible set of functions.)) is instructive in learning about the technologies of the web. I encourage anyone with an interest, to sign up to a cheap shared host such as Dreamhost, and use their one-click WordPress offering to set up your playground for learning about the web. The cost of a domain name and self-hosting WordPress need not exceed $9 or £7/month. ((I am thinking of taking the idea of WordPress as a window on web technology further and am tentatively planning on designing such a course with online journalism lecturer, Bernie Russell. It would be a boot camp for professional journalists wanting (needing…?) to understand the web as a public space and we would start with and keep returning to WordPress as a mainstream expression of various web technologies and standards.))
… and back to web hooks
Within about 15 minutes of Tony tweeting about HookPress, I had watched the video, installed the plugin and sent a realtime tweet using web hooks from WordPress.
It’s pretty easy to get to grips with and if a repository of web hook scripts develops, even the non-programmers like me could make greater use of what web hooks offer.
Web hooks are user-defined callbacks over HTTP. They’re intended to, in a sense, “jailbreak” our web applications to become more extensible, customizable, and ultimately more useful. Conceptually, web applications only have a request-based “input” mechanism: web APIs. They lack an event-based output mechanism, and this is the role of web hooks. People talk about Unix pipes for the web, but they forget: pipes are based on standard input and standard output. Feeds are not a sufficient form of output for this, which is partly why Yahoo Pipes was not the game changer some people expected. Instead, we need adoption of a simple, real-time, event-driven mechanism, and web hooks seem to be the answer. Web hooks are bringing a new level of event-based programming to the web.
I think the use of the term ‘jailbreak’ is useful in understanding what HookPress brings to the WordPress ecosystem. WordPress is an application written in PHP and if you wish to develop a plugin or theme for WordPress you are required to use the PHP programming language. No bad thing but the HookPress plugin ‘jailbreaks’ the requirement to work with WordPress in PHP by turning WordPress’ hooks (‘actions’ and ‘filters’) into web hooks.
WordPress actions and filters, are basically inbuilt features that allow developers to ‘hook’ into WordPress with their plugins and themes. Here’s the official definition:
Hooks are provided by WordPress to allow your plugin to ‘hook into’ the rest of WordPress; that is, to call functions in your plugin at specific times, and thereby set your plugin in motion. There are two kinds of hooks:
Actions: Actions are the hooks that the WordPress core launches at specific points during execution, or when specific events occur. Your plugin can specify that one or more of its PHP functions are executed at these points, using the Action API.
Filters: Filters are the hooks that WordPress launches to modify text of various types before adding it to the database or sending it to the browser screen. Your plugin can specify that one or more of its PHP functions is executed to modify specific types of text at these times, using the Filter API.
So, if I understand all this correctly, what HookPress does is turn WordPress hooks into web hooks which post the output of the executed actions or filters to scripts written in other languages such as Python, Perl, Ruby and Javascript (they can be written in PHP, too) hosted elsewhere on the web. In the example given in the HookPress video, the WordPress output of the action, ‘publish_post‘, along with two variables ‘post_title’ and ‘post_url’, was posted to a script hosted on scriptlets.org, which performs the event of sending a tweet which includes the title and URL of the WordPress post that has just been published. All this happens as fast as the component parts of the web allows, i.e. in ‘real time’.
In other words, what is happening is that WordPress is posting data to a URL, where lies a script, which takes that data and creates an event which notifies another application. Because the scripts can be hosted elsewhere, on large cloud platforms such as Google’s AppEngine, the burden of processing events can be passed off to somewhere else. I see now, why web hooks are likened to Unix pipes, in that the “output of each process feeds directly as input to the next one” and so on. In the case of HookPress, the output of the ‘publish_post’ hook feeds directly as input to the scriptlet and the output of that feeds directly as input to the Twitter API which outputs to the twitter client.
Besides creating notifications from WordPress actions, the other thing that HookPress does (still with me on this ‘learning journey’ ??? I’ve been reading, writing and revising this blog post for hours now…), is extend the functionality of WordPress through the use of WordPress filters. Remember that filters in WordPress, modify text before sending it to the database and/or displaying it on your computer screen. The example in the video, shows the web hook simply reversing the text before it is rendered on the screen. ‘This is a test’ becomes ‘tset a si sihT’.
The output of the ‘the_content‘ filter has been posted to the web hook, which has reversed the order of the blog post content and returned it back to WordPress which renders the modified blog post.
Whereas the action web hooks are about providing event-driven notifications, the filter web hooks allow developers to extend the functionality of WordPress itself in PHP and other scripting languages. In both cases, web hooks ‘jailbreak’ WordPress by turning it into a single process in a series of piped processes where web hooks create, modify and distribute data.
Finally, I’ll leave you with this presentation, which is all about web hooks.
In the presentation, there are two quotes which I found useful. One from Wikipedia which kind of summarises what HookPress is doing to WordPress:
“In computer programming, hooking is a technique used to alter or augment the behaviour of [a programme], often without having access to its source code.”
and another from Marc Prensky, which relates back to my point about using WordPress as a way to learn about web technologies in a broader sense. WordPress+HookPress is where programming for WordPress leaves the back room:
As programming becomes more important, it will leave the back room and become a key skill and attribute of our top intellectual and social classes, just as reading and writing did in the past.
I’m very pleased to announce that my bid with Tony Hirst at the Open University, to develop a community platform for the JISC funding call process based on WriteToReply, was successful. The original bid document is publicly available and currently offers the most information on this six month, £32,500 project.
Note that this is an open project using open source software and we welcome volunteer contributions from anyone. I’ve set up a project blog, mailing list, wiki and code repository. Feel free to join us if this WriteToReply spin-off appeals to you. If you know anyone that might be interested, please do let them know.
If you’ve been following WriteToReply, you’ll know that we use WordPress Multi-User and CommentPress. Eddie Tejeda, the developer of CommentPress will be working with us on the project and this will result in significant further development of CommentPress 2. So, if you’re interested in CommentPress (as many people are), please consider following, contributing to and testing JISCPress.
I should also note that while the project is a spin-off of our work on WriteToReply, neither Tony or I are personally receiving any funds from JISC. The contributions from JISC to cover our time on this project are paid directly to our employers and does not result in any financial benefit to us or WriteToReply (which is in the process of being formalised as a non-profit business). In other words, while WriteToReply is a personal project, JISCPress is part of our normal work as employees of our universities (both Tony and I are expected to bid and win project funds – you get used to it after a while!). Money has been allocated to fund dedicated developer time to the project, which will pay Eddie and Alex, a student at the University of Lincoln, for their work.
Anyway, on with the project! Here’s the outline from the bid document:
This project will deliver a demonstrator prototype publishing platform for the JISC funding call and dissemination process. It will seek to show how WordPress Multi-User (WPMU) can be used as an effective document authoring, publishing, discussion and syndication platform for JISC’s funding calls and final project reports, and demonstrate how the cumulative effect of publishing this way will lead to an improved platform for the discovery and dissemination of grant-related information and project outputs. In so doing, we hope to provide a means by which JISC project investigators can more effectively discover, and hence build on, related JISC projects. In general, the project will seek to promote openness and collaboration from the point of bid announcements onwards.
The proposed platform is inspired and informed by WriteToReply, a service developed by the principle project staff (Joss Winn and Tony Hirst) in Spring 2009 which re-publishes consultation documents for public comment and allows anyone to re-publish a document for comment by their target community. In our view, this model of publishing meets many of the intended benefits and deliverables of the Rapid Innovation call and Information Environment Programme. The project will exploit well understood and popular open source technologies to implement an alternative infrastructure that enables new processes of funding-related content creation, improves communication around funding calls and enables web-centric methods of dissemination and content re-use. The platform will be extensible and could therefore be the object of further future development by the HE developer community through the creation of plugins that provide desired functionality in the future.
CommentPress is, for educators, one of the most important developments to come out of the WordPress community and one of the most significant innovations that I know of in online publishing. I first learned about it when I saw that Yale University Press were using it to invite comment on Yochai Benkler’s book, The Wealth of Networks. In its original form, CommentPress is a theme for WordPress that allows readers to comment on, annotate and discuss paragraphs of text. In fact, although installed as a theme, it transforms a site not only by design, but with functionality you’d normally expect from plugins. In CommentPress v1.x, form and function came as a single package. It’s worth reading about the background to CommentPress. You’ll see that it’s part of a larger course of research by the Institute for the Future of the Book.
Institute for the Future of the Book was founded in 2004 to [… stimulate] a broad rethinking—in publishing, academia and the world at large—of books as networked objects. CommentPress is a happy byproduct of this process, the result of a series of “networked book” experiments run by the Institute in 2006-7. The goal of these was to see whether a popular net-native publishing form, the blog, which, most would agree, is very good at covering the present moment in pithy, conversational bursts but lousy at handling larger, slow-developing works requiring more than chronological organization—whether this form might be refashioned to enable social interaction around long-form texts… We can imagine a number of possibilities: scholarly contexts: working papers, conferences, annotation projects, journals, collaborative glosses; educational: virtual classroom discussion around readings, study groups; journalism/public advocacy/networked democracy: social assessment and public dissection of government or corporate documents, cutting through opaque language and spin (like the Iraq Study Group Report, a presidential speech, the federal budget, a Walmart or Google press release); creative writing: workshopping story drafts, collaborative storytelling; recreational: social reading, book clubs.
We have started to use CommentPress at the University of Lincoln for the discussion of internal documents and feedback from staff has been good. Many are astonished at what it makes possible. A departmental research strategy paper received over 100 comments from nine staff; something we’d never have had by emailing the document out for comment. Of course, I am keen to use it to support courses and a colleague and I have recently applied for funding to use CommentPress in a course with over 100 Criminology students, who are normally asked to critique texts and respond by emailing Word documents to their tutor. Using CommentPress allows for transparent and open, formative feedback and assessment by both staff and student peers.
Outside of my work for the university, I’ve been developing WriteToReply, with Tony Hirst from the Open University. You can read about how we started WriteToReply and you’ll see that CommentPress is fundamental to what we’re trying to achieve and we’re using it for networked democracy, as suggested above. CommentPress is in fact, a comment engine for each document site. Two things make this possible. First, and most obvious, is the fact that readers on a document site can direct comments to specific paragraphs of text. Readers can also respond to other readers’ comments and a happy by-product of our re-publication of the Digital Britain – Interim Report, is that the discussion still continues, despite the consultation period being over. So CommentPress is an engine for on-site comment and discussion. Texts are dissected but remain whole; they also become social objects.
The second important contribution CommentPress has made is the provision of permalinks for each paragraph in the text. This provides a unique URI or URL for each paragraph of text, making linked references from third-party web sites possible. Combined with the trackback/pingback system built into decent web publishing platforms, CommentPress makes remote commenting on text possible, as Tony explains on his blog.
What this means is that the paragraph, action point, section or whatever can become a linked resource, or linked context, and can support remote commenting. And in turn, the remark made on the third party site can become a linked annotation to the corresponding part of the original report… How? Well through the judicious use of trackbacks… So even if you don’t want to comment on the Digital Britain Interim report on the WriteToReply site, but you do care, why not post your thoughts on your own blog, and link your thoughts directly back to the appropriate part of the report on WriteToReply?
It’s this feature, so easily missed, which makes CommentPress a comment engine. An engine suggests an underlying technology that drives something greater. By introducing paragraph permalinks, text can now be linked at a much more accurate and deeper level than was previous possible. Texts are transformed into uniquely identifiable resources of data. Academics can now reference paragraphs rather than page numbers and readers can reflect, comment and participate in the analysis of texts from their own site. For the reader, CommentPress provides a fluid interface to the document as a whole but at a technical level, explodes it across the Internet.
In the running of WriteToReply, we’ve tested CommentPress quite hard and found it to be a complex and fragile tool. Until recently, it hasn’t been updated to reflect the fast changing development of WordPress and because of its extensive use of Javascript, it clashes with other plugins, so while it transforms a WordPress site, it also restricts functionality otherwise possible. Fortunately, CommentPress 2 is being actively worked on and I’ve been helping to test it with Eddie Tejeda, the original developer. It’s currently in beta, but Eddie is responding to my feedback and fixing issues rapidly. There is a mailing list for CommentPress and the code is publicly accessible.
CommentPress 2.2 Beta
If you test CommentPress 2, you’ll immediately see that it’s been split into a suite of plugins and themes and that it’s now much more flexible in terms of compatibility with other WordPress plugins and in being able to select different components, options and themes. Notably, paragraph permalinks are available as a separate plugin, which means that any WordPress blog will be able to have paragraph-level URIs, without necessarily supporting paragraph level commenting. My test site is on WriteToReply. Feel free to have a look and post comments, if you wish. As I write, it’s not quite ready for everyday use, but at the speed which Eddie has been working over the last few days, I’m confident that I’ll be able to use it here at the university and on WriteToReply before the month’s out. If you’re used to using v1.4.1, you’ll notice a lot of change. Remember that it’s still beta software and that not all of the features have been fully implemented yet. It would be great if other people could help test it across various browsers and with different documents. Multimedia is not something I’ve yet been able to throw at it, for example.
Finally, CommentPress needs continued support in terms of testing, reporting issues, bug fxes and feature development. This can be done voluntarily, but given it’s potential to support education, business and government consultations, I for one, will be looking for ways to raise funding to help support all of this. If you know of any possible funding opportunities within UK Higher Education, please do let me know.
In our weekly team meeting, I mentioned that I’d created some visualisations of the RAE research funding allocations. I also mentioned that Tony Hirst had previously done the same for the HEFCE teaching funding allocations. I offered to send everyone links to these, but before do so, I thought I’d have a go at re-creating the HEFCE visualisations myself to get a bit more practice in with IBM’s Many Eyes Wikified. So this is a companion piece to my previous post. All credit to Tony for opening my eyes to this stuff.
So, HEFCE have announced the 2009/10 grant allocations for UK Higher and Further Education institutions and provided full spreadsheets of the figures. I’ve imported the data into Google Spreadsheets and made the three tables publicly accessible as CSV files (1), (2), (3). Note that I’ve stripped out all data relating to FE grant allocations, which is included in the original spreadsheets.
Next, I’ve imported the CSV files into IBM’s Many Eyes Wikified (1), (2), (3), and these wikified tables are now the data sources for the following visualisations.
Recurrent grant for academic year 2009-10
The Pie
Bar Chart
Matrix
Bubble Chart
Comparison with 2008-09 academic year recurrent grant
Yesterday, the results of the funding allocation for research in UK Higher Education were announced and published on the Times Higher Education website.
Successive RAEs have concentrated research cash in the hands of the elite. This time around, the pie has been shared more widely.
The full spreadsheet of results being available, I thought this was a good opportunity for someone to visualise the data, so I published the data on Google Docs as a CSV file, which Tony Hirst fed into IBM’s Many Eyes wiki and now we can really see how the pie has been shared. Click on the images to view the interactive visualisations.
A pie…
Funding allocation by university group
Some bubbles…
FTE staff submitted to RAE by institution and coloured by group
A bar chart…
% change in total recurrent research funding by group
and a matrix…
Cash change in funding by institution and group
You can read about the University of Lincoln’s 628% increase in funding, here and here.
Together with the ITC Department, we’ve recently begun a feasibility study which looks at related areas of the university’s ITC provision. It brings together three, originally separate proposals to look at upgrading our wireless infrastructure, provide a more flexible desktop experience through virtualisation and improve our understanding of and support for Netbooks and Mobile Internet Devices. It’s good to be working so closely with our ITC Department. So often I hear people at other institutions complain about their ITC departments being ‘brick walls’ and showing no flexibility, but fortunately I can’t say that about my experience at the University of Lincoln.
The Head of ITC sent me a link to this video today. It’s a good example of why our study is both necessary and worthwhile.
Related to this, Tony Hirst recently bookmarked this ITC syllabus for 13-14 year olds recently, which, together with the video, provides a clear indication of what’s happening in schools.
We’re working closely with the Student Council and Academics and intend to survey them on the issues raised by the study early in the new year. We’ve started talking to vendors of desktop and application virtualisation ‘solutions’, too (the virtualisation of our server infrastructure is almost complete). We’re also lining up some visits to other institutions that have experience in these areas.
If you or your school, your FE or HE insititution has seriously considered or implemented desktop and/or application virtualisation, a full service wireless infrastructure (i.e. it matches the services on your wired network) and support Linux and XP-based Netbooks and other mobile devices, please do get in touch or leave a comment below.
