Bebop. A BuddyPress plugin for curating personal collections of (teaching) resources

We finished our JISC-funded Bebop project today. One of the main outcomes is a BuddyPress plugin that allows a user to import content they’ve uploaded on other sites, such as Flickr, Vime, YouTube and any site with an RSS feed, into their BuddyPress activity stream. What’s especially nice, is that the user can select which content appears in their activity stream, so it effectively allows them to curate collections of shared resources as part of their profile. There’s a good reason for this, and you can read all about it over on the project blog.

BuddyPress, Bebop and building the staff directory

The Bebop plugin and documentation

p.s. this was a nice outcome of the project, too: The benefits of code review

The cost of developing a good idea

How much does a student hacker need to develop a good idea to the point that it attracts further investment?

I’ve been thinking about this recently for a couple of reasons. I was reading the early Y Combinator site, via the Wayback Machine, about how they reckoned on $6,000 per person for their first Summer Founders Program. Each new startup could expect to receive less than $20K (the average is $17,000 / £10,000), with two or three friends being the ideal number of founders per company. The Summer Founders Program was aimed at undergraduate or graduating students.

I’ve also been looking at JISC’s Elevator funding programme, where people working in UK universities and colleges (with a *.ac.uk email address), are able to pitch an idea to receive up to £10,000 funding from JISC.  That’s the same amount of money Y Combinator seeds their successful applicants with. I think the JISC Elevator is a great idea, but looking at the proposals that have been submitted so far, I’m surprised and disappointed that there aren’t any proposals where the money goes directly to students to develop ideas of their own.  Maybe students haven’t been told? I’ll admit I’ve not publicised it at Lincoln, having been busy bidding for other JISC funds (where graduating 3rd year students are the main contributors to the projects) and awarding funds to projects of our own (where students receive most of the money). Still, I feel bad about not supporting JISC Elevator more. I have voted for one proposal.

I asked Alex, an undergrad and co-worker, how much a student who is hacking on an idea all day, every day, needs to live on in Lincoln, and he reckons about £600/month. That sounds harsh to me, so let’s assume they need £800/month and that there are three of them, because after all, if you can’t persuade a couple of friends that an idea is worth working on, then it probably isn’t a very good idea (or so says Y Combinator). On a related note, Google’s Summer of Code provides students with a $5000/£3000 stipend for the summer.

When I first heard about the JISC Elevator, my immediate thought was that the £10K maximum per project isn’t very much to attract FEC costed projects involving staff, but is perfect for offering to students as bursaries. A bursary, as I understand it, is supposed to cover the costs of living, rather than being seen as a wage, so they’re similar in purpose to the GSoC and Y Combinator funds. On our DIVERSE project, almost all of the money received went to paying the fees and bursaries of two MRes students. We are also prepared to contribute a larger percentage of the overall cost. Our recently funded beBOP project is an example of this, with a recent graduate being employed on grade 4, and the funding from JISC covering only 65% of the overall cost, compared to the maximum 80%.

I’ll admit, I don’t really understand how FEC works and where a lot of the money actually goes, but for the kinds of projects that the JISC Elevator is trying to attract, as well as JISC’s Rapid Innovation calls, I do wonder whether the GSoc or Y Combinator model of funding is a more cost-effective one. Pay students to hack over the summer, with a member of staff overseeing their work and call that the institutional contribution. £10K will pay for three students to hack over the summer, travel to a conference to talk about their work and pay for some servers on Rackspace for a few months. The tools to develop software in the early stages are cheap (a basic Linux stack on Rackspace is £7/month and there are enough open source tools available to explore ideas and develop prototypes, even if the ideal tool happens to be a proprietary one.

At Lincoln, we recognise that, given the opportunity and mentorship, undergraduate students have much to contribute. They’re not simply consumers of education. Like other universities, we’ve been running funding programmes each year that fund students to work on a research project with a member of staff over the summer. At Lincoln, it’s called UROS, the Undergraduate Research Opportunity Scheme. The Student as Producer UROS call was announced a few days ago. The LNCD group, which I co-ordinate awarded five projects £1000 each last week, which focus on the use of technology for education (more info on those projects soon). For the UROS and LNCD funded projects, almost all of the £1000 goes on undergraduate student bursaries. In my experience, undergraduate hackers can produce good work. Work that’s worth funding. Y Combinator thought so, too, and they’re now the most admired angel fund among young hackers. Each Y Combinator funded start-up is now guaranteed $150,000 as follow on funding by another investor. If you go Wayback to the first Summer Founders Program FAQ, you’ll see this:

Why are you doing this?

Partly because we feel guilty that we all got rich almost seven years ago, and still haven’t yet given seed money to new startups; partly because we think it is an interesting hack; and partly because we think it may actually make money.

We suspect that students, and particularly undergrads, are undervalued. Twenty years ago the idea of grad students starting companies would have seemed odd. Not after Yahoo and Google. And if grad students can do it, why not undergrads too?

I agree. Undergraduates can do it and I think institutions, together with JISC, should be thinking about our own Hatchery for Hackers.

Implementing OAuth 2.0

One of the lasting outcomes of our Total Recal ‘rapid innovation’ project in 2010, was that Alex Bilbie wrote the first (and only) OAuth 2.0 server for the CodeIgniter PHP development framework that we use. Since then, he’s been refining it and with every new project, we’ve been using it as part of our API-driven approach to development. As far as we know, the use of the OAuth 2.0 specification, which should be finalised at a forthcoming IETF meeting, is not yet being used by any other university in the UK. There are a few examples of OAuth revision A in use, but OAuth 2.0 is a major revision currently in its 23rd draft.

As a result of his work, Alex was invited to talk about OAuth 2.0 at Eduserv’s Federated Access Management conference last year.

OAuth 2.0

View more presentations by Alex Bilbie

Nick Jackson gave the same presentation at the Dev8D conference a couple of weeks ago.

Since Total Recal, we’ve used OAuth 2.0 for Jerome, data.lincoln.ac.uk, Zendesk, Get Satisfaction, and more recently Orbital and now ON Course.  We’re at the stage where our ‘single sign on’ domain https://sso.lincoln.ac.uk is the gateway to our OAuth 2.0 implementation and it will soon be running on two servers for redundancy. In short, due to various JISC projects helping pave the way, it has been formally adopted by central ICT Services, and staff and students are gradually being given control over what services their identity is bound to and what permissions those services have.

Single Sign On at Lincoln
Single Sign On at Lincoln

The work Nick is doing on the Orbital project is extending Alex’s OAuth 2.0 server to include some of the optional parts of the specification which we’ve not been using at Lincoln, such as refresh tokens and using HTTP Authentication with the client credentials flow. This means that the server is able to drop straight in to a wider range of projects and services.

Recently, JISC published a call for project proposal around Access and Identity Management (AIM), which I am starting to write a bid for. Appendix E1 states:

JISC is particularly interested in seeing innovative and new uses for OAuth. Bids should show how this technology brings benefits to the community and can help address institutional requirements within research, teaching and learning, work based learning, administration and Business Community Engagement.

In Total Recal, we released version 1 of the server code but have learned a lot since that project through integrating OAuth with other services. Version 2 of our OAuth server is more representative of our current implementation and fully implements the latest draft (23) of the specification.

However, this is what access and identity management currently looks like:

SSO Current Situation
SSO Current Situation (click the image)

At the moment, the most widespread use of the OAuth server is Zendesk, our ICT and Estates online support service. Projects such as Jerome, Orbital, and ON Course, as well as three 3rd year Computer Science student dissertation projects are using it, too. The plan is to use OAuth alongside Microsoft’s Unified Access Gateway (UAG), which can talk SAML to OAuth via the OAuth SAML 2.0 specification. Here’s what we intend to do:

SSO Ideal Situation
SSO Ideal Situation (click image)

The primary driver for this is the ‘student experience’ and it cuts three ways:

  1. Richer sharing of data between applications: A student or lecturer should be able to identity themselves to multiple applications and approve access to the sharing of personal data between those applications.
  2. A consistent user experience: What we’re aiming for at first is not strictly ‘single sign on’, but rather ‘consistent sign on’, where the user is presented with a consistent UX when signing into disparate applications.
  3. Rapid deployment: New applications that we develop or purchase should be easier to implement, plugging into either OAuth or the UAG and immediately benefiting from 1) and 2).

Following a recent meeting between ICT and the Library, we agreed to take the following steps:

  1. All library (and ICT) applications that we operate internally must have Active Directory sign-in instead of local databases. Almost all of our applications achieve this already. This is the first step towards step (3).
  2. All web-based applications must offer a consistent looking sign-in screen based on the sso.lincoln.ac.uk design (which uses the Common Web Design). This is the second step towards (3).
  3. All systems must implement web-based single sign on via OAuth, SAML or ADFS and they will be sent to either UAG or the OAuth/SAML server.

The library are going to investigate to what extent we can do (2) with their applications such as Horizon and EPrints, and from then on, systems that are purchased or updated must do (3). It also makes sense to look at EPrints and WordPress in the short-term as applications that can use OAuth.

Two of the outputs we’ll propose to JISC are a case study of this work, as well as further development of the open source server Alex and Nick have been developing including an implementation of the OAuth SAML specification that we’ll share. Like our related work on staff profiles, the need to get access and identity right is becoming increasingly apparent as staff and students become accustomed to the way access and identity works elsewhere on the web. For Lincoln, a combination of OAuth and UAG is the preferred route to achieving consistent sign on across all applications, bridging both the internally facing business applications managed by ICT (e.g. Sharepoint, Exchange, Blackboard) and the more outward facing academic and social applications such as those developed and run by the Library and the Centre for Educational Research and Development.

Work at web scale* on the Orbital project

This job is now formally open for applications.

Just a heads up to say that we’ll be advertising for a Web Developer to work on Orbital, our JISC-funded ‘Managing Research Data’ project. The post, starting in March/April, will be a 12 month, full-time, grade 5 (c.£21K) position.

The Web Developer (‘you’) will be working in the Centre for Educational Research and Development, alongside Nick Jackson, Lead Developer on Orbital, and also benefit from being in a team that includes staff in central ICT services and the Library. Orbital builds on and extends previous work we’ve been doing over the last couple of years, so if you’re interested, you should read through our projects pages.

If we were to summarise our technologies and interests I guess they would be #agile, #opensource, #opendata #LAMP, #php, #codeigniter, #mongoDB, #OAuth, #APIs, #HTML5, #CSS3, #github and moving towards #RDF and #LinkedData.

Just seeing these hashtags listed together should cause your heart to beat with excitement 🙂

When we advertise in January, you’ll see that the job spec is actually a pretty standard affair. What I want to emphasise here is how interesting and fun the job will be.

The key section in the Job Description is what you’d be working on with Nick:

  1. Development and implementation of a set of web services, which re-use and develop our previous, JISC-funded work as well as other initiatives (e.g. SWORD and DataCite DOIs).
  2. Documented source code will be made available under an open source license by the end of the project.
  3. Development and implementation of mechanisms for managing and transferring data, including the use of MongoDB, OAuth, read/write RESTful APIs, SWORD2 interoperability, and integration with the administrative functions of EPrints.

That actually summarises a lot of work.

I’m managing the project and try to run things with as little hierarchy as possible within a university environment. You’ll always know the project priorities and will be trusted to self-organise and deliver on time, working to two-week iterations and, roughly, monthly releases. I regularly reflect on how we work and our overall working environment. For Orbital, I favour the Crystal Clear agile methodology, as does Nick. You’ll be encouraged to reflect on this with us, too.

We work hard, and not always 9-5pm, but we work at a pace that is sustainable over a long period of time. We take our work seriously but, in the spirit of hacking, are always looking for ways to have fun, too. We recognise that we’re fortunate to be working in a diverse and intellectually stimulating academic environment, but are user/product focused at the end of the day. You’ll be working directly with our users, who are Researchers in the School of Engineering and Siemens, and staff in the Library and ICT. You’ll need to be showing them refreshed, working software every couple of weeks and iteratively improving Orbital, based on their feedback and requirements. There may also be times when you’ll be asked to talk publicly about your work and you’ll be encouraged to blog about it every so often, too. I expect the project to produce one or two conference/journal papers, and you’ll be named as a contributor and can take as active role in that as you like.

I hope this sounds like an interesting job. At £21K, I recognise that it will probably attract younger developers looking to gain experience, though of course, we welcome applications from anyone whatever your age. By the time the post starts, we’ll have set up a decent dev/staging/production environment, hosted in the cloud, and relying on Github and Jenkins to keep things versioned, integrated and tested. Nick will have been developing Orbital for a couple of months or more and laid the groundwork for someone to start coding quickly in a supportive environment.

If you’re thinking of applying and don’t live in Lincoln, you’ll be pleased to know that it’s a decent small city, and a relatively cheap place to live. The campus is modern and sits by a Marina in the middle of the city. You can walk to work. I love the place. Oh, and you can choose your own hardware for development, within reason. Most of us use Macs, but whatever suits you. I’ll ask the successful candidate what they prefer when we offer them the job.

If, after reading around the project website, you’ve got any questions about the post, please do get in touch. Thanks.

* Wondering what the hell ‘web scale’ means? Something like this.

Orbital: A proposed Managing Research Data project

Following my usual tradition, I am dutifully posting a bid document that I submitted today to JISC’s Managing Research Data Programme. As you might already know, I do this in an attempt to open up the process of bid writing a bit more, in what is normally a competitive environment. I also hope that it might attract some interest from and possible future collaboration with other people in the university sector, whether we are successful in winning funding or not. We’ve been pretty successful with our bids over the last couple of years, too, and have received good feedback from JISC on the quality of our bids, so it seems like the decent thing to share good practice.

Our proposed project is called Orbital because we’re intending to build services for managing research data that ‘orbit’ around Nucleus, the data store we built during the Total Recal and Jerome projects. The bid was a pretty easy one to write, to be honest. Everything felt right as soon as I read the call documentation because I could see a way of re-using and further developing the work we’ve been doing since I joined the university in 2007. Of course, we’ve set ourselves some new challenges with this project and much work needs to be done in all phases of the project, but having the experience of building web services around large institutional data sets, gives me the confidence that we can tackle what is a really important issue for us – for any university: managing a growing body of research data. It’s also a project that takes me back to my roots, having joined the university to work on our Institutional Repository project. Prior to that I was an Archivist at the BFI National Film and Television Archive and Project Manager for the development of Amnesty International’s Digital Asset Management system. It was good to revisit the the whole digital archiving domain again and I even re-discovered a blog I kept in 2006 while on JISC’s week-long Digital Preservation Training Programme.

Although the bid has been sent off now, and who knows whether it will be funded or not, the process of writing the bid has been really useful. I had planned to spend much of July drafting a journal paper but seeing the call, switched into bid writing mode. Writing bids regularly, I try to get something out of them, despite knowing that they may not be funded. The idea of retrospectively viewing unfunded bids as a waste of time would depress the hell out of me and so I try to approach it as a reflective process, where I talk with colleagues about what we’ve done, where were are now and where we want to go with our work. Through writing this bid, it became really clear how the work we’ve been doing on other projects has brought us to the point where we have a good team of people who have developed a very modern, extensible and flexible technical framework which we can deploy in a number of domains, including managing research data. It’s all in the bid, so I won’t repeat it here, but it’s something we should be proud of.

I think that one of the things that good developers do is identify and/or build the tools they need to do their job effectively. That’s what we’ve been doing with WordPress, the Common Web Design, OAuth, Nucleus, data dot lincoln,  the Jerome discovery tool and the Linking You toolkit, so that now we have the skills and the tools to tackle future work more efficiently and have fun, too. There’s nothing that kills the fun of development more than having to work with crap tools.

Open Data at Lincoln: What have we got?

Tony Hirst recently blogged about the Open Data scene in UK HE, mentioning Lincoln as one of the few universities that are currently contributing HEI-related #opendata to the web. Sooner or later, I’ll write a more reflective post, but here I just wanted to document the current situation (that I’m aware of) at Lincoln. There are two groups that take an interest in furthering open data at Lincoln: LiSC, led by Prof. Shaun Lawson, and LNCD, the new cross-university group I co-ordinate which consolidates a lot of the previous and current work listed below. (For a broader overview of recent work, see this post).

Derek Foster in LiSC recently released energy data from our main campus buildings, updated every 2hrs on Pachube. I was just speaking to Nick and Alex and I think they plan to pull this data into our nucleus datastore, combine it with the campus location-based work we’ve done and generate dynamic heat maps (assuming Derek isn’t already working on something similar??)

LiSC are also mashing open data from the UK Police Crime Statistics database to create a social application called FearSquare and last week put together MashMyGov, a site that randomly suggests mashups using data sourced from Data.Gov.UK.

In the past couple of years, LNCD have worked on:

JISCPress, a 2009/10 project we worked on that didn’t release any data but developed a prototype WordPress platform that atomises documents for publication and comment on the web and spits out lots of data in open formats. It also uses OpenCalais, Triplify and can push RDF Linked Data to the Talis Platform. JISC now use it to publish documents for comment.

Total Recal, a JISC-funded project we completed recently and will roll out across the university this September. As well as providing a fairly comprehensive and flexible calendaring service at the university, it allowed us to work on our space-time data and develop a number of APIs on top of…

Nucleus, the epicentre of our open data efforts. This is a data store, using MongoDB, which aggregates data from a number of disparate university databases and makes that data available over secure APIs. Through a lot of hard work over the last year, Alex and Nick have compiled the single largest data store that we have at the university. Currently, it offers APIs to university events, calendars, locations and people. We’ll also be adding APIs to over 250,000 CC0 licensed bibliographic records held in Nucleus, too (see Jerome below). It also uses the OAuth-based authentication that Alex has developed.

Linking You, is a JISC-funded project we delivered last week to JISC, which looked at our use of URIs, undertook a comparative study of 40 HEI websites (more to come), proposed a high-level data model for use by the HEI sector and made some recommendations for further work. What we’ve learned on this project will have a lasting effect on the way we present our data and on our wider advocacy of open data to the university sector. I really hope that our recommendations will lead us to more discussion and collaboration with people interested in opening university data.

lncn.eu, a URL shortener that Alex and Nick developed in their spare time for a while and has since been formally adopted by the university. Naturally, lncn.eu has an API and can be used (e.g. Jerome) as a proxy for other services, collecting real-time analytics.

Jerome, is a current JISC-funded project that will release over 250,000 bibliographic records under a CC0 license. The data is stored in Nucleus and documented APIs will be available by the end of July. This is a very cool project managed by Paul Stainthorp in the Library (who’s also a member of LNCD).

We’re currently using data.online.lincoln.ac.uk to document the data that is accessible over our APIs. At some point, I can see us moving to data.lincoln.ac.uk – we just need to find time to discuss this with the right people. So far, we haven’t really gone down the RDF/Linked Data route, preferring to offer data that is linked (e.g. locations and events data are linked) and publicly accessible over APIs that are authenticated where necessary and open whenever possible. We are keen to engage in the RDF/Linked Data discussion – it’s just a matter of finding time. Please invite us to your discussions, if you think we might have something to contribute!

DIVERSE: A new JISC-funded project

I’m pleased to report that this week, Dr. John Murray and I were awarded one of JISC’s Teaching and Learning Innovation Grants. As usual, you can read the full bid to JISC on Google Docs. Very roughly, it’s a study of what happens when Social Bookmarking and Learning Analytics meets Student as Producer. I think it’s going to be a really interesting project, and I’m looking forward to working with John, the Principle Investigator, and two of his students.

This project will undertake a cross-departmental study of the use of ‘resource clouds’ for subject research, peer-review and validation of materials. It will build on DIVERSE, a tool for crowd-sourcing relevant subject resources, to determine student understanding of lecture and assignment topics. Through an analysis of qualitative and quantitative data, it will examine why students favour particular resources, determine how reading lists can be more effectively created and provide case studies of how students can contribute to lecture design.

This programme of funding requires a different application process, using a template and interview format. I think this is to improve the efficiency of bidding and judging, although in my case I spent more time on this bid and then preparing for the interview than any previous application! I quite enjoyed it and the online interview afforded me a bit more confidence than sitting face to face in a room as I was able to refer to written notes a lot more than if were I in right in front of the panel.

While talking to the interview panel I realised that I am in a fortunate position to be working in the Centre for Educational Research and Development, which leads the University’s Teaching and Learning Strategy (Student as Producer), as well as sharing the responsibility for technology for research, teaching and learning with the Library and ICT department. It’s these close working relationships with all the right people and the great support and enthusiasm we have from the Vice Chancellor’s Office, the Dean of Teaching and Learning, Head of ICT and the University Librarian, that allows me to talk with confidence about undertaking and sustaining our work.

As you’ll see from the full grant application, development of the DIVERSE tool was funded internally by the Student as Producer project under our Fund for Educational Development. I’m happy to say that DIVERSE will be the first project that our new group provides support for as we intend to do for many other projects. We’ll be ensuring that John and his students are able to integrate the DIVERSE tool into our CWD+Nucleus+OAuth framework over the summer in time for the new academic year.

This is exactly the model that I hope we can keep repeating: students and staff with good ideas around the use of technology for education (it doesn’t have to be software development!)  bid internally for a small amount of funding where required and, if successful, then we provide the appropriate institution-wide advocacy and support (technical integration, training, joint research, etc.). If such projects are also able to attract external funding, it suggests we’re doing the right thing.

JISC INF11 Programme Meeting: From unprojects to services

I’m going to the JISC Information Programme Meeting on Thursday and have been asked to join a panel where I’ll talk about our work at Lincoln under the heading ‘from unprojects to services’. Here are my notes.

Over the last couple of years, staff in CERD, The Library and ICT have worked closely together on a number of ‘rapid innovation’ projects, which have sometimes later attracted JISC funding.  Much of our work has been undertaken at the initiative of individual staff, who have benefited from a supportive ICT environment that allows us the freedom to develop and test our ideas without running into bureaucratic walls. ICT – in particular the head of the department, Mike Day, and head of the Online Services Team, Tim Simmonds – recognised the benefits of employing undergraduate students and recent graduates, and established a post which Nick Jackson and Alex Bilbie share. Alongside this, I have been applying for JISC funding and successful bids have allowed us to employ Nick and Alex full-time rather than part-time. In recent months, this has worked very well and currently much of their time is spent working on JISC-funded projects which bring value to the University. Below, are a list of the services that this culture of innovation has allowed us to work on over the last year or so. Click on the links to go to the services.

The Common Web Design: Distributed HTML5/CSS3 template for internal services
Posters: A repository for visual communications
lncn.eu: The official URL shortner for the university. Provides real-time stats, API and acts as branded/trusted proxy for other services.
Single Sign On: OAuth/SAML/Shibboleth/NTML/Eduroam integration
Zen Desk: University Help Desk
My Calendar: An aggregation of space-time data into a flexible web service. JISC-funded.
Nucleus: Datastore for People, Events, Bibliographic and Location data (and more to follow). Provides (open) APIs to all other services. MongoDB.
James Docherty, a third year student, used the nucleus datastore as a source of data for his final year project: Situated Displays for buildings, showing room booking information, posters and announcements.
Staff Directory: Fast, versatile people-focused search engine
Jerome: Fast, modern, personalised library search portal aggregating books, journals and EPrints data. JISC-funded.
Mobile: A directory of university services for mobile devices
Online Server Monitoring: A simple dashboard for anyone to check whether a service is working
QR Codes: Will be used for asset tags and already being used in rooms to create Help Desk tickets.
  • Most of these services push and pull data to Nucleus, the central, open datastore built on MongoDB. e.g. Zen Desk=People + Locations, My Calendar=Events, Jerome=Bibliographic
  • We’re currently looking at how Nucleus can also be a source for Linked Data. It has open(ish) APIs.
  • CWD sites transparently sign the person in to the site, if they are signed in elsewhere.
  • We like Open Source. SSO is mostly open source software. Alex has released his OAuth 2.0 code. CWD likely to be open source; MongoDB, bits and pieces from Jerome and My Calendar.
  • As we build these services, they are being integrated, too. e.g. lncn.eu will be a URL resolver for Jerome offering realtime monitoring; posters will show up in My Calendar events; CWD is the design framework for My Calendar.
  • Most of these services are for official launch in September. They will be included in the new ICT Handbook, included in brochures and other announcements.
  • We’re working with the Student Union to develop the use of FourSquare around the university.
  • Now that we know we can develop this way and that it works and we enjoy it, we’re hoping to expand from two to four student/graduate developers and have our own budget for hardware/software/conferences and to give to staff and students that want to join us.
  • Our approach links into the University’s Teaching and Learning Strategy: Student as Producer. We want to work with students and staff across disciplines to create useful, innovative and enjoyable online services that make the University of Lincoln a great place to work and study at. It’s not about a team that works on ‘educational technology’, but rather a network of people who develop and support technologies that make Lincoln a productive environment for research, teaching and learning. It’s inclusive, with students (and therefore learning) at its core.