One of the lasting outcomes of our Total Recal ‘rapid innovation’ project in 2010, was that Alex Bilbie wrote the first (and only) OAuth 2.0 server for the CodeIgniter PHP development framework that we use. Since then, he’s been refining it and with every new project, we’ve been using it as part of our API-driven approach to development. As far as we know, the use of the OAuth 2.0 specification, which should be finalised at a forthcoming IETF meeting, is not yet being used by any other university in the UK. There are a few examples of OAuth revision A in use, but OAuth 2.0 is a major revision currently in its 23rd draft.
As a result of his work, Alex was invited to talk about OAuth 2.0 at Eduserv’s Federated Access Management conference last year.
Nick Jackson gave the same presentation at the Dev8D conference a couple of weeks ago.
Since Total Recal, we’ve used OAuth 2.0 for Jerome, data.lincoln.ac.uk, Zendesk, Get Satisfaction, and more recently Orbital and now ON Course. We’re at the stage where our ‘single sign on’ domain https://sso.lincoln.ac.uk is the gateway to our OAuth 2.0 implementation and it will soon be running on two servers for redundancy. In short, due to various JISC projects helping pave the way, it has been formally adopted by central ICT Services, and staff and students are gradually being given control over what services their identity is bound to and what permissions those services have.
The work Nick is doing on the Orbital project is extending Alex’s OAuth 2.0 server to include some of the optional parts of the specification which we’ve not been using at Lincoln, such as refresh tokens and using HTTP Authentication with the client credentials flow. This means that the server is able to drop straight in to a wider range of projects and services.
Recently, JISC published a call for project proposal around Access and Identity Management (AIM), which I am starting to write a bid for. Appendix E1 states:
JISC is particularly interested in seeing innovative and new uses for OAuth. Bids should show how this technology brings benefits to the community and can help address institutional requirements within research, teaching and learning, work based learning, administration and Business Community Engagement.
In Total Recal, we released version 1 of the server code but have learned a lot since that project through integrating OAuth with other services. Version 2 of our OAuth server is more representative of our current implementation and fully implements the latest draft (23) of the specification.
However, this is what access and identity management currently looks like:
At the moment, the most widespread use of the OAuth server is Zendesk, our ICT and Estates online support service. Projects such as Jerome, Orbital, and ON Course, as well as three 3rd year Computer Science student dissertation projects are using it, too. The plan is to use OAuth alongside Microsoft’s Unified Access Gateway (UAG), which can talk SAML to OAuth via the OAuth SAML 2.0 specification. Here’s what we intend to do:
The primary driver for this is the ‘student experience’ and it cuts three ways:
Richer sharing of data between applications: A student or lecturer should be able to identity themselves to multiple applications and approve access to the sharing of personal data between those applications.
A consistent user experience: What we’re aiming for at first is not strictly ‘single sign on’, but rather ‘consistent sign on’, where the user is presented with a consistent UX when signing into disparate applications.
Rapid deployment: New applications that we develop or purchase should be easier to implement, plugging into either OAuth or the UAG and immediately benefiting from 1) and 2).
Following a recent meeting between ICT and the Library, we agreed to take the following steps:
All library (and ICT) applications that we operate internally must have Active Directory sign-in instead of local databases. Almost all of our applications achieve this already. This is the first step towards step (3).
All web-based applications must offer a consistent looking sign-in screen based on the sso.lincoln.ac.uk design (which uses the Common Web Design). This is the second step towards (3).
All systems must implement web-based single sign on via OAuth, SAML or ADFS and they will be sent to either UAG or the OAuth/SAML server.
The library are going to investigate to what extent we can do (2) with their applications such as Horizon and EPrints, and from then on, systems that are purchased or updated must do (3). It also makes sense to look at EPrints and WordPress in the short-term as applications that can use OAuth.
Two of the outputs we’ll propose to JISC are a case study of this work, as well as further development of the open source server Alex and Nick have been developing including an implementation of the OAuth SAML specification that we’ll share. Like our related work on staff profiles, the need to get access and identity right is becoming increasingly apparent as staff and students become accustomed to the way access and identity works elsewhere on the web. For Lincoln, a combination of OAuth and UAG is the preferred route to achieving consistent sign on across all applications, bridging both the internally facing business applications managed by ICT (e.g. Sharepoint, Exchange, Blackboard) and the more outward facing academic and social applications such as those developed and run by the Library and the Centre for Educational Research and Development.
In my previous post, I mentioned a small on-going project I have started to collect short interviews with developers working in universities. It is inspired by The Setup and relates to my interest in hacking in the academy. I hope that over time, it will provide a record of the people working in the developer community and add to the recognition of the work we do, how developers learn and how their working environment impacts upon their work and learning. Here’s the initial announcement I made at Dev8D. Please spread the word.
Calling all Developers working in Higher Education!
I’m setting up a site based on http://usesthis.com but focusing on Developers working in HE. It’s part of a research project I’m embarking on to understand how software developers (in universities) learn their craft.
It would be great if you could send me something. It will be published here http://hackingtheuniversity.net/ It shouldn’t take more than about 15 mins to write and about 5 mins to read (although longer is OK). Over time, I hope to compile a rich profile of developers who happen to work in universities.
How you answer is up to you; browse the site and read other people’s contributions. Alternatively, I’d be happy to accept something different, like this: http://why.usesthis.com/ You don’t need to hyperlink to your software and hardware. I will do that, in many cases automagically!
Here are my interview questions. If you don’t feel like writing, I’d happily accept an audio file to transcribe.
Who are you, and what do you do?
Who taught you how to do what you do?
What tools do you use?
Describe your dream working environment.
I need a nice image of you, too, at least 500×335 pixels.
During my attendance at Dev8D last week, I used the time to start pulling together some ideas I’ve been knocking around for a few months now. Most of the projects I’ve helped set up or led over the last three years have involved working closely with developers working in higher education and ever since JISCPress, we’ve been growing a culture where student developers, either on bursaries, employed part-time or full-time recent graduates (i.e. their first ‘proper job’) are core contributors to the project. My point here is that through these projects I have been able to observe how young hackers learn their craft and make the transition from a formal education in Computer Science to learning on the job (i.e. apprenticeship).
I use the word ‘craft’ deliberately. I think our work is much more closely aligned with craftsmanship than with engineering but I was unsure how to articulate this until I read Pete McBreen’s Software Craftsmanship. McBreen argues that a focus on craftsmanship is to return to the roots of software development:
Good software developers have always understood that programming is a craft skill. Regardless of the amount of arcane and detailed technical knowledge that a person has, in the end, application development comes down to feel and experience.
McBreen distinguished software craftsmanship from software engineering and computer science, not as their opposites but as a different tradition “that happily coexists with and benefits from science and engineering.” He compares the software craftsman to the blacksmith, both of whom transcend science and engineering and benefit from improvements in their tools, materials and understanding. For McBreen, GNU/Linux is an example of software craftsmanship that thrives due to the dedication, skill and craft of the people who contribute to the development of the operating system.
‘Software engineering’ was born out of a so-called ‘software crisis’, identified at a NATO conference in 1968. It was determined that the way out of that crisis was to apply an engineering approach to large scale state of the art defence projects. Since that time, argues McBreen, “the needs of the US Department of Defence have dominated the conversation about software engineering.”
software engineering is not all that relevant to many projects. Software engineering was created to solve the problems of really large groups working on multiyear projects. Most modern software development, however, is done in relatively small teams.
Not surprisingly, the practise of software craftsmanship is often allied with agile development, which emphasises the human aspects of software development, the creative and variable processes involved in creating software that meets human needs. Both share a concern with quality and both encourage continual reflective critique of one’s work in order to improve the software and oneself. In his book on Crystal Clear, for example, Alistair Cockburn identifies ‘reflective improvement’ and ‘osmotic communication’ as two of the three minimum requirements for a Crystal Clear project. In practice, reflective improvement refers to workshops and end-of-iteration discussion about how the work is going and how it might be improved. Osmotic communication refers to how flows of information between developers should be unobstructed by locked doors, walls and corridors. It’s about sitting developers in the same or adjacent rooms so that they absorb information about the project with as few barriers as possible. Access to information, a safe environment where people can respectfully speak their minds and close collegial working are highly valued. At times, developers will program side-by-side or in pairs at one workstation, so as to review each others’ work. He refers to this as ‘peer code peering’.
Both reflective improvement and osmotic communication can be enhanced by the choice of tools developers use and the feedback they provide to her. Craftsmen in all trades rely on their tools to provide feedback – a joiner’s plane, a responsive power drill, etc. Many craftsmen will make their own tools to improve their responsiveness. Cockburn notes that one of the most important tools developers can use is an automated regression test suite, which allows the team to continuously test their work and provides feedback to each developer about the quality of their code. Feedback from a Continuous Integration (CI) suite of tools can be usefully presented by ‘information radiators’, dashboards which typically provide status information of servers, the number of use cases delivered and the number of tests passed. Although, Cockburn doesn’t use the term, I think that reflective improvement and osmotic communication refer to the ‘learning environment’ that the software craftsman creates so as to improve their understanding of their work and further develop their craft.
References to the importance of learning from others and from one’s work are made throughout McBreen’s book, as well as an entire chapter at the end of the book called ‘Perpetual Learning’. There, he outlines how to create a ‘learning environment’ by building a library of books for developers to read, as well as ensuring that they take time out each week to practice or learn something new. Like Cockburn, he emphasises the importance of workshops and a series of seminars where developers discuss their work. In addition, McBreen suggests that developers are encouraged to attend and present at conferences and write papers about their work as well as take on the role of instructor where they are able to do so. Craftsmen continually undertake self-directed learning, preferring non-proprietary, open source tools that are sustained by a community and made freely available to learn from, but more importantly, software craftsmen learn from each other, with master craftsmen mentoring journeymen and apprentices.
In a more recent book on software craftsmanship, Apprenticeship Patterns, Hoover and Oshineye also devote a chapter to ‘Perpetual Learning‘, offering further practical advice to aspiring software craftsmen. They list the following:
Expand your bandwidth: Read books and articles, engage with your peers via conferences, social media and mailing lists, join user groups, study from open educational materials on the web
Practice, practice, practice: Take the time to practice your craft without interruptions, in an environment where you can feel comfortable making mistakes. i.e. ‘deliberate practice’ They borrow the term ‘code katas’, whereby programming exercises are repeated again and again until they become ingrained in the individual.
Breakable toys: Budget for failure by designing and building toy systems that are similar in toolset, but not in scope to the systems you build at work. i.e. build something personal to you, that you will learn from. Develop your own CMS or game that you can afford to break while learning.
Use the source: Seek out other people’s code and read it. Start with the applications and tools you use every day.
Reflect as you work: Become a reflective practitioner of software development. This involves regular introspection into how you are working.
Record what you learn: Keep a record of your journey in a journal, personal wiki, or blog. A chronological record of the lessons you have learned can provide inspiration to those you mentor, since it makes your journey explicit, but it can also give you a vital resource to draw upon.
Share what you learn: Early in your apprenticeship, develop the habit of regularly sharing the lessons you have learned. i.e. keep a blog, run workshop sessions, be part of a community of learners.
Create feedback loops: Create mechanisms for regularly gathering more or less objective external data about your performance. i.e. automated regression tests, information radiators, exams and certifications, pair programming, and asking your peers what they think.
Learn how you fail: Seek to identify the ways in which you tend to fail and try to resolve those that are worth fixing.
In his book, The Craftsman, Richard Sennett chooses to focus on open source software development as a modern form of craftsmanship (“the skill of making things well”). The value of Sennett’s book is its breadth of scope. While making no reference to McBreen’s earlier book, Sennett situates open source hacking and the development of GNU/Linux within the social and economic history of craftsmanship and our relationship with technology. For Sennett, Linux is a “public craft” and open source hackers are a “community of craftsmen.” In terms of learning this public craft, he compares ancient pottery making with open source hacking and finds that only the speed between problem finding and problem solving differentiates the two. In programming, and especially open source programming, the velocity at which we can learn can be much greater than in traditional, material crafts. Our tools and the open, distributed nature of developer communities enhance our opportunities for learning. ((Sennett’s observations also help us consider software craftsmanship together with learning theories such as cognitive apprenticeship, situated learning, and constructivism.))
In The Nature and Art of Workmanship, David Pye does not make reference to software development but does develop a very useful framework for identifying and understanding craftsmanship, which complements much of what Sennett and McBreen describe. For Pye, craftsmanship is the workmanship of risk; that is, work that is constantly at risk of error in the process of creation. A simple example of this is writing with a pen. In contrast to the workmanship of risk (craftsmanship) is the workmanship of certainty; that is, workmanship where the quality is always pre-determined and is usually found in quantity production, and always in fully automated work. An example of this would be modern printing. The workmanship of certainty is most common in modern, industrial society, but has existed in some form for hundreds, if not thousands of years. All types of workmanship exist somewhere on the axis between risk and certainty and furthermore are subject to varying degrees of regulation or freedom.
What distinguishes the degree of risk or certainty for Pye is the extent to which the workman’s tools regulate his work. Pye argues that a pure form of workmanship of risk is hardly ever seen in any trade; for centuries people have developed tools to help regulate their work in some way (e.g. jigs) and guarantee some degree of certainty in the quality of their work. Regulation of work does not necessarily lead to certainty as some tools such as a lathe can be used in combination with a free hand, producing unique objects that are nevertheless regulated in some respects such as their size but not their form.
A simple way of asking whether it is workmanship of risk or certainty is to ask, “is the result predetermined and unalterable once production begins?”
In the drawing above, I have speculated that software craftsmanship as McBreen describes it, would be called workmanship of risk by Pye, with regulation introduced by tools such as Integrated Development Environments (IDE), Continuous Integration (CI) suites and the general operating system environment the developer is working with. Although regulated, like almost all workmanship of risk, the software craftsman often produces something bespoke to the users’ requirements, iteratively working towards the desired result through the writing and refactoring of code. Agile software development recognises that the result of most software development cannot be predetermined and that projects run best when they remain responsive to the users’ feedback. Software Engineering, on the other hand, aims to eliminate as much of the risk as possible and pre-determine the outcome of the design and programming effort, which both McBreen and Pyritz both identify as a form of Taylorism.
The Institute of Electrical and Electronics Engineers defines software engineering as “the application of a systematic, disciplined, quantifiable approach to development, operation, and maintenance of software”. Pyritz questions whether the Taylorist model of scientific management is even viable in software development, quoting Humphrey, who asked ‘Why don’t they practice what they preach?’
The general practices of industrial software engineers are poor by almost any measure”. Why? “The educational system does not provide graduates with the practical skills they will later need. . . . Few software organizations are willing or able to provide the remedial training their new engineers need. Today’s software organizations have few if any role models who consistently demonstrate effective work habits and disciplines.
In learning to become software craftsmen we need role models, too. Both McBreen and Sennett emphasise the importance of apprenticeship and slowly learning by doing with others. This is what JISC’s DevCSI offers to the sector, running regular hack days, DevXS and the annual Dev8D conference. This year over 250 developers attended Dev8D to learn from each other.
While I was at Dev8D, I issued an informal survey asking developers a number of questions about their working environment, how they learn their craft, the tools they use and an assessment of their skills. I shall provide a summary of the responses in another post [UPDATE: here it is], but I am encouraged to do further research in determining how developers (working in tertiary education) learn their craft and how opportunities for learning might be improved.
Another related project I started while at Dev8D was Hacking the University, a simple website intended to collect short interviews with developers working in universities. It is inspired by The Setup and I hope that over time, it will provide a record of the people working in this community and add to the recognition of the work they do, how they learn and how their working environment impacts upon their work and learning.
If you are a developer working in a university, please consider contributing to Hacking the University and telling others about your approach to your craft.
Last week, I submitted a bid to JISC’s OER Rapid Innovation programme, proposing some work to turn BuddyPress profiles into both a consumer and producer application of open content. You can read the bid on Github. It’s the first time I’ve written a bid on Github, having previously use Google Docs to write bids that are publicly accessible during the bid writing process. With software development projects such as this latest one, it made sense to use the same source code repository that will be used to manage the project code as the bid writing process is an important part of the project’s history and code benefits from proper documentation, including in this case, a Use Case. With the exception of having to format the text for the final PDF submission, it was a nice way to write the bid, too, working solely with a text editor and knowing that the documents were properly versioned and backed up on Github.
I see no benefit to writing bids such as this in private, other than hiding the process by which I write grant applications. The projects I propose and the outputs they generate are all open source and usually promote some variation of openness (open access/source/education/data), so why not start with the writing of the bid? Perhaps someone will be generous enough to contribute in some way or even learn something from being able to see the bids in their raw state. It also stakes a claim on the nature of the proposal, too and with a CC license, the idea is sufficiently ‘protected’.
Sometimes there is a lot at stake when proposing a project (e.g. people’s jobs) and, I’ll admit, I didn’t share the Orbital project bid as I was writing it. In hindsight, I don’t think it would have changed anything had I opened up the bid writing process on that occasion either.
It would be nice to write a bid using Github with a partner institution. There are tools which allow you to visualise the activity around the Github repository, which might be interesting to look at. Perhaps something might be learned about how academics undertake this part of their work. It is an exercise I seem to undertake several times a year and a reflective part of my work that I actually enjoy. I was also thinking that Etherpad would likewise be a nice tool to use for collaborative bid writing, because you can literally play back the entire authoring of the document. A version of Etherpad was developed at the DevXS conference, which uses Github as a back end, offering us the best of both worlds it seems. We run our own Etherpad-lite server here at Lincoln, so I should look at how we might integrate it with Github.
One aspect of this latest project proposal is that we work with Matt Gold and his team on the CUNY Commons In A Box project. Matt and I have talked in the past about collaborating on an open data project and this will be a nice rehearsal for something bigger that I hope we can work on together. On that occasion, Matt, we should write the bid together over Github 😉
Just a heads up to say that we’ll be advertising for a Web Developer to work on Orbital, our JISC-funded ‘Managing Research Data’ project. The post, starting in March/April, will be a 12 month, full-time, grade 5 (c.£21K) position.
The Web Developer (‘you’) will be working in the Centre for Educational Research and Development, alongside Nick Jackson, Lead Developer on Orbital, and also benefit from being in a team that includes staff in central ICT services and the Library. Orbital builds on and extends previous work we’ve been doing over the last couple of years, so if you’re interested, you should read through our projects pages.
If we were to summarise our technologies and interests I guess they would be #agile, #opensource, #opendata #LAMP, #php, #codeigniter, #mongoDB, #OAuth, #APIs, #HTML5, #CSS3, #github and moving towards #RDF and #LinkedData.
Just seeing these hashtags listed together should cause your heart to beat with excitement 🙂
When we advertise in January, you’ll see that the job spec is actually a pretty standard affair. What I want to emphasise here is how interesting and fun the job will be.
The key section in the Job Description is what you’d be working on with Nick:
Development and implementation of a set of web services, which re-use and develop our previous, JISC-funded work as well as other initiatives (e.g. SWORD and DataCite DOIs).
Documented source code will be made available under an open source license by the end of the project.
Development and implementation of mechanisms for managing and transferring data, including the use of MongoDB, OAuth, read/write RESTful APIs, SWORD2 interoperability, and integration with the administrative functions of EPrints.
That actually summarises a lot of work.
I’m managing the project and try to run things with as little hierarchy as possible within a university environment. You’ll always know the project priorities and will be trusted to self-organise and deliver on time, working to two-week iterations and, roughly, monthly releases. I regularly reflect on how we work and our overall working environment. For Orbital, I favour the Crystal Clear agile methodology, as does Nick. You’ll be encouraged to reflect on this with us, too.
We work hard, and not always 9-5pm, but we work at a pace that is sustainable over a long period of time. We take our work seriously but, in the spirit of hacking, are always looking for ways to have fun, too. We recognise that we’re fortunate to be working in a diverse and intellectually stimulating academic environment, but are user/product focused at the end of the day. You’ll be working directly with our users, who are Researchers in the School of Engineering and Siemens, and staff in the Library and ICT. You’ll need to be showing them refreshed, working software every couple of weeks and iteratively improving Orbital, based on their feedback and requirements. There may also be times when you’ll be asked to talk publicly about your work and you’ll be encouraged to blog about it every so often, too. I expect the project to produce one or two conference/journal papers, and you’ll be named as a contributor and can take as active role in that as you like.
I hope this sounds like an interesting job. At £21K, I recognise that it will probably attract younger developers looking to gain experience, though of course, we welcome applications from anyone whatever your age. By the time the post starts, we’ll have set up a decent dev/staging/production environment, hosted in the cloud, and relying on Github and Jenkins to keep things versioned, integrated and tested. Nick will have been developing Orbital for a couple of months or more and laid the groundwork for someone to start coding quickly in a supportive environment.
If you’re thinking of applying and don’t live in Lincoln, you’ll be pleased to know that it’s a decent small city, and a relatively cheap place to live. The campus is modern and sits by a Marina in the middle of the city. You can walk to work. I love the place. Oh, and you can choose your own hardware for development, within reason. Most of us use Macs, but whatever suits you. I’ll ask the successful candidate what they prefer when we offer them the job.
If, after reading around the project website, you’ve got any questions about the post, please do get in touch. Thanks.
* Wondering what the hell ‘web scale’ means? Something like this.
I feel like I’ve arrived late to the party. I’d seen David Flanders and others working with 3D printers at Dev8D but I was thinking about other things at the time and never appreciated how significant and wonderful this new technology is and could be. If you’re wondering what 3D printing is, or just don’t get it, watch this and read the articles below. I’m hoping we can get a group together in Lincoln to start making stuff and more printers for others. I know there is interest among some staff and students here and it shouldn’t take much to bring together an informal group interested in exploring the different educational uses of 3D printers (as well as have a lot of fun).
I’ve spent the last five months helping to organise and host DevXS, a national student developer conference. The conference on 11-13th November was fully booked and a great success. Over 170 students attended from across the UK, representing 37 universities, as well as a further 20 tutors and developer mentors working in the Higher Education sector.
You can read more about DevXS on the conference blog which was updated throughout the weekend by a superb team of media students. There are lots of videos and presentation slides on the blog as well as pictures and information about the prize winners and their applications.
It was a really exhausting and satisfying experience to be involved in and not only was it the first conference of its kind in the UK but it looks like it will become an annual event hosted by a different university each year and organised by the JISC-funded DevCSI project.
You can read a report about the conference on the DevCSI website. The Guardian also published an article (originally titled ‘Hacking the Academy’) in the run up to the conference, which I wrote with Mike Neary.
Recently, I posted on the LNCD blog about our work on data.lincoln.ac.uk. You might find it interesting.
One of the by-products outcomes of our recent ‘proper’ projects is data.lincoln.ac.uk. This is simply a site that documents the data we are warehousing in our MongoDB datastore (called ‘Nucleus’), and the programatic methods by which we (and the public) can access that data. Most of the data is licensed for public use, but where appropriate (e.g. personal data), a secure access token must be requested. Currently, outside of our own projects, the only people needing/wanting secure access tokens are some third year computer science students who are using data.lincoln.ac.uk as the basis for their dissertation projects and require access to their own personal event data.
Our approach to publishing open data at the University of Lincoln has been to do so in a way that was immediately useful to the work we were doing…