BenjaminWootton.co.uk

Someone asked me today for recommendations for some books to read on agile.

I came up with the following reading list, which interestingly, doesn’t include any books that are specifically on the topic of agile.

Continuous Delivery

Though this is a relatively new book, I believe this is the one that has been the most beneficial to me as a developer.  All of my jobs in software development have had significant operations components – building and packaging releases, automated testing, configuration management, deployments etc.  Bringing the spirit of automation and repeatability to this in the aim of being able to deliver faster, more repeatably, with less effort is a huge area of low hanging fruit for software teams.

Growing Object Oriented Software

I have used test driven development for a number of years, but this book really bought home how we can relentlessly use tests to optimise the design, clarity, and quality of the code and eventually the products that we are building.  I believe that automated tests at multiple levels of abstraction are absolutely fundamental to agile delivery of software, and that you need to adopt TDD as a fundamental practice if you are going to deliver complex software as early and often as agile implies.

The Pragmatic Programmer

This is the first book that I read which made me start to think of development as a process that is much bigger than simply writing code; things such as your tools and environment, your team, your tools, your testing strategies etc.  All of the items discussed in the book are fertile areas for optimisation which can provide a net benefit to your software product and team.  The Pragmatic Programmer is all about working smarter, not harder, which is something I try to keep at the front of my mind during the day job.

Peopleware

Peopleware concentrates on the human side of software development work and backs up the claims with reference to real data.  Most techies and managers would think it was a little fluffy to talk about concerns such as how the office is laid out, working hours, the need for flow, dress codes etc, but software development is a team sport and the human factor is always more prescient than we give it credit for.  By improving tools you can improve productivity in small increments, but by improving the situation, working practices, and motivation of your people, you’ll likely gain out-sized benefits for relatively small investments.

The Goal

This is not a software book but focuses on system operations and ‘systems thinking’.  If we step back and look at most working software teams, they’re generally part of some broader system providing inputs and outputs.  And like any system, this broad end to end process incorporates bottlenecks, constraints, feedback loops, inefficiencies etc.  By stepping back and viewing the system, we can optimise it and smooth out the workloads, making some really quite substantial improvements to the throughput of the system and the quality of the outputs.

So that’s my list.  I think what all of these titles have in common is that they are all about the things that developers need to spend time on that is slightly around the edge of the act of physically writing code.  All of this extraneous stuff has huge potential for optimisation, but has traditionally just been left to evolve without any critical thought and with minimal attempt to improve upon.  These books are great starting points towards this broader focus.

Last week I attended a talk by Rob Harrop at SkillsMatter on the topic of Continuous Delivery.  [Video Here]

Continuous Delivery is all about setting up your development processes such that you can deliver into production much more frequently than is typical, perhaps with multiple releases per day.

This is a topic I’ve been interested in for a long while having worked on projects with very long release cycles in the past.

There’s a lot being written about continuous delivery and I’ve read a lot of it, but usually the same companies are held up as examples of the practice – Facebook, Twitter, LinkedIn, Flickr – the trendy consumer web companies.  With that said, I was really interested in hearing more about how it was being done in the real world in a smaller company.

I took away these points from the presentation:

People & Process are more important than tools.  There’s a lot of talk about tools such as Cloud and the various Devops platforms in this conversation.  These tools can you save you time, but much more important is getting the team in line such that the good people are trusted and made able to put the systems in place that give them the control to push their work straight into production.

Ruby – Rob and his company have clearly come around to the idea of Ruby as a tool to manage their infrastructure and continuous deployments.  The Ruby community has generated some very high quality testing tools such as Rake, RSpec and Cucumber which are very elegant and efficient tools for carrying out continuous deployments.

Virtualisation – Virtualisation and cloud is key to all of this and tools such as Vagrant & Chef can help to bring up environments quickly and release software in a repeatable and scalable way.

REST & SOA – A service based model has helped with continuous deployment goals as it means they can keep the whole architecture loosely coupled and deploy and rollback components quickly and independently.  REST and SOA can be a real boon for continuous deployment.

Minimise Branching – Most changes and features are small enough that it’s not worth building seperate feature branches.  Heavy branching is slightly orthogonal to continuous delivery as it keeps work and changes off of the deployed mainline.

Artifacts – Be careful with code artifacts such as JARs and Ruby dependencies.  Once we’ve tested something, we need to make sure those same artifacts move through the testing and deployment pipeline in a locked down and repeatable way.

Continuous Deploys – It helps if you can architect for continuous delivery by allowing for some form of canary release or blue green deployment so that we can deploy without impacting the code base.  Considerations such as avoiding server side session state can help here.

It’s clear that Robs firm are benefiting from Continuous Delivery and have convinced the business that it was worth the investment.

It’s also clear that to get to the point of being able to do continuous delivery reliably implies huge change over the above the average development team – processes, people, code quality, quality control etc.

The benefits of continuous builds are so obvious, widely documented and experienced nowadays that anyone with even a passing interest in agile process improvement, a continuous integration server will be one of the first things they set up in their teams.

However, these same teams are often not using the investment in setting up and running continuous integration to it’s full potential.

Here’s a few mistakes and anti patterns that I have come across and read about:

Inadequate Tests

If you do not have enough automated tests in your code base, your continuous integration server is reduced to a fancy shared compiler.

The aim of continuous integration is not only to check that your code continuously compiles, but also to check that the integrated code base continuously holds a certain degree of quality as changes are committed by various developers and those changes interact and intersect.

The only way to practically achieve this is by judiciously implementing lots of unit tests, integration tests, and if possible, automated acceptance tests which can be ran against each build.

Infrequent Runs

Many teams do a daily or twice daily build.  In my experience, this can reduce the value of the continuous integration process as feedback on your check-ins can be too infrequent, and people tend to batch check-ins right before the build.

Ideally, we want people integrating with the shared build early and often, bringing forward as far as possible feedback from the integration process.

This drastically improves the potential for people to be working on the same areas of the code base without treading on each others toes.

Infrequent Commits

Related to the above is the fact that if developers are not committing to the code base frequently (perhaps due to practices such as feature branching) then code is not pushed into the integration server and is not being tested early enough.

If developers are working on complex features, they may find that their code is off of the integrated path that is being tested for days or even weeks at a time, increasing the chance of problems when it comes to merge or test the code later to the release.

No Continuous Deployment

If you have gone to the trouble of setting up a continuous integration server, but do not have a continuous deployment into some environment, your CI server is then effectively just reduced to a fancy shared compiler and test runner.

By frequently deploying the output of your continuous integration process into some environment, you’ll start to realise the benefits in terms of giving your users working software, and bringing forward the stage where you can begin to run acceptance tests and manual tests against the product.

When I first came across the central assertion in The Mythical Man Month that “adding manpower to a late software project makes it later”, I thought that this was claim was surely stretching the truth to make a point.

How could adding more expensive software developers make the project later?  Would this apply even if the developers added were very good ‘rockstar developers’?

However, having experienced this effect a number of times first hand, I have to say I am now in full agreement with Mr Brooks assertions.  Small teams are massively more productive and produce higher quality work, and both of these metrics degrade almost linearly as more people are added to the team.

As a principle, I now believe that software teams should always be kept as small as humanly possible. Adding a new developer to the team should always be the absolute last resort if we want to speed up delivery.

Why do I say this?

Communication Overhead

Well, there’s Brooks central idea that communication paths do not scale linearly as we add more resources into the team.  With 2 developers there is 1 path for communication, with 3 developers there are 3, with 5 there are 10 etc.  Once you get up to 12, you’ll have 66 distinct communication paths between the team members.

The funny thing is, even 12 developers isn’t a lot – it’s probably around the average size of a software team!

I have observed how, very quickly, the weight of this communication overhead starts to harm the team.  Misunderstandings start to form, knowledge is not shared well as well as it should be, and key man syndrome starts to form.  To combat this, we have to introduce meetings and documentation and Wikis and emails.  We need more planning and updates and accountability and sub-dividing of work units.  All of this takes work in itself, and no solutions are perfect despite the ongoing effort and cost that they imply.  A smaller, tight knit team can simply avoid all of this overhead.

Resource Constrained Productivity

I have also observed how smaller teams can also be substantially more productive in terms of output.

When you are resource contrained, you tend to be pragmatic and focus solely on what’s important – the things you have to do to which add the most value, get the project, done and get the product working and shipped.

Teams that are small and slightly resource constrained work in this mode, and a lot of unnecessary fluff and overhead is not carried out as a result, with very little cost to the project or the wider business.

Creativity

Smaller teams are simply much more creative.

As is well documented, designing or creating anything by committee leads to the lowest common denominator as ambitious ideas are chipped away at and rounded off in the name of consensus.

As even the most mundane software system will involve a huge creative element, this loss of creativity can be a real drag on the overall health and quality of the project and end product.

Small teams are particularly important when kicking off a software project.  I think that you should always start projects with a small team of 2-3 people locked in a room with a mandate for blue-sky thinking. Let them come up with some crazy ideas and mock them up and get them in front of users for feedback.

Steve Wozniak, co-founder of Apple takes this a step further:

Most inventors and engineers I’ve met are like me — they’re shy and they live in their heads. They’re almost like artists. In fact, the very best of them are artists. And artists work best alone — best outside of corporate environments, best where they can control an invention’s design without a lot of other people designing it for marketing or some other committee. I don’t believe anything really revolutionary has ever been invented by committee… I’m going to give you some advice that might be hard to take. That advice is: Work alone… Not on a committee. Not on a team.”

I think in a typical organisation we need more than the lone ranger. You need this creative artist type, but you also need someone there to manage the organisation, the corporate politics and the stakeholders.  Optionally, you may also wish to add a ‘completer-finisher’ to round off the worst tendancies of the ideas guy and keep him grounded.

Work Smarter…

Keeping development teams small does not mean that staff should be maxed out and permanently overworked and burnt out.

In parallel, the resource constraint means that we should be striving to keep the software system as simple as possible, and allowing the team to ‘get more done with less’ through techniques such as agile automated testing, continuous deployments, and clean coding.

Each time the team is at maximum capacity, we should seek to work smarter rather than throw more bodies at the problem.  Keep the team, small, agile, nimble and better work will result.

One of the biggest leaps forward in my productivity and enjoyment, as a developer was when I learnt about and adopted Maven for my Java projects.

For various reasons, I was relatively late to the Maven party, but nowadays I could not imagine developing enterprise Java projects without it.  And when I am working with other languages and toolsets, I’ll always look to adopt similar tools to automate my workflows.

If you aren’t familiar, Maven is a tool for managing the lifecycle of a Java project – think compilation, testing, packaging, release, deployment, dependency management etc.  All of the stuff that goes around the edge of simply writing code.

This can be incredibly useful time saver, helping to automate builds and making them much more repeatable.  However, I am also struck by the way the Maven workflow enforces good habis

• Automation – The number one win with Maven is the automation it gives you. Without writing a line of code or configuration, you’ll be able to do things such as compile, build project artifacts, run tests etc by running a single command. This will repeatable on your machine, your colleagues machines, and on your build and CI servers.
• Convention Over Configuration – Ant will have gotten you some of this way
• Testing – Maven has a specific phase explicitly for running tests. If you don’t have unit tests, Maven will make you feel guilty each time you see that phase whizz by. If you do have them, they better be up to scratch because by default, it will fail the build if your tests fail. Maven really drills home the fact that testing and verification is part of what we are doing.
• Separate Integration Testing – Maven specifically seperates out the integration testing phase of the lifecycle, and gives you the tools to perform good integration testing – e.g. by bringing up and tearing down environments cleanly.
• Small, loosely coupled Components – Maven encourages you to produce small defined components which are bundled into separate artefacts. These are then reusable.
• Consistency – Maven encourages consistency and convention over configuration. This is great when moving between companies and projects, but also encourages consistency within the project team. It reduces the special cases that people need to know about how things are deployed.
• Documentation – Maven incorporates a site feature which encourages documentation. As we know, developers don’t like writing documentation but the site can give us information on the code base, and it can bring in reports such as code coverage.
• Release Management – It will give some structure to your releases. Artifacts are versioned. The snapshot system helps you to. You can define dependencies.
• Code Quality Plugins – It’s really simple to integrate Maven with code quality tools such as coverage and static analysis tools, and possibly route these into your site. Teams who use

Now Maven has some critics and I don’t want to be a tool zealot, but I’m frankly baffled that Maven cannot deliver value to the average Java team who are using Ant or are working in an even more ad-hoc style.

With so many tools, techniques, and industry trends crossing over into the mainstream, 2012 is shaping up to be one of the most interesting years in software development in a long while.  The landscape is literally changing underneath us developers at the moment.

Cloud Computing

Businesses are finally allowing their applications and data out of the door and onto external infrastructure such as Amazon and EC2.

Cloud, with it’s ability to scale up and down has definite implications in how we design and build our applications for horizontal scalability.

A more agile infrastructure where developers and operations staff can requisition environments and servers as needed is indeed a brave new world.

Platform As A Service

PAAS is a natural evolution of cloud.

Though we can simply buy a cloud server on which to host our applications, it may also make sense to buy higher level platforms off the shelf that incorporate infrastructure such as application servers, databases, and messaging.

By using off the shelf platforms like this, this further reduces the plumbing code that we have to write and manage, allowing developers to get started faster and work at a higher level of abstraction.

DevOps

With cloud and PAAS, people are talking about the merger of Development and Operations into a new field.

DevOps really places a label on a set of tasks that developers and system adminstrators may have previously found themselves doing, and introduced the notion of using code and scripts to automate these tasks & treat operations as a more repeatable activity.

NoSQL

NoSQL is reaching widespread adoption now, giving us some realistic choices of production proven data stores for our applications that offer real benefits over RDBMS.

We can choose databases that are tailored to the particular requirements of the application and store a database.

NoSQL datbases are also often inherently horizontally scalable.

Mobile Devices

Any new web application has to consider the range of devices that are accessing the sytem and code to support them. Mobile and tablet based access to the web is growing at an enormous pace.

This presents challenges to developers and designers who need to may need to explicitly render mobile views, support a range of different devices, and provide a good user experience to boot.

Restful APIs

APIs and access to data continues to grow unabated. Most people signing up

Again in relation to mobile and diverge in frontends, there’s a requirement to seperate business logic and expose APIs.

Any modern web application needs to explicitly consider exposing the data as an API and JSON and REST are the weapons of choice.

HTML 5

HTML 5 represents a huge toolkit in helping us to build modern web applications

Gelocation and Offline storage in the mobile setting

Responsive Frontend

A few years ago, most people would have been accessing the web and web based software via their laptops or desktop PCs.

Nowadays, they’ll be accessing via.