GOTO Copenhagen: Day 1 Review

I am currently attending GOTO Copenhagen for the first time.

I decided to spend most of the first day in the DevOps and continuous delivery track, as this is the area that is piquing my interest most at the moment.

I have always had large operations components in my development roles, and I am really enjoying learning the best practices here in terms of closing the gap between development and operations, agile infrastructure and continuous delivery.

I will begin by summarising the talks that I heard today.

Patrick Debois: Using monitoring and metrics to learn in development

Patrick's talk began by discussing DevOps at a high level: shortening feedback cycles between development and operations, speeding up releases and infrastructure as code.

However, the meat of this talk was on a less discussed aspect of DevOps: capturing and using metrics to proactively monitor how our application is performing. These are traditionally operations tools that we can potentially open up back to development and the wider business to give joint ownership of the production application to a wider group.

There are clearly a vast number of tools in this space: Nagios, New Relic, StatsD, Munin, CollectD and Ganglia, among others.

Without being disparaging to the above tools, the problem is that monitoring tools are not currently up to scratch for the needs of DevOps. There is a #monitoringsucks hashtag floating on Twitter which is dedicated to describing the shortcomings of the current monitoring space: missing APIs, poor user interfaces, static configuration and a requirement for restarts. All of these add up to the fact that innovation is required if we are to use metrics and monitoring as a better communication tool between development and operations.

Yoav Abrahami: Going CI/CD, the Wix experience

Wix is a platform that allows people to design and host their own high quality websites. With 2 million live websites running against an old Flash designer, Wix recently had to rewrite the platform as HTML5 and also move toward continuous integration and continuous deployment in order to deliver faster and more reliably.

The team at Wix tries to deliver in really small steps: the minimal viable features that can go out. They empower developers to own features from start to finish and generally push features into production multiple times per day.

The system is very highly decomposed and loosely coupled, and they allow versions of components within the system to go out of line. The implication is that they maintain forward and backward compatibility for two or three versions. This sounds like a combinatorial nightmare from a testing and reliability perspective, but they put steps in place to reduce this.

Yoav raised the interesting point that continuous delivery is slightly at odds with SCRUM. Iterations are potentially too slow and standups become: what did we release yesterday, and what did we release today? This is an interesting question for the future: how process is impacted by this new style of continuous delivery.

He also made the point that doing continuous delivery required a whole company engagement in order to understand the implications, pros and cons. Going to see a company that had successfully applied it brought a sceptical management around to the idea.

Itai Hochman: Continuous deployment at Outbrain

I only caught the tail end of this talk, but picked up some interesting comments and ideas from the discussion.

For instance, Itai's organisation still had a QA team, but they had taken steps to ensure that the QA team were not viewed as gatekeepers and that developers were the ones with final responsibility for delivering working features. QA as a quality gatekeeper with sole responsibility for reliability is an anti pattern that we need to avoid in all software delivery teams.

Itai also discussed his preference for emergent architecture rather than having separate architects dictating architecture as in the past. I have heard this opinion many times in the agile world, but it is interesting how the role of the architect intersects with continuous delivery. The architect perhaps had a much greater role to play with big design up front than they do when delivering multiple times per day.

Jeffrey Fredrick: A leap from agile to DevOps

This was a very open and honest conversation about some of the challenges Jeffrey had found in his current organisation with regards to reliability and bringing operations in house for the first time.

One thing Jeffrey observed was how the development organisation had evolved and improved over time, but when they put the operations organisation into place they did not put any of this learning into place and started making the same mistakes all over again. Simple things such as source code control, iterative delivery, pairing and testing were becoming quite mature in the development world but were sporadically applied in the operations world.

The talk also discussed how they were using root cause analysis on production incidents to understand the root of the problems that had caused the incident. Over time, patterns in these analyses would come to light which offered real insight into underlying organisational problems and what was stopping them addressing those problems.

It sounds as though they had a lot of success in bringing the business around to the idea of DevOps. A lot of development teams and product owners are stuck in the mindset of delivering change, but we need to make the case for reliability and robustness and work with product management to get maintenance and monitoring factored in and given priority.

Michael Nygard: Disband the deployment army

The talk started with a humorous picture of around 40 people sweating in an office who had been involved in a deployment at 1am on a weekend.

Luckily this is an extreme example of a deployment army, but deployments and releases are often complex and slightly scary scenarios that involve far too many people. There is a vicious feedback loop at work here in that the harder releases are, the longer we put them off. Then they end up growing even bigger and more scary.

The meat of this talk was explaining how we can move away from this by implementing continuous or at least much more frequent deployments and releases.

If something is painful, do it more often.

Michael accepted that more frequent deployment could result in slightly more production incidents, but the idea is to reduce the mean time to identifying those issues and then resolving them, such that the cost is very small relative to the business benefits gained by frequent releases.

Most people would obviously say that zero production incidents should be the goal, but it is an interesting point that if we can quantify and put a price on production incidents, and then minimise and reduce the probability of them to negligible levels, perhaps these are a fair price to pay in order to remove the overhead of heavyweight releases.

Summary

This was a really interesting first day and I continue to be convinced of the value of DevOps and continuous delivery as I hear more real world experience.

I have heard the continuous delivery pitch a number of times now: feature flags, deploy from main, extensive acceptance tests and so on. This is a slightly predictable talk now. What is more interesting is how the organisations have fared in their transition to doing this, the cultural changes and the end result in terms of observed quality and ability to deliver. It is a self selecting group talking about this, but most of them seem to have had success with it and, dare I say, found it easier than you might imagine.

Monitoring and the idea of an immune system for the application is a slightly fresher idea with a lot of potential. I like the idea of using automated tests to build confidence before release, and then using suites of metrics after release to build confidence in the change. I agree that there is huge scope for innovation in the tools space, but as that comes, high quality application metrics could really be a line of defence almost as useful as tests and QA.

Read my notes from day two, or the standalone note on metrics driven development. The continuous delivery tips are a practical companion.