Metrics Driven Development

One of my main takeaways from day 1 of GOTO Copenhagen was the importance of using metrics to monitor what our applications are doing, how they are running and whether they are currently in some problematic or broken state.

Once we have good metrics and a good set of monitoring systems on top of them, we can be much more aggressive in pushing out changes. This style of monitoring gives us a very effective early warning system for bugs or breakages that have been introduced.

The problem is that internal application metrics can sometimes be hard to capture and move out into monitoring or graphing software.

Scraping events out of log files or polling them out of a database is sub optimal, and even coding them into the application more explicitly can be difficult if they were not accounted for during the early development phases.

A few speakers today made the point that generation of metrics should be included up front as part of the development of individual features. They could in fact be included in the definition of done, such that each feature is not considered complete unless appropriate metrics and monitoring are in place. By encouraging or enforcing metrics and monitoring in this way, we are likely to end up with a very rich insight into the application over time, at least if we are able to handle and dig into the data.

Etsy are often held up as an example of a team making heavy use of metrics in this style. Mike states in his presentation that metrics are a part of every feature.

Metrics driven development?

Today someone mentioned the idea of taking this further and using application metrics to really drive the features that we implement and how we implement them. Even though it sounds a little frivolous at first, I actually think there is a fair amount of potential in the idea.

For instance, imagine the scenario where we have been asked to add a new field to a form, perhaps a postcode on the sign up form of some web application. This sounds like an innocuous requirement that we would usually just add and not think twice about. However, think of all the metrics we could capture before and after the change if we were aggressive with application metrics and monitoring:

• Form completion attempts percentage without the new field
• Form completion attempts percentage with the new field
• Form completion success percentage without the new field
• Form completion success percentage with the new field
• Percentage of the time the new field is omitted
• Percentage of the time the new field is entered but is invalid
• Percentage of the time the new field is entered successfully

Metrics like this would give us great insight into whether the change was successful, whether it led to breakage, how it changed user behaviour and whether it is delivering net value to the business. The bottom few in the list could be used to track and improve the feature over time.

We could also make our metrics more granular so that we can pivot by, say, geography or mobile device to identify any problems in relation to those users. This would create huge volumes of data for an absolutely trivial field, but I could nonetheless see net benefit while the feature is being rolled out.

The point is that even in this trivial example, working in this style does actually sound like a very appealing and value driven way to deliver software:

• Our implementation decisions should be data driven. We should take the smallest possible step and measure the impact in order to maximise value delivery.
• We should only implement features if they deliver a net benefit. If we cannot measure some feature, how can we be sure it did provide a net benefit? Perhaps we should deliver something of more measurable value initially.
• If we cannot monitor some feature, how can we be sure it can reliably and consistently be deployed? Not being able to monitor and measure absolutely reduces the desire to deliver the feature. Perhaps we should deliver something we can more securely deliver first.

My point is this: in a data driven organisation, metrics and monitoring deserve to be brought forward in the process as a factor that heavily influences the development decisions we make. A new form of development, driven by X, has been born.

For the reliability side of the same coin, read how to do rollback well. For the cultural argument, read why DevOps matters to developers.