Skip to content
Benjamin Wootton.Better Software
Software

How To Do Rollback Well

21 July 2012

By Benjamin Wootton

3 min read

I still see a lot of fear out there in the development and operations communities with regards to being able to rollback failed software deploys.

People talk about the difficulty of rolling back, the need to always be rolling forward in the case of error, and the difficulty in testing rollback procedures.

Having written software and been responsible for deploying that software in very high availability environments, this is not a world I have experienced in the slightest.

In my experience:

  • It has been easier to rollback to a known state than to roll forward;
  • It has been riskier to roll forward if a deploy is going badly than it was to roll backwards;
  • It is much easier to test and gain confidence in a rollback than in a roll forward.

I think a lot of the misconceptions people have come from the fact that the average development process simply does not give rollback enough importance and focus in release management.

This is a shame, as rolling back is potentially your best friend with regards to improving the robustness of systems and keeping customers happy.

With a good rollback you can simply hit the red button at the first sign of trouble and have the system back up and running while you look into the situation. It may be that you overreacted in pulling the release, but that is generally better than breaking something.

With that said, here are a few tips that I have found to work well.

Design for rollback

The most important step is to implement an architecture that supports the need to rollback. Componentised, service based architectures lend themselves well to this. Persistent message queues and asynchronous services allow you to bring components down for rollback without impacting the main user base. Work toward an architecture where your application can stay available while you are working on one half of the system.

Test rollback as thoroughly as release

Test rollbacks as thoroughly as your rollouts via the QA process. Throughout development and testing, attempt to get as much, or even more, confidence in your rollback as you have in your release. Foster a mindset that change is good, but the route back to the working system is much more important. People generally prefer working software over new features.

Design the rollbacks and roll forwards to both work idempotently. Ensure you can roll back a bad deploy and then roll it forward again when the time is right. Neither step should be destructive, as we should accept that rollback has a high degree of probability. Have your QA team explicitly test rollout, rollback and rollout again to gain confidence and experience in the process. Any observations or problems should be fed back, and the change should not be signed off until the rollback is of high quality.

Document the procedure

Document the rollback procedures. It is likely there will be a degree of stress involved if we need to roll back the production system, so take the time before the release to write up how to run the rollback process, what to check and what to do in potential failure situations. For example, if the database deploy fails, run script x.sql and check conditions a, b and c. If condition b has occurred, execute y.sql. Ten minutes of anticipation before the change can save a lot of time during a crisis.

Take small gradual steps in your releases. Release intermediate steps behind feature toggles so that we can slowly but surely gain confidence in the feature while keeping rollbacks to a minimum. Try to upgrade components individually rather than in parallel.

Have documented steps in place to assert that the rollback process has put the system back into its original good state. Something simple such as checking the net file size, running a diff or checking the number of database rows is a start. You will then want to back this up with as many automated and manual sanity tests as possible to ensure the rollback was correct.

None of the above comes for free. Each item takes time and effort, but in my experience it has always been worth it when moving the applications I am responsible for forward slowly, surely and with confidence.

For the wider argument, see why DevOps matters to developers and continuous delivery using cloud development tools.

Reference shelf

Sources and further reading for this note.