Development at crisis speed
The foundations of modern web development are replete with stories of speed. We laugh at the tales of plodding, dinosaur-like waterfall models followed by corporates, we admire the diligence and focus inherent in the crushing furnace of start-ups.
On Feb 22, 2011 and in the days afterwards, I learned that there was in fact another development speed, one that made a start-up look like a leisure cruise in comparison.
On that day an earthquake rocked the city of Christchurch and I, along with many other web developers and designers, tried to figure out how to help. I joined the group that eventually became eq.org.nz, the largest online volunteer effort dedicated to finding and organising information about the quake for rescuers and victims alike.
We had some software to base off, but it needed extensive modifications. We had other problems for which we had no existing tools. Our major assets were a group of skilled, highly motivated individuals and all the services of the global internet.
We also had very little time. Every second that ticked by rendered information less useful, deprived us of volunteers and left those assisting and affected with fewer tools to help.
The team built software at crisis-speed by necessity but in hindsight it was an utterly alien place. Where a startup focuses limited development resource with razor efficiency, a crisis works with plentiful development muscle but minimal coordination.
It was the software equivalent of a shotgun. Loosely coupled individuals and teams built whatever they felt (or had been convinced) was best. Thousands of lines of code that would never see use were written, useful tools and changes were adopted, and those that missed the mark fell into disuse.
We built and rebuilt mapping tools, twitter filters, finders and matchers of every kind, groups coalesced to solve a particular problem then fell apart back to other tasks.
We hit problems. Many of the best practices for start-ups no longer served us well in such a chaotic environment. Having a large codebase tracked within github became a bottleneck as those with access struggled to maintain a dev/release pattern in the face of a barrage of changes from every direction. The thought of a weekly, or even daily production release cycle was laughable - we were "releasing" just as fast as we could.
Every shortcut that we could think of was taken. Test suites were ignored, debugging was performed on production, critical changes bypassed the version repo entirely in order to avoid the bottleneck, uncoupled tools were built and deployed on completely different hosts.
For some time it wasn't even clear to us that we were in a different place. It was difficult to recognise that we were ditching standard start-up patterns not because we were lazy but because they were fundamentally incompatible with what we were attempting to achieve.
As the development effort eventually wound down, I attempted to learn from our experience and a small number of core lessons became apparent:
1. Tools must be small and loosely coupled. You cannot have everything going through one big check/review process, individual tools need to be adaptable at their own pace, by the developers who are owning and making the changes, even if this results in duplication of code.
2. Communications between devs need to be crystal clear. Pull requests without context, local changes without comments etc waste valuable time while someone tries to figure out what is going on.
3. At least one person per shift needs to have a job that consists of almost nothing else but knowing what is going on. This person does not have time to be coding, they need to be the go-to guy for questions, and they need to maintain a log and perform hand-off after their shift.
These lessons are important for the development of software to be used in disasters. There is no value in creating large monolithic apps - when the crisis strikes the size of the app will prevent rapid adaption. An ecosystem of cooperating applications, clearly documented, secure by default and each versioned separately will win out on the day.
We also need to investigate methodologies appropriate to the crisis environment. How do we best retain reliability and security when speed is of the absolute essence? how can we take patches built in such an environment and return them to the upstream knowing the priorities that built them? what tools and processes do we need to help development teams self-organise when every second counts?
As software becomes the key to organising and streamlining relief efforts, the answers to these questions will become of critical importance.
