3 min read

Building Chowdeck: The Night I Broke Production

Source: http://x.com/i/article/2047632367691423744

One evening, 4 months into my journey at

@chowdeck, I broke production and crashed the app.

For at least 5 minutes, customers could not place orders. That was at least 5 minutes' worth of revenue gone. And what was worse? I had no idea what I had done until the next morning.

Earlier in the week, I had written a feature to optimize how we calculated delivery distances between vendors and a customer’s address. Everything looked good on paper, and I had tested extensively on the dev environment. Or so I thought…

In reality, I had failed to cater for edge cases. My code was written as if the data and the situation would be perfect for every customer. What if there were no vendors around the customer? What if a vendor had no city attached to it? As I mentioned in

my previous post about joining Chowdeck, this was my first experience building something that went live and served thousands (and later, millions) of users immediately. And while it is exhilarating as a software dev to see your features being used instantly in real-time, it also means that a single bug could shut down operations in the blink of an eye and cost the organization money.

I knew this theoretically. But it’s one thing to know something, and it’s another to experience it first-hand for the very first time. So on that fateful evening, some minutes after my PR got merged, I naively clocked out for the day.

The next day arrived, and I had a call scheduled with the CEO. I joined the call feeling perky and got the biggest shock of my life: I had brought down the entire operation last night. Thankfully, as soon as complaints started rolling in from customers, the PR was immediately reverted, and things were restored to normal.

When I dropped from the call, I felt very embarrassed and disappointed in myself. Not because of the stern reprimand I had received, but the gut-wrenching realization that I had done a poor job. As an overachiever, it’s one of the worst feelings ever. When you’ve rarely failed, and you encounter failure or what looks like it, your first instinct is to shrink away from it. But you rarely learn from success. “Failure” is where the actual life-changing lessons are.

So looking back, I am grateful for that experience, as out of it came several key principles that have come to guide and form the basis of my software engineering career and excellence:

  1. I never deploy customer-facing features during peak periods. In fact, the best deployment time is almost always at midnight, where the impact of a downtime is as minimal as possible.
  2. Once deployment is complete and the feature is live, I immediately test it on all my devices. Are existing features working correctly on all platforms? Is anything broken? Is the new feature working as expected?
  3. At the same time, I’m monitoring the error logs and reports like a hawk for at least 1 hour after deployment. Is there any error being thrown? If there is, I spot it way before the first complaint comes in.
  4. I consistently check on the feature weeks and months after. Is a user using the feature in ways I never expected or prepared for?
  5. I started introducing a feature flag (on/off switch) into most of my features and making them backward compatible. If something goes wrong, instead of waiting for the PR to be reverted and the deployment cycle to restore the system to normal, the feature can simply be toggled off, saving us crucial minutes in terms of revenue and user experience.

These instilled in me the twin skills of ownership and responsibility as I started owning the features and systems I built end-to-end: from inception to testing to deployment to monitoring to continuous improvements. Most times, I knew when anything was wrong with my features before anybody else.

As Chowdeck grew, we introduced even more standard safety measures to prevent bugs going into production, such as a live replica of the production environment accessible only to internal customers. If all went well there, there was a 99% chance your feature wouldn’t break anything when eventually deployed to the main production environment.

But the one I love the most? As an engineer, your feature cannot go live unless you explicitly confirm that you’re ready and available to monitor it conscientiously.

Femi says every great engineer has broken production at least once in their lifetime. Have you ever? How did you feel? And what was the biggest lesson you took from that experience?

**#BuildingChowdeck**