Postmortems in Software Development

August 1st 2012, 9:30 a.m. on the NYSE open. Knight Capital flips the switch on a new release—and instantly starts bleeding money. One of eight servers was still running an old flag, so an ancient feature called “Power Peg” re-awakened and sprayed ridiculous orders across 148 stocks. Forty-five minutes, four million trades, and $440 million in losses later, the firm needed a bailout to stay alive. (Side note: I’ve never managed to dig up their internal post-mortem. If you ever find it, send it my way.)

Same pattern, different decade. In 1990 a single misplaced break; in AT&T’s switch software created a reboot cascade that knocked out long-distance service for roughly 60,000 people and cost the company about $60 million. Flights were delayed, pagers were silent, the whole nine yards. One character, nationwide havoc. I’d love to pretend we’ve evolved past that, but we both know we haven’t.

Those two disasters would make fascinating public post-mortems—sadly I can’t find any. I’m guessing a small army sifted through packet captures for months, but the write-ups never saw daylight.

I’ll keep it smaller-scale. Over the years I’ve had projects skid off the runway and a multi-hour datacenter failure that shaved a few months off my life. Watching a major incident unfold in real time feels like slow-motion physics: you can see the collision coming but you still can’t reach the brake.

The worst for me? A high-profile product launch—press releases queued, clients on livestream, champagne somewhere on ice. Ten minutes in, payments started hard-failing because of a certificate mismatch that hadn’t shown up in staging (classic). We scrambled, fixed it, owned it, survived. Still hurt.

We pulled an all-hands, combed logs, published a timeline, and added checks so the same class of bug can’t ship again. The relief was real, but so was the hangover. (I’m talking adrenaline, not champagne.)

🏄 Post-mortems are the autopsies that keep the next patient alive. Skip them and you rerun the same crash.

Aviation and bridge engineering treat failure analysis as sacred ritual. Black boxes, FDR read-outs, load tests on snapped beams—the works. That rigor is why commercial flying feels routine and why bridges rarely fold under rush-hour traffic.

Our servers don’t carry passengers, but they do process paychecks, medical data, and the odd rocket launch. Consequences may be less visible, yet they’re expensive enough to bankrupt a company before lunch.

So yes, we should borrow a page from aviation—but with nuance. I’ve worked in teams where the post-mortem template turned into bureaucratic theatre: 20 pages, zero insight. The trick is honesty over ceremony. Root-cause analysis when something breaks, near-miss reviews when something almost breaks, and—this one’s underrated—“brag docs” when a risky change doesn’t break. (I stole that idea from someone on my team who swears it keeps alert fatigue in check.)

Perspective matters too. After an outage everyone suddenly “knew it was going to fail.” That hindsight bias will poison your analysis if you let it. Ask what was knowable at the time, not what’s obvious afterwards. I get this wrong more often than I want to admit.

The only unforgivable mistake is failing to learn. Everything else is tuition.

Alright—how do we set up for the next mess?

Get your team ready

“Psychological safety” sounds like HR jargon, but the test is simple: can an engineer push a bad config, say “I broke prod,” and still feel welcome at Monday stand-up? If not, you’ll never get the real story during a crisis.

Mistakes happen—occasionally by design. I’ve seen teams run controlled failure drills (“GameDays”) every sprint. It normalises error and surfaces weak spots long before customers notice. Caveat: overdo it and you burn people out. I could be wrong, but quarterly exercises feel about right for a midsize team.

The Google SRE book nails the culture shift away from blame. Quick excerpt:

VP Ash: Someone must have known this was risky—why didn’t you listen?
SRE Dana: Everyone had good intent. Maybe let’s ask if there were warning signs we missed and why.

When the pager screams you want problem-solvers, not self-preservation instincts. If your team hides mistakes, the incident will outlast your runway.

Enjoyed the read? Join a growing community of more than 2,500 (🤯) future CTOs.

Email address

Subscribe

Free, no spam, unsubscribe at any time.

How to gauge safety? Start with your own reaction. If I curse at a junior for a minor slip, I’ve signalled that survival > learning. After that, watch hallway chatter, read PR comments, ask in 1:1s whether folks feel comfortable flagging brittle code. The answers are rarely subtle.

Then rehearse. Table-top the ugly scenarios: leaked database, ransomware, production schema drop. Talk through who calls legal, who pauses ad spend, who rotates secrets. It feels theatrical—until it isn’t.

Before sh*t hits the fan

You need a runbook you can follow at 3 a.m. with half a coffee in your system. Not a PDF buried in Confluence. Something discoverable—pager message, Slack bookmark, whatever works.

If you’re ISO-27001 aligned you likely have a risk register already. Good start. But keep it alive: new third-party dependency? Update the register. New data residency law? Same story. I’ve lost count of audits that failed because the “living document” hadn’t been touched since 2019.

Simplified plan:

1. Enumerate plausible incidents.
2. Define sequence of actions per incident.
3. Assign on-call roles (incident commander, comms lead, tech lead).
4. Spell out “done” criteria so you know when you’re clear.

The role split looks excessive until you’re juggling exec calls, rollback commands, and social-media blow-ups in parallel. Borrow the “incident commander” pattern from firefighting; it forces single-threaded decision flow and reduces cross-talk.

I binge post-mortems the way some people binge Netflix—there’s a nice public list. Reading through other teams’ disasters costs nothing and occasionally saves you from repeating one.

On communication: default to candour. If personal data is at risk, call your DPO and, if law requires, law enforcement—immediately. Trying to massage the timeline usually backfires and drags regulators into the mix.

After sh*t hits the fan

The outage is over, alerts are green, adrenaline drops. Now the real work starts. You need the narrative before rumours fill the vacuum.

Form a small squad—anyone paged during the incident plus a neutral facilitator. Collect artefacts: deploy hashes, CloudTrail logs, Slack threads, customer tickets. I aim for a 48-hour draft while memories are fresh. Faster is better, but people need sleep first.

Keep drafts collaborative. A shared doc beats a heroic single author because it catches memory gaps and prevents hindsight bias. I’ve seen a post-mortem shift root cause from “buggy code” to “ambiguous runbook step” once the operations folks annotated the timeline.

Google’s template is solid. My stripped-down version usually includes:

Title & Owners
Date/Time
Executive Summary (2-3 sentences)
Impact (customers, revenue, SLAs)
Timeline
Root Cause(s)
Short-Term Fixes
Long-Term Actions (owners + due dates)

Notice there’s no “culprit” section.

Rules of thumb:

1. Get below the surface. “Database error” is not root cause. Was it schema drift? Connection pool starvation? Mis-configured AZ failover?
2. Write concrete actions. “Improve monitoring” is fluff; “Add 95th-percentile latency alert in Prometheus” is testable.
3. Sequence matters. Quick guardrails first (feature flag, extra alert). Ambitious refactor later, maybe.
4. Question assumptions. Just because the cluster scaled yesterday doesn’t mean it will tomorrow. Probe the “obviously safe” areas.

Common Mistakes

Things that make any incident worse:

1. Public finger-pointing.
2. Hand-wavy customer updates.
3. Dodging responsibility.
4. Quietly deleting evidence or logs.

I once read a post-mortem so drenched in acronyms that even I had to open Google. No apology, no plain-English explanation, just “technical difficulties.” Customers remember that stuff—and they screenshot.

Assume your post-mortem might end up on Hacker News. Write accordingly.

For a deeper dive, the “Post-mortem Culture” workbook is a fun rabbit hole.

Conclusion

Incidents are inevitable. How you react—technically and emotionally—decides whether you end up stronger or merely bruised. Keep the comms clear, the analysis honest, and the fixes actionable. That’s pretty much the whole playbook.

Missed something? Ping me.

---

Other Newsletter Issues:

• Habits of great software engineers
• Networking as an introvert CTO
• Database Migrations
• Proper Documentation