The incident you learn from is worth having

Nobody wants an incident on their watch. The instinct when something breaks is to fix it quietly, fast, and not make any noise about it. I get it. Nobody wants the spotlight at 3am because something they deployed three weeks ago just took down a data centre.

Here's the thing: in the last two years, we've filed over 1000 Incident Record Files (IRF) at Infobip. That's the visible record of things that went wrong - and more importantly, of things that got better because they went wrong. The engineers who show up in those IRFs as authors, not just as participants, are the ones whose systems get more reliable over time. Not because they have fewer incidents. Because they treat each one as material to work with.

This is about that. Not the theory of incident management - there's a whole space for that. This is about why the process matters, what the two-year view of our own IRFs actually shows, and one principle I want everyone to walk away with: an incident is not failure. Leaving the incident with just a bandaid is.

Observability: the foundation you build before the fire

You can't manage what you can't see. But there's a subtler version of this that gets engineers into trouble: you can have every tool in place and still not be watching.

One of the worst detection gaps I've experienced personally: a complete processing outage in one of our biggest data centres. Zero processed requests. For fifteen hours. I caught it accidentally over morning coffee when the graphs looked wrong. The monitoring alert had fired on time, overnight. It was sitting in a Slack channel with no escalation policy because it was still in testing. It was there. Nobody was looking.

This happens in subtler ways too. A P1 (priority 1 incident) from 2024: an alert for high memory usage fired during an incident. It was accurate. Nobody acted on it. The incident went on for 25 minutes before anyone formally declared it. Another P1 from 2025: the contributing cause in the IRF listed "lack of visibility plus missing alerts" for a specific processing layer. After the incident, four tasks were created to fix the monitoring. Those tasks didn't exist before the incident. They should have.

Perhaps most insidious: a 2024 P1 where a feature was silently discarding traffic as "normal expected behavior," with no metric that could detect it. The discard wasn't a bug - it was a design decision that nobody made observable. When something is wrong you want a signal. When something is working as designed but the design is wrong, you also want a signal.

The checklist is short. Ask yourself:

• Do you have meaningful metrics for your service's critical paths - not just that the service is running, but that it is actually doing what it's supposed to?
• Do the alerts on those metrics have escalation policies that reach a human at 3am?
• Are your runbooks current enough that the on-call engineer from a different team could execute them?
• Is there an on-call rotation, or is it "whoever happens to be around"?

If any of those is "not yet" or "sort of" - go fix it before your next incident, not during. Hope is not a strategy. Neither is assuming someone else is watching.

When it happens: report it

An incident is any failure that prevents clients from using our platform as intended. Not a bug with a workaround, not a UX rough edge - a real, client-impacting failure.

When you have one: type /incident in #incident-management. Describe what's happening. That single command from our internal Incident Management bot creates a dedicated thread, assigns an Incident Commander, opens the IRF task, and posts a pinned status summary to the main channel for stakeholders. 30 seconds of typing to avoid 30 minutes of "who owns this?" and "where are we tracking this?"

The Incident Commander (IC) is the single point of accountability for coordination. Their job isn't to fix the thing - it's to make sure the thing gets fixed. One person coordinating five teams moves faster than five teams coordinating themselves. The rule that took me a while to internalise: the more complex the incident, the less the IC should be troubleshooting. As complexity grows, coordination matters more than individual technical contribution.

The dual approach: stop the bleeding, then close the wound This is the part I most want to stick. Let me be direct about it.

When an incident is active and customers are impacted, your first job is not to fix it. Your first job is to stop the impact. Those are not the same thing.

Rollback. Restart. Route around it. Reduce the scope. Turn the feature off. Put it behind a flag. None of these fix the root cause. All of them can restore service in minutes while root cause investigation continues in the background. A well-timed restart that buys you two hours is better engineering than spending two hours looking for the perfect solution while your clients' traffic burns.

Some examples from our own incident history:

A P1 where a service was processing zero requests - not crashing, not logging errors, just silently dead. Everything looked healthy: SSH worked, the service was running, no OOM, no exceptions. Nine million messages queued in a retry backlog before anyone figured out why. The mitigation was to provision fresh instances on an older, known-good version of the service. Not the right long-term answer. The right immediate answer. The root cause - a library upgrade that silently changed how write failures propagate, causing threads to hang indefinitely - was identified and fixed the next day. Two separate timelines. Both necessary.

A P1 where a cron job disabled 40,000 users. A cache layer returned no data, and the job treated "no data" as "no recent activity" and disabled everyone's accounts. Mitigation: stop the job, re-enable the accounts via a database script. Duration: 90 minutes. Root cause: the cache was the wrong storage choice for a decision this consequential. Long-term fix: move to a database with transactional guarantees. The incident forced the architecture conversation that should have happened earlier. It's now tracked.

A P1 major DDoS - over a billion requests in under two hours against one endpoint. Major outage for our main portal. Mitigation: reroute DNS through a DDoS protection service that had been set up weeks earlier during a different incident but never deployed, kept in reserve. The preparation from a previous incident was the mitigation for this one. Duration to restore service: about 90 minutes. The mitigation itself introduced side effects - other services that depended on the original domain configuration broke, mobile SSL pinning broke, regional portal traffic broke. Every side effect was documented in the IRF and became a tracked follow-up. 30+ items across five teams. All linked. All with owners.

Now the cautionary example. We have an IRF in our system with "2nd occurrence" in the title. Think about what that means: the first incident was mitigated, documented, and the planned actions were written down. Then they didn't happen - or happened incompletely. Same incident, second time. That's not bad luck. That's the cost of a well-filed IRF with poorly-followed-through planned actions.

And the honest admission from our own post-incident review on a prolonged P1 from 2024: "This did fix the issue somewhat, but there's still a problem when threads are fully exhausted... In order to fix it long term, we need to have a better understanding." That sentence, written honestly in an IRF, is exactly what the process is for. Not to document that you solved it completely. To document accurately what you did, what it achieved, and what remains.

The principle:

An incident is not failure. Leaving the incident with just a bandaid is.

The planned actions in your IRF must become real, linked tasks with owners. Not "we should look at this." Not a sentence that expires the moment the incident tab is closed. A task. Tracked. If you're not going to do it, write why in the IRF - that's still useful information. But don't write planned actions you're not planning to act on.

Blamelessness: the culture that makes all of this work

Everything above only works if engineers actually report incidents, fill in IRFs honestly, and propose long-term fixes they might be asked to deliver. None of that happens in a blame culture.

The most well-written IRF I've seen recently was authored by the engineer who caused the incident. Their configuration, their mistake, first person, no deflection, no "circumstances led to." Entirely clear about what happened. The contributing cause? Legacy code with an ambiguous column type that had been in the system for years, with no validation on the configuration values and no staging test for the affected path. The fix: clean the legacy. Nobody blamed. System improved. That's blameless culture in action - and it's why the IRF is useful. If the author had been defensive, the IRF would have been vague, the root cause would have been obscured, and the fix would have been incomplete.

Another P1: the contributing cause section read: "the architecture allowed silent message discard as a normal expected behavior." That sentence is doing a lot of work. It's saying: this isn't about who made a bad decision. It's about a design that had a dangerous implicit assumption. The fix is architectural: whitelist-based discard policy, audit log, human review required for discard decisions. Systemic, not personal. Those are also the kinds of fixes that actually prevent recurrence.

The five whys is built for this. Keep asking why until you reach the system failure that made the human mistake possible, not just the human mistake itself. "The engineer made the wrong configuration choice" stops short. "The engineer made the wrong configuration choice because the configuration system accepts any value without validation, there's no staging equivalent of this environment, and the feature had never been load-tested with this combination of settings" - that's something you can act on.

Blameless does not mean unaccountable. It means accountability attaches to the system and to follow-through, not to individuals as scapegoats. The most accountable thing you can do after an incident is file an honest IRF with realistic planned actions and then actually do them.

The IRF: bureaucracy that actually pays

There's a quote I come back to regularly: "If you can't explain it simply, you don't understand it well enough." One of the hardest things I do in SRE is write the short summaries that give the full picture of each incident without anyone having to read through every Slack message. When I can't write a clean paragraph about an incident, it's not because the incident was too complex. It's because I don't understand it well enough yet, and I need to dig more.

The 72-hour deadline for the IRF exists for the same reason: memory fades, details drift, the context that was obvious during the incident becomes ambiguous a week later. File it while it's fresh. File it while the people involved remember which decision led to which outcome and in what order.

1000+ IRFs from the last few years is an enormous amount of institutional knowledge. Patterns emerge when you look across them: the same alert threshold firing too late for the same class of infrastructure. The same missing runbook for the same failure mode. The same "we didn't know who owned it" for services that haven't had an ownership review in two years. Those patterns only become visible from the data. They don't surface from a single incident, or from incidents that were quietly fixed and never documented.

The Weekly Incident Review - every incident from the prior week, reviewed chronologically by the SRE troubleshooter - is how that learning gets socialised. It's not a blame meeting. It's a retrospective: what worked, what didn't, what we're improving. The goal is not to assign fault. The goal is to find the one or two improvements that make the next incident less damaging than this one.

A final note on planned actions. Writing them into the IRF is necessary but not sufficient. Each planned action should have a linked task. If a planned action will take more than a sprint to address, it should have a parent epic. If a planned action turns out to be not worth doing - because the cost outweighs the risk, or a better solution emerged - write that in the IRF too. The IRF isn't just a record of the incident. It's a record of the reasoning.

What you can do today

This is an offer, not a lecture.

If your service doesn't have the observability coverage you want - metrics, alerts with escalation policies, runbooks, an on-call rotation - fix that before your next incident, not during it. Prioritise it the same way you'd prioritise a security vulnerability. It is one.

If you have an active incident: declare it. Name it, open a dedicated channel for it, get the right people in the room. The instinct to handle it quietly in a side thread is the instinct that turns a 30-minute incident into a 4-hour one. Coordination infrastructure isn't overhead. It's how you move faster.

If you're not sure whether you have an incident: declare it anyway and escalate. A false alarm costs you 20 minutes. Missing a real one costs your customers hours. The asymmetry is not close.

If you've resolved an incident: write the post-mortem while the memory is fresh. Not a paragraph. A real document - timeline, contributing causes, lessons, planned actions with owners. Then track the actions. If you write a post-mortem and don't create the follow-up tasks, you've documented a problem you're comfortable having again.

You can't make incidents stop happening. You can make your system measurably better every time one does. The post-mortems your team has already filed - whatever the number - are the proof it works. Use them. Add to them. Follow through on them.