Guide
Why production incidents keep coming back
By Campion Fellin · Published · Updated
Why does the same production incident keep happening, and how do you stop it?
Usually because the fix addressed the symptom, not the cause. Restarting a service, adding capacity, or patching the line that threw the error makes the alert stop, but the condition that triggered it is still there. Stopping repeats takes three things: enough observability to see what actually happened, protected time to trace the problem to its root, and a fix that is tested against the original failure. If your team is too busy firefighting to do that, an outside engineer focused only on the investigation is often the fastest way out.
Common reasons incidents repeat
- The fix was a restart, rollback, or capacity increase that relieved the symptom.
- Logs and metrics are not detailed enough to show what happened before the failure.
- The failure depends on timing, load, or data that is hard to reproduce outside production.
- Several small problems combine, so fixing any one of them looks like it worked for a while.
- No one has protected time to investigate once the immediate incident is over.
How to find the real cause
- Collect every past occurrence and look for what they have in common: time, load, data, deploys, or dependencies.
- Add the logging and metrics needed to see the moments before the next failure.
- Form a specific hypothesis and try to reproduce the failure on purpose.
- Keep asking why until you reach a condition you can change, not just an event you can observe.
Making the fix stick
- Verify the fix against the original failure, ideally by reproducing it first.
- Add a test or alert that would catch the same problem earlier next time.
- Write down what was found and why, so the next person does not start from zero.
- Watch for the next few occurrences of the original trigger to confirm the fix holds.
When to get outside help
If the same incident has returned several times, your team has likely already tried the obvious fixes. A senior engineer whose only job is the investigation, without on-call duties or feature work, can often make faster progress. They should leave behind the fix, the explanation, and the improved monitoring.
Frequently asked questions
What is a root cause analysis?
A structured investigation into why an incident happened, going past the immediate trigger to the underlying condition that allowed it. The output is a written explanation and a set of changes that prevent the same failure.
Are blameless postmortems worth the time?
Yes, when they lead to concrete changes. Focusing on how the system allowed the failure, rather than who made a mistake, makes people more willing to share what happened, which is what the investigation needs.