Our on-call rotation before and after we started tracking MTTR

For two years we ran an on-call rotation for our own infrastructure with no formal tracking of how long incidents actually took to resolve, on the assumption that if nobody was complaining, it was working. When we finally started measuring mean time to resolution, the number was worse than anyone expected, and the reason was not what we guessed.

What we assumed the problem would be

We expected the data to show slow initial response — engineers not picking up pages quickly. It did not. Median time to acknowledge a page was under four minutes, which is good by any standard.

What the data actually showed

The time lost was almost entirely in diagnosis, not response: a median of 47 minutes from acknowledgment to identifying the actual cause, largely because our dashboards were organized by service, not by the kind of failure, and the on-call engineer frequently was not the person most familiar with the affected system.

What we changed

We built a small set of failure-mode runbooks (elevated latency, failed deploys, database connection exhaustion) that are the first thing an on-call engineer opens, regardless of which service is affected, rather than relying on service-specific dashboards as the starting point.

• Median time to diagnosis dropped from 47 minutes to 19 minutes over the following quarter.

• We now rotate who writes the post-incident review, not just who was on call, which spreads the diagnostic knowledge instead of concentrating it.

• We stopped treating 'nobody is complaining' as a proxy for 'this is working' — it only measures whether incidents are rare, not whether they are handled well.