How to reduce MTTR without just working faster

First, be careful what the number means

"MTTR" expands to at least four things: mean time to repair, recover, respond and resolve. Teams routinely compare figures with different definitions. Choose a definition, write it down, and record the start and end points you use. A sensible default is the time from when user impact began until user impact ended.

The mean itself also misleads, because incident durations are heavily skewed. Consider this example of ten incidents, with nine lasting 30 minutes and one lasting 8 hours:

Total = (9 × 30) + 480 = 750 minutes
Mean  = 750 ÷ 10 = 75 minutes
Median = 30 minutes

The mean is 2.5 times what a typical incident looks like, and a single event controls it. Next quarter, without that one outlier, MTTR "improves" to 30 minutes with no change in your practices. Conversely, a real improvement can vanish behind one bad night. Published SRE research has made the same case: at typical incident counts, MTTR is a poor signal of whether things are improving.

So report the median and a high percentile alongside the mean, and report the count too. Then analyse phases, because that's where the actions are.

Split the timeline into phases

Here is an example incident, rebuilt from its timestamps:

(Shares are rounded, so they add up to 101%.)

Do this for your last 20 incidents. A spreadsheet is enough. The source data comes from alert history, chat logs and deployment records. Usually one phase takes an outsized share, and it often isn't the one that people assumed. A team convinced that it needs quicker alerts might discover that detection takes 6 minutes and diagnosis takes 38.

Detection: stop relying on users

If users or the service desk report incidents before monitoring does, fix that first. Count how many of your incidents were first reported by a person.

Symptom-based alerts catch failures that you didn't anticipate, while cause-based ones only catch those you did. Multi-window burn-rate alerts strike a good balance: fast for severe outages and slower for gradual burns. Add synthetic probes for critical paths, because they find failures during quiet periods, when no real traffic exists to produce errors. Short windows speed up detection but create noise, which is why alert quality belongs in this discussion.

Triage: the wasted 14 minutes

Triage time is largely organisational. Common culprits:

• Pager noise. If most pages can be ignored, the real ones get slower responses. Run the audit in the alert fatigue guide.
• Unknown ownership. Maintain a service-to-team mapping that the alert system can read. "Who owns this?" shouldn't be a question in an incident.
• Serial escalation. The alert goes to a generalist, who pages another team, who pages a third. Each handoff repeats context and costs time. Route by the affected service, and let responders pull in help directly.
• No declared incident. Agree on simple criteria for declaring one, and make declaring it cheap. Assign an incident lead, who coordinates rather than debugs, and keep one channel for communication. On larger incidents, separating coordination from hands-on work usually pays for itself.

Diagnosis: where most of the time goes

Diagnosis is typically the largest phase, and also the hardest to improve by working harder. Three questions in order cut through most of it.

What changed? Check this first, since recent changes are a frequent cause, and a good answer can skip the rest of the investigation. The change correlation guide gives a routine, including why config edits slip past the usual checks.

Where is the problem? Use the dependency map to see whether upstream or downstream services started failing at the same moment. The earliest onset on the deepest dependency is a strong lead. Split the failing metric by version, pod, node and zone. A clean split often solves the problem by itself.

Why is this request slow? Pick a slow trace, and find the span taking the time. Then jump to that span's logs. Time often disappears in moving between tools: copying a timestamp from a chart, searching logs in another window, trying to work out which pod served the request. Keeping a trace_id in logs and using shared resource attributes such as service.name remove a lot of this manual joining. If you are starting from nothing, distributed tracing probably gives you the largest gain.

Watch for anchoring. Once a team has a theory, they tend to look only for evidence that supports it. Have the incident lead ask periodically: what would prove this wrong? If 20 minutes go by with no progress, name other possibilities out loud.

Mitigation: separate it from the fix

The most effective single change for many teams: mitigate first, understand later. Users do not care whether you know the root cause. If the last deployment looks suspicious and rollback is safe, do it now, while keeping the evidence. Diagnosis can continue without user impact and without time pressure.

For that to work, mitigation has to be routine and reliable:

Schema changes are often the thing blocking rollback. An expand-and-contract approach, where the new schema works with both old and new code for a release or two, keeps rollback possible.

Capture evidence before restarting. A restart may clear the fault and destroy the only information about it. Automating a thread dump or diagnostics capture on alert fires protects this data, and it is one of the safest automations available. See the troubleshooting guides for Java and .NET on what to collect.

Mitigation access is a real constraint in regulated environments, as a separation-of-duties control can leave the on-call engineer waiting for an approver at 3 a.m. Design a pre-approved emergency change process in advance with your risk team: define which actions are allowed, by whom, and how they are recorded. Don't wait until an incident to find out.

Verification: when are you really done?

Teams lose time here in two ways: declaring victory too early and reopening an hour later, or watching dashboards for an hour out of uncertainty. Define it beforehand. For example: the SLI has been back within the objective for 15 minutes, and no related alerts are firing. Use the same user-facing signal that detected the problem. CPU returning to normal does not prove that users are fine.

Practice, and learn from what happened

Incident response is a skill, and skills decay with disuse.

• Run game days. Break something in pre-production, and have the on-call engineer respond with real tools. You will find stale runbooks, missing permissions, and dashboards no one can locate. Measure the phases there too.
• Write blameless reviews. Ask what made it hard to detect, diagnose and mitigate. Timing per phase makes reviews concrete: "diagnosis took 38 minutes because the logs had no trace IDs" produces an action item. "Be more careful" doesn't.
• Track the actions. A review whose actions never get done is just a story. Prioritise the ones that would have shortened the longest phase.
• Keep runbooks close. Link them from the alert, and date them. A runbook unused for a year should be treated as suspect.

Don't optimise the number instead of the outcome

Once MTTR becomes a target, people can game it: closing incidents early, splitting them, not declaring them at all. Also, fewer, larger incidents can make MTTR look worse while reliability actually improves, since you've eliminated the short, easy ones. Pair it with SLO attainment, incident counts by severity, and how often users report problems first. An error budget policy gives you an agreed way of connecting these numbers to decisions.

Where Wrentrix helps

Wrentrix's help is mainly in the diagnose phase. It brings traces, logs, method-level timing and Kubernetes change history into one place. The problem engine groups signals by topology into one incident, and automatic RCA ranks candidates with a clickable evidence chain, using a model you host locally. For mitigation, actions can run through webhooks or Ansible AWX job templates, automatically or with approval. It won't fix unclear ownership or unpractised rollback, and those may be your bigger gaps.

Frequently asked questions

What is a good MTTR?

There is no universal value, because it depends on the definition, the severity mix and the type of system. Compare your own median and high percentile over time, split by severity and by phase. External benchmarks are rarely comparable, since teams measure from different start and end points.

How do you calculate MTTR?

Add up the duration of all incidents in a period, then divide by the number of incidents. Define the start and end clearly, for example from the start of user impact to the end of user impact. Report the median and a high percentile as well, since a single long incident can dominate the mean.

What is the difference between MTTA, MTTD and MTTR?

MTTD measures time to detect a problem, MTTA measures time for someone to acknowledge the alert, and MTTR measures time to restore service (or, depending on the definition, to repair or resolve). Tracking them separately shows you which phase is slow. MTTA is mostly about paging and staffing, while MTTR usually depends on diagnosis and mitigation.

What is the fastest way to reduce MTTR?

Measure phases across your last 20 or so incidents, then address whichever is the largest. For many teams this means checking recent changes first and rolling back sooner, since mitigation before full diagnosis cuts user impact straight away. Reliable rollback and a good change timeline are often cheap compared with new tooling.