Start with the budget
An SLO of 99.9% over 30 days allows 0.1% of requests to fail. For a time-based view:
- Minutes in 30 days: 30 × 24 × 60 = 43,200
- Budget: 43,200 × 0.001 = 43.2 minutes of total failure
Burn rate is how fast you are consuming that budget, relative to the rate that would use exactly all of it by the end of the window.
burn rate = observed error ratio / (1 - SLO)With a 99.9% SLO, the allowed error ratio is 0.1%. An actual error ratio of 0.1% is a burn rate of 1, and you finish the month with nothing left. An actual error ratio of 1.44% is a burn rate of 14.4.
The time to exhaust the budget follows directly: 30 days divided by the burn rate. At a burn rate of 14.4, that's 720 h / 14.4 = 50 hours. At 1,000 (100% errors at 99.9%), it's 43.2 minutes.
Why simple threshold alerts fail
Consider the obvious rule: "alert if the error ratio exceeds 0.1% over 5 minutes." A blip of 30 seconds at 2% errors averages to 0.2% over 5 minutes and pages somebody, while consuming about 0.02% of a 30-day budget (0.5 min × 2% = 0.01 full-outage minutes, out of 43.2). Nobody needed to wake up for that.
Now the opposite: "alert if the error ratio exceeds 0.1% over 1 hour." A full outage fires within seconds, since 0.1% of an hour is only 3.6 seconds of total failure. But once fired, the alert can stay active for about an hour after the fix. And a slow leak at 0.09%, which uses 90% of the month's budget, never fires at all.
The way out is to set thresholds according to how much budget has been used.
Where 14.4 comes from
Decide: what budget loss is worth waking someone up for, and over what window? The SRE Workbook's answer is 2% of the 30-day budget in one hour.
burn rate = budget fraction × window of the SLO / alert window
= 0.02 × 720 h / 1 h
= 14.4The same formula gives the rest:
| Severity | Budget spent | Long window | Short window | Burn rate | Calculation | Error ratio at 99.9% |
|---|---|---|---|---|---|---|
| Page | 2% | 1 h | 5 m | 14.4 | 0.02 × 720 / 1 | 1.44% |
| Page | 5% | 6 h | 30 m | 6 | 0.05 × 720 / 6 | 0.6% |
| Ticket | 10% | 1 d | 2 h | 3 | 0.10 × 720 / 24 | 0.3% |
| Ticket | 10% | 3 d | 6 h | 1 | 0.10 × 720 / 72 | 0.1% |
These are defaults for a 30-day window. If you use 28 days, 0.02 × 672 / 1 = 13.44, so adjust. And you should change the percentages if your organisation's tolerance differs. What matters is that you can defend the choice.
What the short window is for
Each row has two windows, and both must exceed the threshold. The long window confirms that it is significant. The short window confirms it is still happening.
A worked example, assuming a 99.9% SLO: an incident causes 5% errors for 20 minutes, then gets fixed.
With only the 1-hour window. The threshold is 1.44%. After 20 minutes, the 1-hour average peaks at 5% × 20/60 = 1.67%, which fires. After the fix, the incident's errors stay in the window. The average stays at 1.67% until 40 minutes after the fix, and then the incident starts leaving the window. It falls below 1.44% when the remaining incident minutes satisfy 5% × m/60 < 1.44%, or m < 17.28, which happens about 43 minutes after the fix. So the alert keeps firing for 43 minutes after the problem is over.
With the 5-minute window added. After the fix, the 5-minute average falls below 1.44% once fewer than 1.44 minutes of incident remain in the window, about 3.6 minutes after the fix. The alert resolves on its own in under 4 minutes.
An alert that clears itself quickly means responders can trust its state, and the next incident isn't hidden behind one still firing. Using a short window of one twelfth of the long window is the Workbook's rule of thumb.
How quickly does it detect?
Time to detect, for the 1h row, at 99.9% and with an error ratio e:
detection time = (threshold / e) × 1 hour| Error ratio | Burn rate | Time to fire at 14.4 over 1 h | Budget used by then |
|---|---|---|---|
| 100% | 1,000 | 0.0144 h ≈ 52 s | 2% |
| 10% | 100 | 0.144 h ≈ 8.6 min | 2% |
| 2% | 20 | 0.72 h ≈ 43 min | 2% |
| 1% | 10 | Does not fire on this row | Caught by the 6 h row |
A hard outage fires in under a minute. A milder problem fires later, but with the same budget spent. That property, equal cost at the moment of alert, is what the method is designed for. It also means that a complete outage takes about a minute to page, which may be too slow for your most critical path. Add a separate fast symptom alert, or a synthetic probe, if so.
Writing the rules
Express this in any alerting system that can compute a ratio over a time window. The logic
for a request-based SLO with a 99.9% target is below. Here, error_ratio(w)
means failed requests over window w, divided by total requests over
w.
SLO target: 99.9%, so allowed = 0.001
page if error_ratio(1h) > 14.4 * allowed and error_ratio(5m) > 14.4 * allowed
page if error_ratio(6h) > 6 * allowed and error_ratio(30m) > 6 * allowed
ticket if error_ratio(1d) > 3 * allowed and error_ratio(2h) > 3 * allowed
ticket if error_ratio(3d) > 1 * allowed and error_ratio(6h) > 1 * allowedImplementation advice:
- Pre-compute the ratio for each window as a recorded series. Long-window calculations over raw high-cardinality data can be expensive.
- Sum across all pods before you divide. Don't average per-pod ratios, as a quiet pod with one failure would weigh as much as a busy one.
- Make sure missing data doesn't look like "no errors." If the numerator or denominator vanishes, you need a separate absence alert.
- Keep the SLO target in one place, as a parameter, and not as a value repeated in each rule.
Latency SLOs work the same way
Turn latency into a good/bad event: "a request is good if it completes in under 400 ms." The
SLO becomes "99% of requests are good," and error_ratio becomes the share of
requests that are too slow. A histogram bucket boundary at exactly your threshold helps, so
that you're not interpolating. Note that the budget here is 1%, so 14.4 × 1% = 14.4% of
requests being slow is your fast-burn threshold. Burn rates above 100 are impossible with
this SLO, since no more than 100% of requests can be slow.
That points to a related limit: at a 99% target, a burn rate of 14.4 is still reachable, but at a 90% SLO the maximum burn rate is only 10, so the 14.4 row can never fire. Very loose SLOs need their own thresholds.
Low traffic
Burn-rate alerts assume there are enough events for the ratio to mean something. Take a service with 10 requests per hour on 99.9%: a single failure makes the 1-hour ratio 10% and pages at once, though it may be a user's typo reaching a bug. Options, in rough order of preference:
- Add synthetic traffic. It adds a steady stream of events, though it doesn't test everything real users do.
- Group services. Combine small related services into one SLO. A failure anywhere in the login flow counts together.
- Require a minimum count. Add a condition such as "and at least 20 failures in the window."
- Use a looser SLO or a longer window. Be honest that a very low-volume service can't support three nines statistically. With 10 requests per hour, you have 7,200 requests per 30 days, and a 99.9% target allows only 7.2 failures.
Before you turn on paging
- Check last quarter. Replay rules against history. How many times would they have fired? Did those correspond to real incidents?
- Start as tickets. Run them silently or as tickets for a few weeks.
- Check the SLI itself. If health checks or bots dominate the request count, the ratio hides real user errors. Filter them out.
- Check the measurement point. A load balancer sees failures that the service never logs, such as connection failures, so it often makes a better measurement point.
- Attach a runbook. Every page should link to what to do. Pair each alert with a dashboard built on the golden signals.
- Agree on a policy. Alerts tell you about spending. A written error budget policy says what happens when you've spent too much.
How Wrentrix handles it
Wrentrix's SLO feature tracks error budgets and uses 4 burn-rate window pairs for multi-window alerts that resolve themselves. Its problem engine groups related signals by topology into one incident, and "explain this screen" is available on any SLO if you want a local model to describe what you are looking at.
Frequently asked questions
What is a burn rate in SLO terms?
It is how fast you are consuming your error budget, relative to the rate that would use it up exactly by the end of the SLO window. A burn rate of 1 uses the whole budget in exactly one window. At a burn rate of 14.4 on a 30-day SLO, the budget lasts 50 hours.
What does 14.4 mean in burn-rate alerting?
It is the burn rate at which 2% of a 30-day error budget gets used in one hour: 0.02 × 720 hours / 1 hour. The SRE Workbook suggests paging at that level. At a 99.9% SLO, that corresponds to an error ratio of 1.44%.
Why do multi-window burn-rate alerts use two windows?
The long window shows that enough budget has been spent to matter, while the short window shows the problem is still occurring. Both must exceed the threshold to fire. That lets the alert resolve shortly after the fix instead of staying active until the incident rolls out of the long window.
How do burn-rate alerts work for low-traffic services?
They can be jumpy, since one failure can create a large error ratio. Common fixes are adding synthetic traffic, grouping related services into one SLO, requiring a minimum failure count and using a looser target. Pick the option that still reflects what users experience.
Do I need four burn-rate alert pairs?
Not always. Many teams start with the two page rows and add one or both ticket rows once those are stable. The ticket rows catch slow burns that the page rows miss, so without them, a steady low-level failure can use your budget unnoticed.