Alert fatigue: audit your pager and cut the noise

Run the audit first

Before changing a threshold, export every page from the last 30 days, with alert name, start time, resolve time, and whether anyone did anything. Put it in a spreadsheet and label each one. Do this with the people who were on call, since they remember what the data cannot show.

Here is an example of what that can reveal. A team with 300 pages in 30 days averages 10 a day (300 ÷ 30). After labelling:

• 210 self-resolved inside 10 minutes (70%)
• 60 duplicated another page (20%)
• 30 needed action (10%)

Only 1 page in 10 deserved to wake someone. If you handle just the two noisy groups, the pager falls from 10 a day to 1 a day (30 ÷ 30) without losing a single page that led to action. Numbers like these are illustrative; yours will differ, and that is why the audit comes first.

Then sort by alert name. Pages tend to cluster heavily in a few rules, so fixing the top handful usually accomplishes more than a broad threshold review.

The SRE literature suggests a limit of about two incidents per on-call shift, because each real incident needs time for diagnosis, repair and follow-up. It's a useful sanity target, not a hard law. Track pages per shift and out-of-hours pages over time, since those two numbers are what burn people out.

The test for a page

A page interrupts a person. Every page should pass all of these:

1. Users are affected, or soon will be. The alert describes a symptom or an imminent, certain failure.
2. A human must act now. If it can wait until morning, it's a ticket.
3. A human can act. There is something the responder can do with the access they have.
4. It says what to do. It links to a runbook or at least names the first step.

The most common failure is CPU at 85% paging somebody at 3 a.m. while latency and errors are normal. No user is affected, and often nothing is wrong. Use cause-based signals like CPU, memory and queue depth on dashboards and for diagnosis, and page on the symptoms: latency, errors and availability as your users experience them. The golden signals guide sets out which signals to prefer.

There are exceptions for causes where failure is certain and the warning time is the whole point: a disk that fills in 4 hours at the present rate, or a certificate that expires in days with renewal known to be broken. Note how each one is a prediction with a deadline, rather than a static level.

Fixing flapping and self-resolving alerts

Alerts that fire and clear within minutes are usually the largest group. The mechanisms below are common to most alerting systems, though names differ. The example uses a generic rule format, so treat the field names and the metric as illustrative.

alert: CheckoutHighErrorRate
expression: error_ratio{service="checkout"} > 0.05
for: 10m
keep_firing_for: 15m
labels:
  severity: page
annotations:
  summary: "Checkout errors above 5% for 10 minutes"
  runbook: "https://runbooks.internal/checkout-errors"

**Pending duration (for).** The condition has to hold for this long before firing. It removes brief spikes, at the cost of slower detection. A blip that clears in 2 minutes and needs nothing from a human has no business paging anybody.

**Clear hysteresis (keep_firing_for).** The alert stays active for some time after the condition clears. This stops a metric bouncing around the threshold from producing a new page every few minutes. A separate, lower clear threshold has the same effect.

Rate windows. A one-minute rate on a low-traffic service is mostly noise. Two errors in a minute with 20 requests reads as 10%. Use longer windows at low volumes, and require a minimum request count so a quiet period at night can't trigger a page.

Ask why it flaps. A frequent flap may be a real intermittent fault. Damping it hides the signal, so open a ticket to look at it during working hours.

Grouping and suppressing duplicates

One failure commonly produces dozens of alerts: a dead node takes out pods, each with several rules, and each dependent service then reports errors. Three techniques help:

A routing sketch, again with generic field names:

route:
  group_by: [alertname, namespace, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match: { severity: page }
      receiver: payments-oncall
    - match: { severity: ticket }
      receiver: payments-queue

inhibit:
  - source: { alertname: NodeDown }
    target: { severity: warning }
    equal: [node]

Static inhibition rules only capture dependencies you thought to write down, and they go out of date as the architecture changes. Grouping by actual service topology addresses that, because the dependency data comes from traces rather than from a rule somebody wrote two years ago.

Be careful with suppressing too widely. An inhibition rule that mutes everything in a namespace can hide a second, unrelated failure. Scope with equal labels and review suppressions after incidents.

Replace static thresholds where they keep failing

SLO burn-rate alerts

Rather than alerting on a fixed error percentage, alert on how quickly you are spending your error budget. The SRE Workbook's multi-window recipe is widely used:

Both windows must exceed the burn rate, so the short one makes the alert reset soon after the problem ends. As a check on the arithmetic with a 30-day budget (720 hours): burning at 14.4 for 1 hour consumes 14.4 ÷ 720 = 2% of it. Short error bursts that do not threaten the objective no longer page anyone, while a serious outage still does in minutes. See SLO burn-rate alerts for the derivation.

Baselines for seasonal signals

Static thresholds are a poor match for traffic with a weekly rhythm. A level that's alarming at 4 a.m. Sunday may be normal at 11 a.m. Monday. Comparing against the same hour of the week handles this. Anomaly alerts generate noise of their own, though, so they need the same audit. A useful rule of thumb is to page only on anomalies in symptom signals, and treat the others as context for an investigation.

Keep it from growing back

Alert rules accumulate after every incident, as nobody wants to be the person who removed the alert that would have caught the next one. A few habits counter that:

• Give every alert an owner. An alert with no owning team goes to the queue for deletion.
• Review weekly. Spend 15 minutes on the previous week's pages at the on-call handover. Each noisy page gets a decision: fix, tune, delete or leave for now, with a reason.
• Make deletion cheap. Keep rules in Git, so a removed rule can be restored in seconds. Silence a doubtful alert for a month first, and if nobody misses it, remove it.
• Budget for follow-up. If on-call people have no time to fix what paged them, the same pages return every rotation.
• Test the alerts that remain. A rule that has never fired may be broken. Fire drills in a pre-production environment confirm the routing, the runbook link and the signal.

Mistakes that make things worse

• Raising all thresholds at once. You swap false alarms for missed incidents. Tune one rule at a time, using its history.
• Routing noise to a chat channel. A channel with hundreds of unread alerts is just a quieter pager that nobody reads. If no one acts on it, delete it.
• Adding AI summaries without pruning. A summary of 200 meaningless alerts is still 200 meaningless alerts. Remove the noise first.
• Measuring alert volume alone. Zero pages might mean a silent outage. Track missed incidents, meaning those first reported by users, alongside volume.

How Wrentrix approaches this

Wrentrix's problem engine groups signals by topology into one incident. SLOs use 4 burn-rate window pairs with multi-window alerts that resolve themselves, and hour-of-week baselines are learned from your own history. The alert wizard works with channels and runbooks, and alerts can carry AI-generated summaries from a local model. The audit above is still your job. Tooling cannot decide what deserves to wake your team.

Frequently asked questions

What is alert fatigue?

Alert fatigue is what happens when responders receive so many alerts, particularly false or non-actionable ones, that they start to ignore or delay their response to them. The danger is that real incidents get treated with the same indifference as the noise. It also wears people out and makes on-call rotations hard to staff.

How many alerts per on-call shift is too many?

The SRE literature commonly suggests no more than about two incidents per shift, since each one needs time to diagnose, repair and follow up. That is a target, not a rule, and an alert that needs no real work still costs an interruption. Track pages per shift and out-of-hours pages, and investigate any upward trend.

How do I reduce false positive alerts?

Export a month of alert history, label each page by whether it required action, and focus on the rules producing the most noise. Add pending durations, require a minimum traffic level, and move from cause-based thresholds to symptom or burn-rate alerts. Delete any alert that no one acts on.

Should we alert on CPU and memory usage?

Generally not as a page. High CPU alone doesn't indicate a user problem, so it belongs on dashboards and in tickets for capacity work. Page when a symptom appears or when exhaustion is both imminent and certain, such as a disk predicted to fill in a few hours.