Golden signals, RED and USE

The three methods on one page

The golden signals come from the Site Reliability Engineering book. RED was described by Tom Wilkie as a simplification for microservices. USE comes from Brendan Gregg's performance analysis work. The easiest way to remember how they relate: RED is the golden signals without saturation, and USE is the view from the other side of the service, looking at the things it consumes.

A good rule is to alert on RED, and diagnose with USE. Users feel request errors and latency. They do not feel 90% CPU, unless something else is also going wrong.

How to apply them: a walk through an incident

Here is an invented incident showing how the methods hand over to one another.

**14:10, RED on checkout.** Duration p95 rises from 280 ms to 3.1 s. Rate is unchanged. Errors go from 0.05% to 1.2%. This is real and users are affected.

14:12, RED on its dependencies. pricing looks normal. orders-db-proxy shows client-side duration up by a factor of ten. So the problem is at or below the database path.

14:14, USE on the database host.

• CPU: utilization 45%, nothing there.
• Memory: utilization 70%, no swapping, nothing there.
• Disk: utilization 98%, average queue length 40 and rising. Found it.
• Disk errors: the kernel log is clean, so it is not hardware failure.

14:16, USE on the connection pool. Utilization 100%, with 60 threads waiting. That's a consequence, not a cause: slow disk means slow queries, which hold connections longer. For more on that pattern, see connection pool exhaustion.

14:18, what changed? A nightly reindex job, moved to the daytime by a config change, is saturating the disk.

RED located the failing path. USE named the resource. Neither on its own would have told the whole story.

The signals in detail

Latency and duration

Measure successes and failures separately. A failing endpoint that returns a 500 in 2 ms will make your overall latency look better during an outage. Track percentiles from histograms, not averages. Pay attention to these:

• A percentile computed on each pod can't be averaged across pods. You need to sum the histogram buckets and calculate from the total.
• Bucket boundaries limit your accuracy. If the buckets jump from 1 s to 5 s, a value reported as 3 s is only an interpolation.
• The p99 of a low-traffic endpoint in a 1-minute window may rest on a handful of requests.

Traffic and rate

Requests per second usually works, but choose what fits: messages consumed, active sessions, transactions. Traffic gives you context for everything else. A sudden drop to zero looks excellent on an error chart. Watch for unexpected falls as well as rises, and compare with the same hour last week, since most traffic is strongly seasonal.

Errors

Be specific about what counts as an error:

• HTTP 5xx is obvious, and 4xx usually isn't your fault. But a sudden flood of 401s after a deployment probably is.
• HTTP 200 with an error body doesn't get counted unless you do something about it.
• Slow responses beyond a set limit can count as errors for SLO purposes.
• Retries hide errors from users while still costing you capacity. Measure at both the client and the server.

Saturation

This is the most-misunderstood signal. Utilization tells you how busy something is. Saturation tells you how much work is waiting because it is too busy. A resource can be at 100% utilization with no queue and be perfectly fine. Conversely, an average of 60% over a minute can hide bursts that build a queue.

Practical saturation measures:

The pressure stall information files under /proc/pressure/ on modern Linux kernels are especially helpful, because they directly report the share of time tasks were stalled for a resource.

Mapping them to Kubernetes

USE covers more than hardware. Anything with limited capacity counts, including software resources. Here are starting points using metric names commonly exposed by the kubelet and the kube-state-metrics project. Confirm the exact names in your own cluster, and treat each line below as the calculation to perform over a suitable window, expressed in whichever query language your metrics stack uses.

RED for an HTTP service (OpenTelemetry semantic conventions metric)
  rate     = per-second increase of http.server.request.duration count
  errors   = same, filtered to http.response.status_code >= 500, divided by rate
  duration = p95 from the http.server.request.duration histogram

USE for a container
  CPU utilization  = rate(container_cpu_usage_seconds_total) / CPU limit or request
  CPU saturation   = rate(container_cpu_cfs_throttled_periods_total)
                     / rate(container_cpu_cfs_periods_total)
  Memory use       = container_memory_working_set_bytes / memory limit
  Memory errors    = kube_pod_container_status_last_terminated_reason == "OOMKilled"

USE for a node
  CPU utilization  = 1 - idle share of rate(node_cpu_seconds_total)
  Disk saturation  = rate(node_disk_io_time_weighted_seconds_total)
  Network errors   = rate(node_network_receive_errs_total)

CPU throttling is often missed. A container at 40% average CPU can be heavily throttled when it bursts up to its limit, which appears as latency with nothing else to explain it. See Kubernetes right-sizing for how to deal with that.

Dashboards that follow the methods

A layout that works for most services:

1. Top row: RED. Rate, error ratio and duration percentiles, with the SLO line marked. This row answers "is it broken?"
2. Second row: deploy markers and changes. Vertical lines on the time axis for deployments and config changes. You will look at this first during an incident.
3. Third row: dependencies. RED for each downstream service, from the client's point of view.
4. Fourth row: USE for what the service consumes. CPU and throttling, memory, connection pool, thread pool, disk if it has any.
5. Links. From this page to traces and logs filtered to the same time range and service.

Resist adding more. A service dashboard you can read in 30 seconds beats one that has fifty panels.

Alerting on them

Page on symptoms users feel. Ticket on causes that will become symptoms.

The commonest cause of noisy paging is alerting on every USE metric. If you are fighting that, read how to fix alert fatigue.

Where the methods fall short

• Batch jobs and pipelines. RED doesn't map neatly. Use freshness (how old is the newest output?), run duration, success of last run and backlog.
• Message consumers. Rate and errors apply, but lag is the signal that actually matters.
• Very low-traffic services. Percentiles and ratios become noisy. Synthetic probes can give a steady signal, and longer windows help.
• Business failures. A service can have flawless RED metrics while silently charging the wrong amount. Add a few business metrics, such as orders completed.
• Shared resources you can't see. A storage array or a noisy neighbour on a hypervisor won't appear in your container's USE data, except through its effects.

How Wrentrix surfaces these

Wrentrix shows endpoint p50/p95/p99 and grouped errors from traces, with resource usage as min/avg/p95/peak for Kubernetes and OpenShift workloads, and CPU, memory, disk and network for Linux hosts. Hour-of-week baselines judge values against what is normal for that time, and SLOs handle the paging. See the Kubernetes and OpenShift monitoring page for the workload view.

Frequently asked questions

What are the four golden signals?

They are latency, traffic, errors and saturation, as described in the Site Reliability Engineering book. Together, they summarise the health of a user-facing service. If you can only measure four things for a service, these are the suggested ones.

What is the difference between RED and USE?

RED (rate, errors, duration) describes the work a service does, as seen from the requests. USE (utilization, saturation, errors) describes the resources consumed to do that work. RED is typically used for alerting, while USE helps you find the bottleneck.

What is the difference between utilization and saturation?

Utilization is how busy a resource was, as a share of time or capacity. Saturation is the amount of work that had to wait because the resource was busy, such as a queue. A resource can be highly utilized with no saturation, and short bursts can cause saturation at a modest average utilization.

Should I alert on high CPU?

Usually not as a page, because high CPU alone doesn't mean users are affected, and well-tuned systems may run quite hot. Alert on latency and errors, and use CPU and throttling as diagnostic data. A sustained trend is better handled as a capacity ticket.

Which method should I start with?

Start with RED on your most important user-facing services, since it is simple and maps directly to user experience. Add saturation signals for the resources most likely to be limiting, such as connection pools and CPU throttling. Then expand USE to the rest of your resources when you need to diagnose something.