What changed? Using change correlation to shorten incidents

An example: the ConfigMap nobody connected

This is an illustrative scenario. At 16:40, error rates on a checkout service rise from near zero to around 8%. The on-call engineer sees that the last deployment was two days ago and rules out a release. Forty minutes of looking at the database and the network follow.

The actual sequence:

Nothing changed at 16:40 except pod identity. The cause was an edit two days earlier, and the trigger was a node drain. An error spike that affects one third of pods is the clue here. Splitting errors by pod would have shown a clean division between old and new, and comparing pod start times with the ConfigMap's edit time would have finished the job.

This is what makes change correlation difficult. The relevant change may not be recent, and it may not be recorded where you first look.

What Kubernetes tells you, and for how long

Events expiring after an hour is the one that most often causes trouble. In an incident that started overnight, the evidence has already gone. Ship events to durable storage. The OpenTelemetry Collector has a k8sobjects receiver that can watch events, and the Collector on Kubernetes guide covers the deployment pattern.

Quick commands when you have nothing else

Start with recent activity across the namespace:

kubectl get events -n payments --sort-by=.lastTimestamp | tail -40
kubectl rollout history deployment/checkout -n payments
kubectl get pods -n payments \
  -o custom-columns=NAME:.metadata.name,STARTED:.status.startTime,NODE:.spec.nodeName,RS:.metadata.ownerReferences[0].name

That last command is more useful than it looks. Pod start times and ReplicaSet names show which pods are running which version, and you can correlate the failing ones with either.

To compare two revisions:

kubectl rollout history deployment/checkout -n payments --revision=7 > /tmp/r7.txt
kubectl rollout history deployment/checkout -n payments --revision=8 > /tmp/r8.txt
diff /tmp/r7.txt /tmp/r8.txt

Find when a config object was last touched:

kubectl get configmap checkout-config -n payments \
  -o jsonpath='{range .metadata.managedFields[*]}{.manager}{"\t"}{.operation}{"\t"}{.time}{"\n"}{end}'

A kubectl rollout restart shows up as a new revision whose only difference is the kubectl.kubernetes.io/restartedAt annotation on the pod template. It's an easy one to miss if you only look at image tags. The cluster-side view of what's different from your manifests is kubectl diff -f manifest.yaml.

On OpenShift, oc provides the same subcommands. Cluster-level changes also matter there: oc get clusterversion and oc get machineconfigpool show upgrades and node configuration rollouts, either of which can restart pods across many namespaces. The OpenShift monitoring guide goes into platform-level signals.

Why ConfigMaps and Secrets are the blind spot

Editing a ConfigMap or Secret does not produce a Deployment revision and does not restart anything. What happens next depends on how the pod consumes it:

So a bad value can sit dormant for days, and take effect during an unrelated restart. Two approaches make these changes visible:

Roll on change. A common Helm pattern places a hash of the config in the pod template. Any config change then produces a normal rollout at a time you chose, with a revision you can see and undo.

spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

Record the change. Keep a timeline of config object edits, whether through the audit log, a GitOps diff or a monitoring tool. For Secrets, record that a change happened and a hash or version. Never store the value.

Changes outside the cluster's view

A good timeline also covers the things that do not show up in any Kubernetes API. When nothing in the namespace changed, go through this list:

• Feature flags and runtime configuration services
• Database schema migrations, index builds, vacuum runs and statistics changes
• Certificate rotation, which is a change that can happen by itself on a schedule
• Upstream dependencies deployed by another team
• Network changes: firewall rules, load balancer settings, DNS records
• Node-level changes: kernel updates, drains and autoscaler activity
• Scheduled jobs, batch imports and month-end reports
• Traffic mix changes, such as a new client version or a marketing send

Correlation is a lead, not a verdict

Lining up events and spikes generates hypotheses. Treat it with care, because in a busy cluster there is always something that changed just before the problem.

Some tests that separate cause from coincidence:

1. Blast radius matches. Did only pods from the new ReplicaSet degrade? Did only the changed service's callers suffer? A change affecting service A cannot explain errors that began in service B upstream of it, unless there's a path between them.
2. Time order is right. The change must come first, allowing for how the change takes effect. A deployment that finished after the spike began did not cause it, although its pods may have started earlier, so use pod ready times instead of the deploy's completion time.
3. The signal fits the mechanism. A timeout config change should produce timeouts. A memory limit change should produce OOM kills. If the change could not plausibly produce that symptom, keep looking.
4. Reverting helps. Rollback is both a mitigation and a test. If kubectl rollout undo fixes things, you have strong evidence. Be careful with the converse: a rollout can appear to fix an issue simply because it restarted the pods, clearing a bad state.
5. Topology fits. Changes on the same dependency path rank higher than something else that happened to be nearby in time.

Common false positives are autoscaling events (usually a result of load, not its cause), pod restarts that follow the failure, and frequent automated deployments in unrelated services. Filter out or down-rank these. Note that a change arriving during a period of ongoing noise needs careful reading, so use a comparison with the previous hours rather than a pure time match.

Make your own telemetry carry change data

Some small habits make correlation much easier for any tool you use:

• Set service.version in OpenTelemetry resource attributes, using the image digest or Git SHA. Comparing errors by version becomes a one-click query. OTEL_RESOURCE_ATTRIBUTES=service.version=1.42.0,deployment.environment.name=prod does this without code changes.
• Annotate rollouts with kubernetes.io/change-cause. This value appears in kubectl rollout history, which otherwise shows <none>.
• Emit a deploy marker from CI to your dashboards. Include the commit, pipeline URL and owner.
• Keep ConfigMaps in Git and treat out-of-band edits as a defect, with a drift report.

A five-minute routine

At the start of any incident, before forming a theory:

1. Pin down when the problem began, using a chart with a resolution finer than the alert window. Alerts fire after the start, sometimes well after it.
2. List changes in the hour before this point, then widen to 48 hours for config objects.
3. Split the failing metric by version, pod, node and zone. A clean split is frequently the fastest route to the answer.
4. Check the upstream and downstream services on the map for the same onset time.
5. If a recent change fits and rollback is safe, roll back first and investigate after. The guide to reducing MTTR explains why mitigation should come before diagnosis.

How Wrentrix lines it up

Wrentrix's Kubernetes and OpenShift monitoring records spec, ConfigMap and Secret changes and rollout restarts, and lines them up against latency and error spikes. The problem engine groups related signals by topology into one incident, and automatic RCA ranks candidates with a clickable evidence chain, so you can check each suggestion rather than taking it on trust. Candidates are still hypotheses. The tests above apply to them too.

Frequently asked questions

How do I see what changed in a Kubernetes deployment?

Run kubectl rollout history deployment/<name>, then use --revision=<n> on two revisions and diff the output. That only covers the pod template, so also check when the ConfigMaps and Secrets it uses were modified, and check recent events. If you need to know who made a change, you need the API server audit log.

How long does Kubernetes keep events?

The default retention is one hour, controlled by the API server's --event-ttl flag. Events therefore often vanish before anyone investigates an overnight incident. Export them to durable storage if you want them for incident work.

Does updating a ConfigMap restart pods?

No. Pods using a ConfigMap as an environment variable keep the old value until the container is recreated, and mounted volumes update eventually with no restart, except for subPath mounts which do not update. This delay is why a config change can cause an incident long after the edit. A checksum annotation on the pod template is the standard way to force a rollout.

What is change correlation in observability?

Change correlation is the practice of placing deployments, configuration edits and other changes on the same timeline as metrics, errors and latency, so likely causes can be identified quickly. It generates candidates rather than proof. Blast radius, timing order and the effect of a rollback are what confirm or reject each candidate.