A practical guide to OpenShift monitoring

OpenShift ships with more monitoring than many teams realise. Before adding anything, it pays to know what the platform already gives you, how to turn on the part that is off by default, and which questions it will not answer.

What you get out of the box

OpenShift Container Platform 4 installs a metrics stack in the openshift-monitoring namespace, managed by the Cluster Monitoring Operator. It covers the platform itself: nodes, control plane components, etcd, the API server, Kubernetes objects and more. It comes with a set of alerting rules, and the web console shows metrics, alerts and dashboards under Observe.

Look at what is running: 

oc get pods -n openshift-monitoring
oc get clusteroperator monitoring
Layer What it covers Default state
Platform monitoring Nodes, control plane, cluster components, resource use of all pods On
Alert routing Grouping and sending platform alerts Running, but needs receivers configured
User workload monitoring Your application's own metrics and alert rules Off
Logging Container, infrastructure and audit logs Requires installing the Red Hat logging operators
Distributed tracing Traces Requires Red Hat operators, including the Red Hat build of OpenTelemetry
Network observability Network flows Requires the Network Observability Operator

Available operators and their versions change between OpenShift releases, so check the docs for your version on docs.redhat.com. Also remember that the platform stack has short default retention and limited scale. It is designed to keep the cluster healthy, not to be your long-term metrics store.

Step 1: Route the platform alerts

A fresh cluster has alerts firing that nobody receives, and the console warns you about this until you configure receivers. Do this first, in the alert routing settings under Administration > Cluster Settings, by adding your email, webhook or paging receivers.

Also note the Watchdog alert, which is always firing by design. It is a dead man's switch. Route it to an external system that pages if it stops arriving. Otherwise, a dead monitoring stack looks exactly like a quiet night.

In a restricted network, verify that your receivers are reachable from inside the cluster, since a blocked SMTP or webhook path fails silently from the responder's point of view.

Step 2: Turn on user workload monitoring

If your applications expose metrics, enable the user workload layer. It runs as a separate set of components in the openshift-user-workload-monitoring namespace. 

apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true
oc apply -f cluster-monitoring-config.yaml
oc get pods -n openshift-user-workload-monitoring

If the ConfigMap already exists, edit it rather than replacing it, so you don't lose existing settings. Afterwards, teams declare scrape targets and alert rules through custom resources in their own namespaces, and developers see their data under Observe in their project. Grant access using the roles that the docs list, such as monitoring-edit: 

oc policy add-role-to-user monitoring-edit dev-user -n payments

Settings for that layer, including retention, go in the user-workload-monitoring-config ConfigMap in its own namespace. Check the supported fields for your version, as an unsupported key can cause problems in reconciliation. Red Hat also documents limits on what you may change in the default stack, so don't patch the operator-managed resources directly. The operator will revert your change, and it may leave you in an unsupported state.

Metric cardinality is the main risk for this layer. One label containing a user ID or a request path can consume a great deal of memory. Review what teams expose, and use the documented limits for scraped samples.

Step 3: Know the first commands in an incident

Before opening a dashboard, these answer most of the first questions: 

oc get clusterversion
oc get clusteroperators
oc get nodes
oc get machineconfigpool
oc adm top nodes
oc adm top pods -n payments --sort-by=cpu
oc get events -n payments --sort-by=.lastTimestamp | tail -30
oc get pods -n payments -o wide
oc describe pod <pod> -n payments
oc logs <pod> -n payments --previous

What each answer tells you:

  • **clusteroperators** shows any platform component that is DEGRADED, or not AVAILABLE. If authentication, ingress or the monitoring operator itself is degraded, your application symptoms may have a platform cause.
  • **machineconfigpool** shows whether nodes are in the middle of a configuration rollout. Nodes that are draining and rebooting explain a lot of pod churn, and this is an OpenShift-specific cause people often forget.
  • **--previous logs** show what the container printed before its last crash. Without this flag, you see only the fresh container.
  • **events** are short-lived, and they disappear after a short period by default. Capture them early, or ship them elsewhere.

For a support case or a deep investigation, oc adm must-gather collects a large diagnostic bundle. Note that it needs its image available, so plan for that in a disconnected cluster, and treat the bundle as sensitive data.

Alerts worth knowing

The default rules are extensive. A few to learn by name:

Alert What it usually means First step
KubePodCrashLooping A container keeps restarting Check --previous logs and exit code; 137 suggests OOM
KubeAPIErrorBudgetBurn The API server's own SLO is burning Look for heavy clients and etcd latency
ClusterOperatorDegraded A platform component is unhealthy oc describe clusteroperator <name>
KubeNodeNotReady A node has stopped reporting ready Node conditions, kubelet state, network
Watchdog Always firing; proves the alert path works Alarm if it stops

Don't silence platform alerts just because they're unfamiliar. Look them up first. The console's alert detail page has a description, and many rules include a runbook link. In a closed network, those links may not open, so consider mirroring the runbooks you rely on.

What the built-in stack won't tell you

Platform monitoring answers "is the cluster healthy?" and "how much is this pod using?" Teams then run into these gaps:

Request-level questions. Why is /checkout slow for some users? CPU and memory graphs can't tell you. You need traces or at least application metrics. On OpenShift, you can use the OpenTelemetry operator for this, as shown in the auto-instrumentation guide.

Correlation. Metrics, logs and traces are separate views. Getting from a latency spike to the logs for the affected requests takes manual steps unless you've added trace IDs and links.

Change history. An event log and a rollout history exist, but they're not lined up against your latency graph. "A ConfigMap changed four minutes before errors rose" is often the answer, and it's hard to see. See what-changed analysis.

Long-term trends. With short retention, comparing this month to last quarter takes a separate store. Hour-of-week baselines need history too.

Right-sizing. The data is available, but you have to query it for each workload, and interpret it. The right-sizing guide shows how.

Multi-cluster views. One cluster's console shows that cluster. With several clusters, you need something on top.

Services outside the cluster. Most regulated estates still have important Linux hosts, and databases outside OpenShift. Their health doesn't appear here, unless you add it.

Permissions and agents

Any monitoring agent you add must fit OpenShift's security model. Many products require a privileged DaemonSet, hostPath mounts or hostNetwork, which means granting a non-default SCC. Security teams will rightly ask why. Before you install anything, check which SCC each pod needs and what it does with that access. The restricted-v2 guide explains how to verify this.

Also think about admission webhooks. Agents that inject into pods use mutating webhooks, and a badly configured one can block pod creation across the cluster, including during an upgrade. Check its failurePolicy and namespaceSelector, and make sure it excludes openshift-* namespaces.

Disconnected clusters

In a disconnected OpenShift cluster, you mirror release images and operator catalogs into an internal registry. For monitoring:

  • Remote health reporting can't send data out, so consider disabling it in the documented way, rather than leaving it to fail.
  • Runbook links may need internal copies.
  • Every additional operator needs to be included in your mirror.
  • Alert receivers must all be internal.
  • must-gather needs its image mirrored.

How Wrentrix fits on OpenShift

Wrentrix runs on OpenShift under the default restricted-v2 SCC, with least-privilege pods, no root and no extra SCCs, and it installs through an operator or a Helm chart. You add the cluster in the UI, download a generated YAML (oc apply -f wrentrix-k8s-agent.yaml), and nodes and pods appear in about two minutes. It adds workloads, events, HPA, right-sizing hints, change correlation and traces on top of what the platform offers, and it can run in a fully disconnected cluster. It complements the built-in stack, which you should keep for platform alerting. The Kubernetes and OpenShift page has more detail.

Frequently asked questions

Does OpenShift come with monitoring built in?

Yes. OpenShift 4 includes a metrics and alerting stack managed by the Cluster Monitoring Operator, plus dashboards in the web console. It covers the platform by default, and you can enable user workload monitoring for your own applications. Logging and tracing need additional operators.

How do I enable user workload monitoring in OpenShift?

Set enableUserWorkload: true in the config.yaml of the cluster-monitoring-config ConfigMap in the openshift-monitoring namespace. Components then start in openshift-user-workload-monitoring. Developers need the appropriate monitoring roles in their projects to manage scrape targets and rules.

Why is the Watchdog alert always firing in OpenShift?

It is designed that way, to prove that the alerting pipeline works. Route it to an external system that raises an alarm if it stops arriving. Silencing it removes one of the simplest checks that your monitoring is alive.

Can I use the built-in OpenShift monitoring as long-term storage?

It isn't designed to be your long-term store, and its retention and scale are limited. Red Hat documents a remote write option that sends metrics to another system. Check the supported settings for your version, and size any local retention increase carefully.

Do monitoring agents need privileged access on OpenShift?

Many agents ask for it, especially for host-level metrics and log tailing, but not all do. Check the openshift.io/scc annotation on each pod after installing. If a tool needs a non-default SCC, ask the vendor what each privilege is for, and have your security team review it.