Auto-instrumentation with the OpenTelemetry Operator

What actually happens when a pod starts

Understanding the sequence makes every later failure easier to diagnose. When a pod carrying an inject annotation is created:

1. The API server calls the operator's mutating admission webhook.
2. The webhook finds the matching Instrumentation resource.
3. It adds an init container that copies the language agent from an image into a shared emptyDir volume.
4. It mounts that volume in your application container and sets environment variables that make the runtime load the agent, for example JAVA_TOOL_OPTIONS for Java.
5. It adds OTEL_* variables: service name, endpoint, sampler and resource attributes.
6. Your application starts with the agent loaded, and your image remains unchanged.

Two consequences follow straight away. Injection only happens at pod creation, so existing pods are unaffected until they are recreated. And your container now has a dependency on an init container image, which has to be pullable from wherever the node is, including inside a closed network.

Prerequisites

• A Kubernetes or OpenShift cluster, with permission to install cluster-scoped resources.
• TLS certificates for the operator's webhooks. The standard manifest expects a certificate-management operator to be already installed. The project's Helm chart offers other ways of providing certificates, so read its documentation for your version.
• An OTLP destination: usually a Collector in the cluster. See running the Collector on Kubernetes.

On OpenShift, Red Hat ships a supported build of OpenTelemetry through OperatorHub. If you are on a Red Hat subscription, prefer that, and check its documentation, since available features and defaults can differ from upstream.

Upstream, with certificates handled, installation means applying the release manifest that the project publishes:

kubectl apply -f opentelemetry-operator.yaml

For production, pin a specific release rather than the latest one, and mirror it for closed networks.

Define an Instrumentation resource

This resource describes how to inject. It usually belongs in the application's namespace, and one namespace can have several.

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default
  namespace: payments
spec:
  exporter:
    endpoint: http://otel-gateway-opentelemetry-collector.observability.svc:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1.0"
  resource:
    addK8sUIDAttributes: true
  java:
    env:
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: http/protobuf
  nodejs:
    env:
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: http/protobuf

Things to check here:

• Protocol and port must match. Defaults differ by language and have changed across versions. Some language agents default to http/protobuf, others have used gRPC. Using 4318 with HTTP or 4317 with gRPC is the rule. Setting OTEL_EXPORTER_OTLP_PROTOCOL per language, as above, removes the guesswork.
• **sampler.argument is a string.** Unquoted numbers get rejected by the schema.
• Pin agent images. By default, the operator uses the agent version that it was built with. Each language section accepts an image field, which lets you pin a version from your own registry. In an air-gapped cluster, you need this anyway.

Annotate the workload

Annotate the pod template, not the Deployment's own metadata. This is the most frequent setup mistake.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-api
  namespace: payments
spec:
  selector:
    matchLabels:
      app: order-api
  template:
    metadata:
      labels:
        app: order-api
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"
    spec:
      containers:
        - name: app
          image: registry.example.internal/payments/order-api:1.8.2

The available annotations:

The value can be:

• "true", which uses the Instrumentation in the same namespace
• "my-instrumentation", a name in the same namespace
• "other-namespace/my-instrumentation", a specific one elsewhere
• "false", which opts out, and is useful when the namespace is annotated as a whole

You can also place the annotation on a namespace, which applies it to all pods there. That is convenient and risky at the same time, so do a single workload first.

In a pod with several containers, such as one with a sidecar, specify which ones to instrument with instrumentation.opentelemetry.io/container-names: "app".

Language-specific details:

• .NET on musl. Images based on the musl C library need instrumentation.opentelemetry.io/otel-dotnet-auto-runtime: "linux-musl-x64". Otherwise the default is for glibc and the profiler fails to load.
• Python versions. The auto-instrumentation image is built against specific Python versions. A mismatch with your app's Python version, or with packages that have native extensions, can cause import errors.
• Java. The agent is attached through JAVA_TOOL_OPTIONS. If your startup script overwrites that variable, the agent will not load.

Roll it out and verify

Trigger a rollout of the workload you annotated:

kubectl rollout restart deployment/order-api -n payments

Then check that injection happened:

kubectl get pod -n payments -l app=order-api \
  -o jsonpath='{.items[0].spec.initContainers[*].name}{"\n"}'

kubectl get pod -n payments -l app=order-api \
  -o jsonpath='{range .items[0].spec.containers[0].env[*]}{.name}{"\n"}{end}' | grep -E 'OTEL|JAVA_TOOL'

kubectl logs -n payments deploy/order-api -c opentelemetry-auto-instrumentation-java

An init container named opentelemetry-auto-instrumentation-java should be present, along with OTEL_SERVICE_NAME and the other variables. The service name defaults to something derived from the workload, such as the Deployment name, unless you set it yourself.

Failure modes

Think about what happens when the webhook is down

Every pod creation that matches a mutating webhook depends on it. The failurePolicy decides what happens when it can't be reached: Ignore admits the pod without injection, and Fail rejects the pod. Look at what is actually in your cluster:

kubectl get mutatingwebhookconfigurations
kubectl get mutatingwebhookconfiguration <name> -o yaml | grep -E 'name:|failurePolicy|timeoutSeconds'

For observability tooling, prefer a design where a broken monitoring component cannot stop your applications from scaling. Also run more than one operator replica, and keep the webhook's scope narrow with a namespaceSelector so that system namespaces are excluded. The admission webhooks guide goes into more detail, including latency budgets.

Roll-out order we suggest

1. Staging, one service. Compare startup time, memory and p95 latency, with and without the agent.
2. Raise memory limits as needed. Do this before production, not during it.
3. Production canary. One Deployment with low risk and real traffic. Leave it running for a few days.
4. Sampling. Pick a ratio or a tail-based design based on measured volume.
5. Noise reduction. Health checks and metrics scrapes can make up many of your traces. Filter them at the Collector or by agent configuration.
6. Namespace-level defaults. Only after several teams have succeeded. Keep the "false" opt-out documented.

What auto-instrumentation cannot tell you

It gives you server, client and database spans at the framework edges. A 900 ms span in your own handler stays an opaque 900 ms. Your options are manual spans, a profiler or method-level capture. It also doesn't name your business concepts: you need manual attributes to segment latency by customer tier or payment type.

How Wrentrix uses this

Wrentrix uses the OpenTelemetry operator for auto-instrumentation and accepts OTLP from any SDK. Separately, it adds method-level tracing for Java and .NET through its own admission webhook hooks, so those services are traced to the method on their next restart. Wrentrix never restarts workloads itself, so that timing is up to you. Its webhook is fail-open: if the agent is unavailable, pods start without instrumentation.

Frequently asked questions

Do I need to restart pods for OpenTelemetry auto-instrumentation?

Yes. Injection happens when a pod is created, so running pods stay uninstrumented until they are replaced. A rolling restart or your next normal deployment will do it. Applying an annotation or an Instrumentation resource does not change existing pods.

Why is the OpenTelemetry Operator not injecting my pod?

The most common reason is that the annotation is on the Deployment's metadata rather than the pod template. Other common causes are a missing Instrumentation resource in the namespace, a pod that was not recreated and an unreachable webhook. Check the operator logs and the pod's spec for an added init container.

Does the OpenTelemetry Operator work on OpenShift?

Yes, and Red Hat provides its own supported build of OpenTelemetry through OperatorHub. Injected init containers need to work under your pod's security context constraint, so test under restricted-v2. Some modes, such as Go auto-instrumentation, need elevated privileges that this SCC does not provide.

Can I use the operator with an application that already has an OpenTelemetry SDK?

Yes, but avoid injecting a language agent as well, since that may produce duplicate spans or conflicts. The inject-sdk annotation is designed for this case: it sets the OTEL_* environment variables without adding an agent. The SDK in your application then reads those values at startup.

How much overhead does auto-instrumentation add?

It varies with the language, the libraries and the traffic. The main costs are extra startup time and more memory per process, plus a small per-request cost. Measure it on your own service under load and raise memory limits as needed before you roll out widely.