Running the OpenTelemetry Collector on Kubernetes

Choose a topology first

The Collector can run per node, as a shared service, or both.

For traces and logs delivered by SDKs over OTLP, start with a gateway Deployment alone. Add an agent DaemonSet when you need to tail node-level logs or collect host metrics. Add a sampling tier when you have decided on tail-based sampling, and not sooner.

Choose a distribution

Don't use a default image without looking at what it contains. The project publishes several distributions:

• otel/opentelemetry-collector-k8s is curated for Kubernetes, and it is a good default.
• otel/opentelemetry-collector-contrib includes nearly every component. It is useful for experiments, with a larger attack surface.
• You can build your own with the OpenTelemetry Collector Builder (ocb), with only what you use. In regulated environments, this is worth considering, because each included component is something your scanner reports on.

The official Helm chart now requires you to choose an image repository explicitly, so this decision is made up front.

A gateway config, line by line

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  k8sattributes:
    auth_type: serviceAccount
    passthrough: false
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.uid
      - sources:
          - from: resource_attribute
            name: k8s.pod.ip
      - sources:
          - from: connection
    extract:
      metadata:
        - k8s.namespace.name
        - k8s.deployment.name
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.node.name
  batch:
    send_batch_size: 8192
    timeout: 2s

exporters:
  otlphttp:
    endpoint: https://apm.example.internal/otlp
    compression: gzip
    headers:
      authorization: ${env:APM_INGEST_AUTH}
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlphttp]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, k8sattributes, batch]
      exporters: [otlphttp]

A few parts deserve explanation:

• Processor order matters. memory_limiter goes first, so it can refuse data before anything else allocates memory. batch goes last, so it batches what is ready to send. The k8sattributes processor needs the client's connection information, and placing it before batch avoids losing it.
• **memory_limiter refuses data under pressure.** Receivers return an error so that SDKs can retry. This is deliberate back-pressure, and it is a good deal better than an out-of-memory kill that loses everything in memory.
• The sending queue is in memory by default. A restart loses its contents. A file_storage extension can back it with disk, if you are prepared to give the pod a persistent volume.
• **max_elapsed_time sets how long a backend outage can be covered.** After that, batches are dropped. Do the arithmetic for your own volume rather than assuming that an outage of any length is safe.

The endpoint above is a placeholder. The otlphttp exporter appends /v1/traces and /v1/logs to it.

Installing with Helm

The project maintains a chart. Save something like the following as values.yaml, adding the config above under the config key. The chart merges your values with its defaults, so remember to look at the rendered result.

mode: deployment

image:
  repository: otel/opentelemetry-collector-k8s

replicaCount: 3

presets:
  kubernetesAttributes:
    enabled: true

resources:
  requests:
    cpu: 500m
    memory: 1Gi
  limits:
    memory: 2Gi

extraEnvs:
  - name: GOMEMLIMIT
    value: 1600MiB
  - name: APM_INGEST_AUTH
    valueFrom:
      secretKeyRef:
        name: apm-ingest
        key: authorization

Add the OpenTelemetry chart repository under the name open-telemetry, using the address given in the OpenTelemetry documentation, then install:

helm repo update
helm install otel-gateway open-telemetry/opentelemetry-collector \
  -n observability --create-namespace -f values.yaml
helm template otel-gateway open-telemetry/opentelemetry-collector -f values.yaml | less

The presets.kubernetesAttributes setting adds the k8sattributes processor and the RBAC it needs. If you manage RBAC yourself, that processor needs read access along these lines:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: otel-collector-k8sattributes
rules:
  - apiGroups: [""]
    resources: ["pods", "namespaces"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["apps"]
    resources: ["replicasets"]
    verbs: ["get", "watch", "list"]

The ReplicaSet permission is what lets it resolve a pod to its Deployment name. Without it, k8s.deployment.name goes missing. Exact needs depend on the attributes you extract, so check the processor's README for your version. Chart value names also evolve, so read the chart's values.yaml for the version you pin. In a closed network, mirror the chart and image internally; see installing Helm charts in air-gapped networks.

The memory example above follows simple arithmetic: GOMEMLIMIT of 1600 MiB is 80% of the 2 GiB limit, which gives the Go runtime room to collect garbage before hitting the container limit. It is a starting point, not a measured recommendation for your load.

The k8sattributes trap behind a Service

With a gateway, the "connection" source for pod association sees the IP of whichever client connected. If everything comes through a node-level agent, that address is the agent's pod, and every span could be tagged with the agent's metadata. This is a confusing bug because the output looks plausible.

To avoid it:

• When the app connects directly to the gateway, connection-based association usually works.
• When traffic passes through an agent, run k8sattributes on the agent, or have the agent add k8s.pod.ip as a resource attribute.
• Or inject k8s.pod.uid and k8s.pod.name into the app through the Downward API and OTEL_RESOURCE_ATTRIBUTES, which removes the ambiguity altogether.

gRPC does not balance through a normal Service

A Kubernetes ClusterIP Service balances at the connection level. gRPC keeps long-lived connections, so a client stays with whichever pod it first reached. After scaling up, new pods can sit nearly idle. You have a few options:

• Use OTLP over HTTP from your SDKs. Its shorter requests redistribute more evenly, although keep-alive still has some effect.
• Use a headless Service with client-side balancing, where the SDK supports it.
• Use the server-side max_connection_age keepalive setting on the OTLP gRPC receiver, so that clients reconnect periodically.

Log collection and the DaemonSet

To tail container logs from /var/log/pods, a Collector needs a hostPath mount on every node, and the filelog receiver. The chart has a presets.logsCollection setting for this in daemonset mode.

On OpenShift, hostPath is not allowed under the default restricted-v2 SCC. Using it means granting an additional SCC to that service account, which your security team will want to review. The alternatives are to send logs over OTLP from the application, or to use the platform's own logging. The restricted-v2 guide explains what is permitted and why.

Monitoring the Collector

A silent pipeline is the worst kind of failure. The Collector exposes its own metrics, and three questions matter most:

1. Is it refusing data? Look at otelcol_receiver_refused_spans. Non-zero usually points to memory limiter pressure.
2. Is it failing to send? Look at otelcol_exporter_send_failed_spans.
3. Is the queue filling up? Compare otelcol_exporter_queue_size with otelcol_exporter_queue_capacity. A queue that keeps rising means the backend can't keep up.

Exact metric names can vary across versions, so confirm them against your running instance. Put alerts on these and make someone own them. Also, the debug exporter is handy for looking at pipeline output, but remove it when done.

Sizing it

There is no universal number. Run a load test with your real span sizes. Then:

1. Send a known rate, and watch CPU and memory at a steady state.
2. Set a memory limit with headroom for the queue and batches.
3. Add a HorizontalPodAutoscaler on the gateway. A stateless gateway scales well, but a tail sampling tier needs more care since membership changes move traces around.
4. Use at least three replicas and a PodDisruptionBudget, so that node drains do not interrupt the path.

Using a Collector with Wrentrix

Wrentrix accepts OTLP traces and logs over HTTP with gzip, and it works alongside the collectors you already run, so you can add it as another otlphttp exporter in an existing pipeline. See the tracing page for what it does with the data.

Frequently asked questions

Should the OpenTelemetry Collector run as a DaemonSet or a Deployment?

Run a Deployment as your central gateway if SDKs send over OTLP, since it is simpler and scales by replica count. Add a DaemonSet when you need something on every node, such as tailing container logs or collecting host metrics. Many production setups use both.

What is the difference between the contrib and k8s Collector images?

The contrib image contains nearly all community components, while the k8s distribution contains a curated set aimed at Kubernetes use. Fewer components mean a smaller attack surface and less to patch. If neither fits, the Collector Builder can produce an image with exactly what you need.

Why is my OpenTelemetry Collector running out of memory?

Common causes are a missing or misordered memory_limiter, a large sending queue while the backend is down and a tail sampling buffer sized too large for the pod. Set GOMEMLIMIT below the container limit as well. Look at the queue and refused-span metrics to find which one applies.

How do I add Kubernetes metadata to my traces?

Use the k8sattributes processor with RBAC that lets it read pods, namespaces and ReplicaSets. It matches each span to a pod and adds attributes such as namespace and deployment. Make sure pod association does not pick up the IP of an intermediate agent by mistake.

Can the OpenTelemetry Collector run on OpenShift without extra privileges?

A gateway that only receives OTLP normally needs no special privileges and can run under restricted-v2. Node-level log tailing needs hostPath, which that SCC doesn't allow. Red Hat also offers its own build of OpenTelemetry through an operator, which you may prefer for supportability.