Tail-based sampling explained

The problem with deciding early

Take a service with 2,000 requests per second, where 0.2% fail, and suppose you can afford to keep about 5% of traces. These figures are an example.

With head-based sampling, each trace is kept or dropped at the very start, when nothing is known about the outcome. You keep 5% of everything:

• Traces kept: 2,000 × 0.05 = 100 per second
• Failed traces kept: 2,000 × 0.002 × 0.05 = 0.2 per second, or one every 5 seconds
• Failed traces lost: 95%

With tail-based sampling, you can keep all failures and fill the remaining budget with a sample of everything else:

• Failed traces per second, all kept: 2,000 × 0.002 = 4
• Remaining budget: 100 − 4 = 96 traces per second
• Sampling rate needed for healthy traces: 96 / 1,996 ≈ 4.8%

The stored volume is the same, but you get 20 times as many failed traces. Rare failures and slow outliers are the traces you actually want during an incident, which is the whole argument for tail sampling.

Why it is harder than it looks

A tail sampler has to see all spans of a trace before it decides, which creates three problems.

Spans arrive from many services over time. The sampler doesn't know when a trace has finished. It waits a fixed period after the first span and then decides on whatever has arrived. Spans that come later miss the decision.

All spans of a trace must go to the same instance. If order-svc spans land on collector A and pricing spans land on collector B, neither has the full picture, and they may decide differently. You end up with partial traces. A normal round-robin Service does exactly this.

Everything is held in memory. You buffer every span of every in-flight trace, including the 95% you'll throw away.

The two-tier architecture

Because of the routing requirement, you normally run two layers of Collectors:

apps --OTLP--> [ routing tier ]  --by trace ID-->  [ sampling tier ] --OTLP--> backend
               loadbalancing exporter              tail_sampling processor
               stateless Deployment                one headless Service, N pods

The routing tier uses the loadbalancing exporter, which hashes the trace ID to pick a sampling pod. Here it is using DNS on a headless Service:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20

exporters:
  loadbalancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true
    resolver:
      dns:
        hostname: otel-sampler-headless.observability.svc.cluster.local
        port: 4317

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [loadbalancing]

The insecure: true line is for an in-cluster example. If your policy demands TLS inside the cluster, configure certificates instead. The loadbalancing exporter and tail_sampling processor are not in every Collector distribution, so check that your image includes them.

The sampling tier and its policies

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  tail_sampling:
    decision_wait: 30s
    num_traces: 200000
    expected_new_traces_per_sec: 2000
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 2000
      - name: keep-payments
        type: string_attribute
        string_attribute:
          key: service.name
          values: [payment-gateway]
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5
  batch: {}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlphttp]

A trace is kept if any policy here says to keep it. Note the following:

• A single global threshold_ms is a blunt tool. A 2-second threshold might mark half the traffic of a report endpoint as slow while missing a login that normally takes 40 ms and now takes 1.5 s. You can combine policies with attributes using and or composite policy types for per-service thresholds. Read the processor's README for your version, since policy types are added over time.
• probabilistic is the baseline that gives you healthy traces for comparison. Without some healthy traces, you can't see what normal looks like, and percentile views get badly skewed towards the slow ones.
• composite policies with rate allocations can limit the volume a single noisy service consumes.
• Put tail_sampling after memory_limiter and before batch.

Sizing: work it out before you deploy

Assume, as an example:

• 2,000 new traces per second, across all instances
• an average of 15 spans per trace
• 2 KB of in-memory cost per span (this needs measuring, since it depends on your attributes)
• decision_wait of 30 s
• 3 sampler pods

Then:

• Traces in the buffer: 2,000 × 30 = 60,000
• Spans in the buffer: 60,000 × 15 = 900,000
• Memory in total: 900,000 × 2 KB = 1.8 GB
• Memory per pod: 1.8 GB / 3 = 600 MB

Add headroom for batching, the exporter queue, spikes and Go runtime overhead. A burst of traffic or a sudden rise in span count raises this directly. Set num_traces to comfortably exceed the expected buffer. In this example, 200,000 is over three times the 60,000 needed. If the limit gets hit, older traces get evicted before their decision.

Long traces are the sizing trap. One background job with 100,000 spans sits in memory like thousands of normal traces.

Choosing decision_wait

decision_wait is how long after the first span the sampler waits to decide. Too short and late spans miss the decision. Too long and memory rises in step. A reasonable approach:

1. Look at your trace durations, not your averages. Find the p99 end-to-end duration.
2. Add the export delays. SDKs batch spans, commonly for a few seconds, and Collectors batch again.
3. Set decision_wait a little above that.

For a p99 of 8 seconds with up to 10 seconds of batching delay, 30 s is safe and 20 s is probably fine. For traces that take minutes, such as async workflows, tail sampling is a poor fit, and you may want to sample those on a separate pipeline.

Spans arriving after a decision take on the decision already made for that trace, if it's still in the decision cache. The decision_cache settings in recent versions let you keep that memory longer. Check the docs for your version.

Scaling and restarts

When a sampling pod is added or removed, the routing tier's hash ring changes. Traces in flight during the change may have their spans split across two pods, so you get a short window of incomplete traces. This usually isn't a problem, but it does mean that autoscaling on a jittery signal would keep producing partial traces. Prefer a fixed size with headroom, or scale conservatively.

A sampler restart also loses its in-memory buffer, including any errors in it. Do rolling updates one pod at a time, and accept a small loss. You can't get a full guarantee, since it's a sampling system.

Monitor the sampler's own telemetry. The processor publishes metrics about late-arriving spans, traces dropped too early and policy decisions. Names change across versions, so list what your instance exposes. A rise in dropped-too-early traces means num_traces or memory is too low.

Metrics must come from before sampling

If you calculate error rates or request rates from sampled spans, your numbers are skewed: keeping every error while keeping 5% of the rest makes the error rate look many times higher. Do one of these:

• Generate RED metrics from spans before the sampling tier. The spanmetrics connector in the Collector does this, and it needs to run ahead of the sampler.
• Use the metrics the application or auto-instrumentation produces itself.
• Have the backend compute aggregates from unsampled data.

This also matters for SLOs: burn-rate alerts on sampled data would misreport the budget.

Head and tail together

They are not mutually exclusive. Very high-traffic services can apply a head ratio first and a tail sampler afterwards, which reduces buffering. But any error dropped at the head never reaches the tail, so use this only if the volume forces you. A parentbased_traceidratio sampler in the SDK respects the upstream decision, so all services agree. See W3C Trace Context for how that flag travels.

When not to use tail sampling

• Your volume is low enough that you can keep everything. Simplicity wins.
• Head sampling at a modest rate plus metrics and logs gives you enough, and failures are not rare.
• You have no one to run and monitor another stateful tier.
• Your traces commonly run for minutes.

Tail sampling in Wrentrix

Wrentrix's tracing includes tail-based sampling, and it accepts OTLP from your SDKs or existing collectors. For what happens to the traces you keep, see the tracing page. If you need details on how it would behave at your volumes, ask us.

Frequently asked questions

What is the difference between head-based and tail-based sampling?

Head-based sampling decides at the start of a trace, before the outcome is known, and it passes that decision downstream. Tail-based sampling buffers spans and decides once the trace is complete, using facts like errors and duration. Tail sampling keeps more useful traces, but it needs more memory and more infrastructure.

Why do I get partial traces with tail sampling?

Most often, spans of the same trace reached different sampler instances, because no trace ID routing tier exists. Late spans can also miss the decision if decision_wait is too short, and scaling or restarts can split traces. Use the loadbalancing exporter with routing_key: traceID, and review your timing.

How much memory does the OpenTelemetry tail sampling processor need?

Roughly, it needs traces per second × decision_wait × spans per trace × memory per span, plus headroom. Attribute size has a big effect on the per-span cost, so measure it with your own traffic. Use memory_limiter and watch the processor's metrics to catch evictions.

Does tail-based sampling affect my metrics?

It will if you compute metrics from the sampled spans, since the kept traces are biased towards errors and slow requests. Generate metrics before the sampling stage, or take them from the application directly. Your SLOs and error rates should never depend on sampled data alone.