Distributed tracing that stays inside your network

Follow one slow request across every service it touched, using standard OpenTelemetry instrumentation and a backend you run yourself.

An incident tracing is built for

Here is an invented but typical example. Customers report that payment confirmation sometimes takes eight seconds. The payments team checks their dashboard: CPU is fine, the database looks fine, and average latency has moved from 180 ms to 240 ms, which nobody would page on. Meanwhile orders blames payments, because that is where its timeout fires.

Averages hide this kind of fault, and per-service graphs cannot show it, because the problem is a property of a path and not of a single process. A trace of one slow request shows the path directly. In this example, orders calls payments, payments calls fraud-check, and fraud-check makes four sequential calls to a rules service. Three return in 20 ms. The fourth waits 7.6 seconds, and the trace shows it was retried after a connection reset. Now the conversation changes from "whose service is slow" to "why does the rules service reset connections on one pod".

That is the whole value of distributed tracing: each request carries an identity across process boundaries, every unit of work records its timing against that identity, and the backend reassembles the tree. If you want the concepts from the beginning, including spans, context propagation and parent-child relationships, read what distributed tracing is. The rest of this page covers how the capability works in Wrentrix and what you do as the engineer setting it up.

What you get once spans arrive

Wrentrix is OpenTelemetry-native. It accepts OTLP traces and logs over HTTP, as protobuf or JSON, with gzip compression. Any OpenTelemetry SDK can send to it, so there is no proprietary tracer to adopt and no lock-in at the instrumentation layer. From those spans it builds:

  • A live service and dependency map, drawn from what the traffic actually did and not from a diagram someone maintained by hand.
  • Endpoint latency at p50, p95 and p99, so a service with a healthy median and a bad tail shows up as exactly that.
  • Error groups, which collapse thousands of failures with the same shape into one item you can triage.
  • Logs linked to traces, so you can go from a slow span to the log lines written during that request.
  • Tail-based sampling, so slow and failed traces can be kept without storing every healthy one.

The percentile split matters more than it first appears. Take an example endpoint with p50 of 90 ms, p95 of 140 ms and p99 of 3.2 s. If a page makes 20 backend calls to endpoints like this one, and each call independently has a 1% chance of landing in that tail, the chance that a page load hits at least one is 1 − 0.99²⁰, which is about 18%. A tail that looks small per call becomes ordinary per page. Golden signals and RED goes further into which numbers to watch per service.

Getting spans into Wrentrix

There are three common routes. Most environments end up using more than one.

Route 1: an SDK sending directly

If a service already uses an OpenTelemetry SDK, you can usually redirect it with the standard environment variables and no code change. The endpoint and any credentials come from your own installation, so treat the value below as a placeholder. 

export OTEL_SERVICE_NAME=checkout-api
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=prod,service.version=4.12.0
export OTEL_EXPORTER_OTLP_ENDPOINT=https://<your-wrentrix-otlp-endpoint>
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_COMPRESSION=gzip

Set OTEL_SERVICE_NAME deliberately. The service map, the endpoint tables and every later alert are keyed on service identity, so a pod that reports unknown_service:java produces a map nobody can read. Agree a naming convention before rollout, because renaming a service later splits its history.

Route 2: through an existing collector

Wrentrix works alongside existing collectors. If you already run an OpenTelemetry Collector, add one more exporter and keep whatever you have. This also handles SDKs that default to gRPC: the collector receives gRPC and forwards over HTTP. 

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}

exporters:
  otlphttp/wrentrix:
    endpoint: https://<your-wrentrix-otlp-endpoint>
    compression: gzip

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/wrentrix]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/wrentrix]

Sending the same data to two backends during an evaluation is simply a second exporter in the exporters list. That lets you compare without touching applications. The guide on running the collector on Kubernetes covers agent and gateway deployment patterns.

Route 3: the OpenTelemetry operator

For Kubernetes and OpenShift workloads with no instrumentation at all, Wrentrix uses the OpenTelemetry operator for auto-instrumentation. In the upstream model, an Instrumentation resource describes where to export and how to propagate context, and an annotation on the pod template selects the language. 

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: default
spec:
  exporter:
    endpoint: http://otel-collector.observability.svc:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-java: "true"

Injection happens when a pod is created, so running pods are unaffected until their next restart. The operator guide walks through language differences, including exporter protocol defaults, which vary between language images and are worth checking against the docs for your operator version.

For Java and .NET, Wrentrix adds its own method-level tracing through an admission webhook, which is covered separately. Spans tell you which service and which call. Method data tells you where inside the process the time went.

Keeping traces connected

The most common reason a tracing rollout disappoints is broken context. The tree stops at a boundary and you get two short traces in place of one long one. Check these places first:

  • Gateways and proxies. A component that terminates a request and opens a new one must forward the traceparent header. Most do so if configured, some do not by default.
  • Mixed propagation formats. Older services may use a different trace header format. The propagator list has to include a format both sides understand. The W3C Trace Context guide explains the header format.
  • Asynchronous work. Message queues need context injected into message headers. Whether a consumer span should be a child of the producer or only linked to it depends on the workflow, and instrumentation libraries differ.
  • Thread pools. Hand-written executors in Java and .NET can lose ambient context, which appears as orphaned spans.

The service map helps here. An edge you expect but cannot see, or a service with heavy traffic and no callers, is usually a propagation gap and not a mysterious new client.

Why sampling is decided at the tail

Storing every span from a busy production system is expensive, and most of that data describes requests that went fine. Head sampling decides at the start of a request, by probability, before anyone knows how it will turn out. It keeps a proportional share of everything, which means it discards the same proportion of your rarest failures.

Tail-based sampling waits until the trace is finished, then decides. Here is a worked example with invented numbers. A system produces 1,000 traces per second, of which 0.5% have an error and 1% are slower than a threshold, with no overlap assumed. Keep all errors (5 per second), all slow traces (10 per second) and 5% of the remaining 985 (49.25 per second). You store about 64 traces per second, or 6.4% of the volume, and you keep every trace you would want during an incident.

The trade-offs are real. Spans have to be buffered until a decision is made, which costs memory. All spans of one trace need to reach the same decision point. Late-arriving spans can miss the decision, so long-running or asynchronous traces need more thought. If you use the sampling policies in an upstream collector, a keep-rule set looks like this: 

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: keep-errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: keep-slow
        type: latency
        latency:
          threshold_ms: 1000
      - name: baseline
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

Sampling also affects any metric derived from spans, which is one reason to think about it before you build SLOs on the same data. The tail-based sampling guide covers multi-collector routing and how to pick the wait time.

Where tracing is not the right tool

Tracing answers "where did this request spend its time". It does not replace everything else:

  • A saturated node or a noisy neighbour shows in Kubernetes and OpenShift monitoring before it shows clearly in spans.
  • A failure that produces no request, such as a batch job that never started, leaves no trace at all.
  • Spans have their own overhead and their own storage bill. Instrumenting every small function as a span is usually a mistake, and it is the problem method-level histograms were designed to avoid.
  • Attributes can carry sensitive data. SQL statements, URLs with identifiers and custom attributes all flow to the backend. Self-hosting keeps that data in your network, but you still decide what should be collected and who can view it. See security and access control.

Questions about ingest capacity or retention for your specific traffic depend on your data and hardware, and we would rather discuss them with your numbers than print a generic figure. Ask us at info@wrentrix.com.

Frequently asked questions

Do I have to replace my OpenTelemetry instrumentation to use Wrentrix?

No. Wrentrix accepts standard OTLP traces and logs, so any OpenTelemetry SDK can send to it and existing collectors can stay in place. In most cases the change is an endpoint and possibly a credential, either in the SDK environment variables or as an extra collector exporter.

Does Wrentrix accept OTLP over gRPC?

The ingest described for Wrentrix is OTLP over HTTP with protobuf or JSON and gzip. If an SDK defaults to gRPC, set OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf or send it through a collector that receives gRPC and exports with otlphttp. Ask us if your environment cannot do either.

What is the difference between head-based and tail-based sampling?

Head-based sampling decides at the start of a request, so it cannot know whether that request will fail or run slowly. Tail-based sampling waits until the trace has finished and can then keep errors and slow traces while dropping most healthy ones. The cost is buffering spans until the decision is made.

Yes. Logs linked to traces let you move from a slow or failed span to the log lines written during that request. This depends on logs carrying trace context, which OpenTelemetry logging integrations can add, so it is worth verifying on one service first.

Does trace data leave my network?

No. Wrentrix runs on your own OpenShift, Kubernetes or Linux infrastructure, and no telemetry is sent outside. It also runs fully air-gapped, as described on the air-gapped observability page.