What is distributed tracing?

Distributed tracing records the path of a single request through every service it touches, with timing for each step, so you can see where the time went and where an error started.

Reading one trace

Start with what you will actually be looking at. Here is a simplified trace from POST /orders, drawn as a text waterfall. Each bar is a span, indented by its parent: 

trace 4bf92f3577b34da6a3ce929d0e0e4736                       total 1,840 ms

api-gateway     POST /orders              [==============================] 1840 ms
  order-svc     POST /orders              [ =============================] 1815 ms
    order-svc   SELECT customers          [ =]                               22 ms
    order-svc   SELECT customers          [  =]                              21 ms
    order-svc   SELECT customers          [   =]                             23 ms
    pricing     POST /quote               [    =======================  ]  1490 ms
      pricing   cache GET quote:88213     [    =]                             2 ms
      pricing   SELECT price_rules        [     ======================  ]  1460 ms
    order-svc   INSERT orders             [                           = ]    41 ms
    order-svc   publish order.created     [                            =]     6 ms

This one picture gives you several facts that no metric would show directly:

  • Nearly all the time is in one SQL statement inside pricing.
  • order-svc called SELECT customers three times in a row. It is small here but could be an N+1 pattern that gets worse with larger orders.
  • The cache lookup ran first and was probably a cache miss, which explains why the SQL ran at all.
  • The calls ran one after another, so any parallelism the team assumed is not there.

The core concepts

Trace

A trace is the complete record of one request or transaction. All its parts share a 16-byte trace ID.

Span

A span is one timed operation: an incoming HTTP request, an outgoing call, a database query or a piece of business logic. A span has:

  • a name, such as POST /orders or SELECT price_rules
  • a start time and an end time
  • a span ID and the ID of its parent span
  • a kind (server, client, internal, producer or consumer)
  • attributes such as http.response.status_code or db.system.name
  • a status, which can mark it as an error
  • events, such as a recorded exception with a stack trace
  • links to related spans in other traces, used for batch and messaging patterns

Context propagation

This is the part that makes tracing distributed. When service A calls service B, A adds the trace ID and its current span ID to the request. B reads them and makes its own span a child of A's. On HTTP, the standard is the traceparent header: 

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

We break down that format in the W3C Trace Context guide. Most broken traces trace back to a hop where this header was lost.

Client and server spans

A single call between two services creates two spans: a client span in the caller and a server span in the callee. The gap between them is revealing. If the client span shows 900 ms and the server span shows 50 ms, the time was spent outside the server's handler: in the network, a proxy, a connection pool, a queue in front of the server or waiting for a thread.

That gap also explains a frequent complaint: "Our dashboard says the API takes 50 ms but the caller says 900 ms." Both are right.

Patterns to recognise in the waterfall

Shape Likely meaning What to check
One long leaf span Slow query or external call Query plan, indexes, downstream service
Many short identical spans in sequence N+1 query or chatty API Batch the calls, check ORM loading settings
Gap between parent start and first child Time in code not covered by spans, or waiting Method-level data, thread pool, GC pause
Client span far longer than server span Queueing or a connection pool wait Pool metrics, proxy, network
Staircase of retries with the same target Retry behaviour hiding failure Retry policy, timeouts, downstream errors
Child span seeming to start before its parent Clock skew between hosts Check NTP time sync on the nodes
Trace ending abruptly, with no downstream Lost propagation, or uninstrumented service Instrumentation at that hop

How traces, metrics and logs fit together

The three kinds of signal answer different questions:

  • Metrics tell you that something changed and how much. They are cheap to keep for a long time, but they don't explain individual requests.
  • Traces tell you where time and errors occur within a request. They are expensive, so they are usually sampled.
  • Logs tell you what a piece of code reported at a moment. They hold details such as an error message or a business value.

The strength comes from linking them. A log line that carries trace_id takes you straight to its trace, and a trace takes you to the logs for only that request, instead of grepping across a hundred pods. When a logging library has OpenTelemetry integration, this ID is typically added for you.

Sampling

A busy system can generate huge volumes of spans, and keeping everything is rarely worthwhile. There are two main options.

Head-based sampling decides at the start of a trace. With a 10% ratio, 10% of traces get recorded. The decision travels downstream in the sampled flag so all services agree. It is cheap, but it decides before knowing the outcome, so it discards 90% of your errors along with 90% of everything else.

Tail-based sampling decides after the trace is complete, so it can keep all errors and slow requests and only a fraction of the rest. It needs to buffer whole traces, which takes more memory and some architecture work. See the tail-based sampling guide.

An example of how the choice affects you: assume 1,000 requests per second and 0.1% errors, which is one error per second. With 5% head sampling, you keep about one error trace every 20 seconds. If a rare bug affects one request in 100,000 (one every 100 seconds at this rate), you would capture one about every 2,000 seconds, or roughly one every 33 minutes. Tail sampling with a "keep all errors" rule would capture all of them.

What tracing does not do well

Being clear about limits saves disappointment:

  • Async and batch work. A consumer handling a batch of messages from many producers doesn't fit a single parent-child tree. OpenTelemetry uses span links for this, and backends vary in how well they display them.
  • Time inside a span. A 2-second span with no children tells you only that those 2 seconds were spent somewhere within it. Profilers or method-level tools are needed to go deeper.
  • Long-running workflows. A process that runs for hours or days fits poorly in one trace. Consider separate traces connected with links.
  • Uninstrumented components. Legacy systems, mainframes and many third-party services create gaps. The client span still shows how long you waited.
  • Sensitive data. SQL text and URLs can hold personal data. Use parameterised queries, and filter attributes in the SDK or collector.

Getting started

  1. Pick a request path that matters to the business, and that crosses three or more services.
  2. Use auto-instrumentation first. It usually gets you HTTP, database and messaging spans with no code. See getting started with OpenTelemetry.
  3. Set service.name consistently. A trace with three names for one service causes endless confusion.
  4. Check for broken traces at gateways, message queues and thread pool handoffs. Fix propagation before you do anything else.
  5. Add manual spans only where you keep wishing you had them.
  6. Put trace_id into your logs.
  7. Decide on sampling once you know your actual volume.

How Wrentrix uses traces

Wrentrix accepts OTLP over HTTP from any OpenTelemetry SDK and builds a live service and dependency map from it, with endpoint p50/p95/p99, error groups and logs linked to traces. For the "gap inside a span" problem, method-level tracing adds method timing for Java and .NET with no code changes, and by rule for Python and Node.js. See the tracing page for details, or the live demo to click through sample traces.

Frequently asked questions

What is the difference between a trace and a span?

A trace is the whole journey of one request. A span is one timed operation within it, such as an HTTP call or a database query. Spans link to their parents through IDs, forming a tree that shares one trace ID.

What is the difference between tracing and logging?

Logs are separate events written by code, usually with no built-in link to each other across services. Traces are structured records with timing and parent-child relationships across services. Adding the trace ID to logs gives you both views of one request.

How much overhead does distributed tracing add?

It is usually small for typical auto-instrumentation, but it depends on span volume, attribute size, language and the export settings. Overhead grows when you create spans in hot loops or capture large payloads. Measure it in a load test on your own service to get a figure you can trust.

Do I have to change my application code for distributed tracing?

Often not at first. Agents and auto-instrumentation libraries for Java, .NET, Python and Node.js can create spans for common frameworks with no code changes. Business-specific spans and attributes do need code, so most teams add a few over time.

Why are my traces broken into separate pieces?

Almost always because a hop lost the context. Common culprits are a proxy or gateway that doesn't pass the headers, a message queue without propagation, a thread handoff that loses the current context or different services using incompatible propagation formats. Find the last span before the break and look at what its outgoing call carried.