Getting started with OpenTelemetry

This walkthrough takes one small Python service from no instrumentation to useful traces, with the settings that carry over to Java, .NET and Node.js. Allow about an hour.

OpenTelemetry (often shortened to OTel) is an open-source project that gives you vendor-neutral APIs, SDKs, a wire protocol called OTLP and a Collector for traces, metrics and logs. It does not store or display anything. You point it at a backend of your choice. The practical gain is that you instrument once and can later change backends by changing configuration.

Maturity differs by language and signal, so check the status tables on opentelemetry.io for your stack.

Step 1: A service to instrument

Create a small service using only the Python standard library, plus one OpenTelemetry package for the server side. It has an endpoint that calls another HTTP endpoint, which gives us a parent-child relationship to look at.

# app.py
import json
import random
import time
from socketserver import ThreadingMixIn
from urllib.request import urlopen
from wsgiref.simple_server import WSGIServer, make_server

from opentelemetry.instrumentation.wsgi import OpenTelemetryMiddleware


class ThreadingWSGIServer(ThreadingMixIn, WSGIServer):
    daemon_threads = True


def respond(start_response, payload):
    start_response("200 OK", [("Content-Type", "application/json")])
    return [json.dumps(payload).encode()]


def app(environ, start_response):
    sku = environ["PATH_INFO"].rsplit("/", 1)[-1]
    if environ["PATH_INFO"].startswith("/price/"):
        time.sleep(random.uniform(0.01, 0.05))
        return respond(start_response, {"sku": sku, "price": 42})
    with urlopen(f"http://localhost:8000/price/{sku}", timeout=2) as response:
        quote = json.load(response)
    return respond(start_response, {"status": "ok", "quote": quote})


if __name__ == "__main__":
    make_server("", 8000, OpenTelemetryMiddleware(app), server_class=ThreadingWSGIServer).serve_forever()

Set up a virtual environment:

python3 -m venv .venv
source .venv/bin/activate
pip install opentelemetry-instrumentation-wsgi

Step 2: Add zero-code instrumentation

The opentelemetry-distro package gives you a launcher that wraps your program. The opentelemetry-bootstrap command then detects your installed libraries and installs matching instrumentation packages. With a mainstream web framework, that alone produces server spans. The standard library's server has no such hook, which is why the code above wraps the app in the WSGI middleware; the outgoing urllib call is instrumented with no code at all.

pip install opentelemetry-distro opentelemetry-exporter-otlp-proto-http
opentelemetry-bootstrap -a install

Run with console output first, so you can see what is produced without any backend:

export OTEL_SERVICE_NAME=order-demo
export OTEL_TRACES_EXPORTER=console
export OTEL_METRICS_EXPORTER=none
export OTEL_LOGS_EXPORTER=none
opentelemetry-instrument python app.py

In another terminal:

curl http://localhost:8000/order/abc

You should see JSON spans printed in the service's terminal. Look for three of them: a server span for /order/abc, a client span for the outgoing urllib call and a server span for /price/abc. All three share a trace_id, and each parent_id points at its caller. If they do, you have working propagation, even though this example happens to call itself.

Step 3: Send to a real backend over OTLP

OTLP can run over gRPC (default port 4317) or HTTP (default port 4318). HTTP is usually easier to get through proxies and firewalls, and it is what we will use here. When you set the base endpoint variable, the SDK appends /v1/traces for traces.

export OTEL_SERVICE_NAME=order-demo
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_EXPORTER_OTLP_COMPRESSION=gzip
export OTEL_RESOURCE_ATTRIBUTES=service.version=1.0.0,deployment.environment.name=staging
export OTEL_METRICS_EXPORTER=none
export OTEL_LOGS_EXPORTER=none
opentelemetry-instrument python app.py

If your backend needs an auth token, OTEL_EXPORTER_OTLP_HEADERS=authorization=Bearer%20<token> sets the header. Values must be URL-encoded, hence %20. Keep tokens in a secret, not in a Dockerfile.

These variables come from the OpenTelemetry specification and work across most language SDKs, which is why they are worth learning:

Variable Purpose
OTEL_SERVICE_NAME Sets service.name, the most important attribute you will set
OTEL_RESOURCE_ATTRIBUTES Comma-separated key=value pairs that describe the service
OTEL_EXPORTER_OTLP_ENDPOINT Base URL for the OTLP exporter
OTEL_EXPORTER_OTLP_PROTOCOL grpc, http/protobuf or http/json; SDK support varies
OTEL_EXPORTER_OTLP_HEADERS Headers for auth
OTEL_TRACES_SAMPLER For example parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG The sampler's argument, for example 0.25
OTEL_PROPAGATORS Defaults to tracecontext,baggage
OTEL_SDK_DISABLED Set to true to turn the SDK off

Step 4: Add a manual span where it counts

Automatic spans cover the framework edges. They do not know that your code applies a discount. Add spans with the API only where your team needs that view.

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer("order-demo.pricing")


def apply_discount(order_total, customer_tier):
    with tracer.start_as_current_span("apply-discount") as span:
        span.set_attribute("app.customer.tier", customer_tier)
        try:
            return calculate_discount(order_total, customer_tier)
        except ValueError as error:
            span.record_exception(error)
            span.set_status(Status(StatusCode.ERROR))
            raise

The opentelemetry-instrument launcher configures the SDK for you, and this code only uses the API. Libraries should depend on the API alone, too: without a configured SDK, the API does nothing, at almost no cost.

Some habits to keep from the start:

  • Name spans for the operation, with low cardinality: apply-discount, not apply-discount-customer-88213. IDs go in attributes.
  • Use semantic convention names when one exists, such as http.request.method, http.response.status_code, url.path and http.route. Use your own prefix, such as app., for business attributes.
  • Don't put personal data in attributes unless you have a reason and a policy for it.
  • Don't wrap hot loops in spans.

Step 5: The same idea in other languages

The zero-code approach exists for other languages too. The commands differ, but the environment variables stay the same.

Java. Download the agent JAR from the OpenTelemetry Java instrumentation releases and attach it:

export OTEL_SERVICE_NAME=order-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
java -javaagent:./opentelemetry-javaagent.jar -jar order-api.jar

Node.js. Install the auto-instrumentation meta package and load it before your application:

npm install --save @opentelemetry/api @opentelemetry/auto-instrumentations-node
export OTEL_SERVICE_NAME=order-web
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
node --require @opentelemetry/auto-instrumentations-node/register app.js

.NET. OpenTelemetry .NET Automatic Instrumentation installs through a script from its releases page, which sets the profiler environment variables. On Kubernetes, the operator can do this for you, as explained in the operator guide.

Step 6: Fix what you will probably hit first

Symptom Likely cause Fix
No spans at all Wrong endpoint or a blocked port Switch temporarily to the console exporter to separate SDK from network problems
HTTP 404 from the exporter A signal path doubled up or missing With OTEL_EXPORTER_OTLP_ENDPOINT, give only the base URL. OTEL_EXPORTER_OTLP_TRACES_ENDPOINT needs the full path
gRPC exporter hitting port 4318 Protocol and port do not match Use 4317 for gRPC and 4318 for HTTP
Service appears as unknown_service service.name is not set Set OTEL_SERVICE_NAME
Separate traces for each service Lost propagation, or differing propagators Check that traceparent arrives at the next hop and set the same OTEL_PROPAGATORS
Spans disappear on short jobs Batch exporter did not flush on exit Shut down the provider cleanly at exit
Missing spans from a library Instrumentation package not installed Re-run bootstrap after adding dependencies
Forked workers lose spans Pre-fork application servers Check the OpenTelemetry Python docs on working with fork process models

Step 7: What to do next

  • Add a Collector. Sending straight from the SDK is fine for learning. In production, a Collector gives you a place for batching, retries, filtering and routing. See running the Collector on Kubernetes.
  • Decide on sampling. Start at 100% in staging. In production, make your decision from real volume. Tail-based sampling covers the options.
  • Connect logs. Add the trace ID to your log format so that logs and traces connect.
  • Standardise attributes. Agree with other teams on service.name, service.namespace and environment values, because these are painful to fix later.

Sending to Wrentrix

Wrentrix accepts OTLP traces and logs over HTTP, in protobuf or JSON and with gzip, so the variables above apply directly. It works alongside existing collectors. The distributed tracing page describes what you get from that data. OTLP over gRPC and OTLP metrics are not part of what we list here, so ask us if you depend on them.

Frequently asked questions

Is OpenTelemetry a monitoring tool?

No. It is a set of APIs, SDKs and a Collector for producing and shipping telemetry, plus the OTLP protocol. You still need a backend to store, query and alert on the data. That separation is what makes the instrumentation portable.

Should I use OTLP over gRPC or HTTP?

Both are fine for most systems. HTTP on port 4318 tends to pass more easily through proxies and load balancers, and gRPC on port 4317 can be somewhat more efficient on persistent connections. Check what your backend or collector accepts, then use the protocol it supports.

What is the difference between the OpenTelemetry API and SDK?

The API defines the calls that your code and libraries make, and does nothing by itself. The SDK is the implementation that samples, processes and exports the data. Libraries depend on the API only, while an application configures the SDK or lets an agent do it.

Can I use OpenTelemetry without changing my code?

Yes, for many frameworks in Java, .NET, Python, Node.js and others. A launcher, agent or Kubernetes operator instruments common libraries automatically. You will probably add some manual spans later for business operations that a framework can't see.

Do I need an OpenTelemetry Collector?

Not to start, since SDKs can export directly to a backend that speaks OTLP. In production, most teams add one for batching, retries, enrichment, filtering and sampling. It also keeps backend credentials and routing out of your application configuration.