Log Pattern Clustering Inside Your Network

From lines to patterns

Look at these four lines from an imaginary service:

2026-11-03T14:02:11Z WARN  pool=orders-db acquire took 2041ms, waiting=17
2026-11-03T14:02:11Z WARN  pool=orders-db acquire took 3380ms, waiting=23
2026-11-03T14:02:12Z ERROR order 88213 failed: timeout after 5000ms
2026-11-03T14:02:12Z ERROR order 88219 failed: timeout after 5000ms

To a human, that is two messages, each appearing twice. To a text search, it is four different strings. Clustering recovers what the human sees by separating the fixed text from the parts that vary:

WARN  pool=orders-db acquire took <*>ms, waiting=<*>     count: 2
ERROR order <*> failed: timeout after <*>ms              count: 2

Scale that up and the problem changes shape. A busy service may emit millions of lines per hour, but those lines usually come from a modest number of log statements in the code. Instead of scrolling through a stream, you are reading a table of a few hundred templates with counts. The log pattern clustering guide describes the algorithms behind this approach and how to evaluate the results, independent of any product.

The question patterns answer in an incident

Full-text search is only useful if you know what to type. When you get paged for an unfamiliar failure, you don't. You are looking for an unknown string, and the most useful query would be something like "show me what is different". Patterns make it possible to ask.

Wrentrix clusters pod logs, host logs and OTLP logs into patterns, and flags patterns that are new and patterns that are rising. Those two flags answer different questions, and each catches a class of failure that the other misses.

Flag	What it means	Typical cause	What to do with it
New	A message shape not seen in earlier history	Fresh deploy, new code path, a dependency returning an unfamiliar error	Compare the first-seen time with recent changes
Rising	A known message shape, at a higher rate than usual	Retry storms, growing saturation, a failing dependency, an expired token	Find what started the climb and what it correlates with
Stable and noisy	Always present at high volume	Chatty debug statements, harmless warnings	Usually ignore during incidents, consider fixing afterwards

The third row matters as much as the first two. Almost every production system has lines such as WARN connection reset by peer that have appeared thousands of times a day for years. A text search for "error" returns them first. A view sorted by change pushes them out of your way, which is what actually shortens the first ten minutes of an incident. Reducing MTTR looks at where those minutes normally go.

A worked example

Suppose (invented numbers, for illustration) that an API starts returning 500s at 14:02, right after a deploy at 13:58. Raw error volume is up, but this service always logs plenty of errors, so the graph alone says little. The pattern list shows:

A new pattern, first seen at 13:59: Failed to parse config key <*>: unknown field <*>, count 412.
A rising pattern, up from its usual rate of about 3 per minute to about 300 per minute: order <*> failed: timeout after <*>ms.
A stable pattern, unchanged: connection reset by peer, about 900 per minute all week.

The stable pattern would have dominated a simple search, even though it is irrelevant. The rising pattern is the symptom. The new pattern is the likely lead, since it appeared one minute after the deploy and three minutes before the customer impact. You still have to confirm it, because a new pattern is a correlation and not a proof. But you have gone from millions of lines to one sentence that you can check against a config diff. Wrentrix can also put that deploy on the same timeline through change correlation, and the approach is laid out in general terms in what changed: change correlation.

Where the logs come from

You do not need one logging standard across the estate to get value. Wrentrix takes logs from three sources:

Pod logs, from Kubernetes and OpenShift workloads, without changing the application.
Host logs, from Linux servers, including the systemd journal. See Linux server monitoring.
OTLP logs, sent through any OpenTelemetry SDK or collector, which is the route that carries trace context most directly.

A collector pipeline that forwards logs alongside traces could look like this. Replace the placeholder endpoint with the one from your installation.

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}

exporters:
  otlphttp/wrentrix:
    endpoint: https://<your-wrentrix-otlp-endpoint>
    compression: gzip

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/wrentrix]

The link to traces is what turns a pattern into a request you can investigate. From a slow span in distributed tracing you can reach the log lines written during that request, and that needs trace_id and span_id on the log record. OpenTelemetry logging integrations add these fields for you. If your application writes plain text to stdout, the IDs are only present if your log format includes them, so check a sample before counting on the link. For the common Java logging libraries and .NET's logging abstractions, this is usually a small configuration change and not a rewrite.

Writing logs that cluster well

Clustering works with any logs, but some habits help it and others quietly hurt it. These apply whichever tool you end up using:

Keep the message text fixed and put the values in fields. payment failed with order_id=88213 as a field clusters perfectly. A message built by concatenating a full customer record does not.
Do not put a stack trace in the message of a single-line record. Multi-line stack traces split across records unless your collection pipeline handles them. Configure multi-line handling or emit them as structured fields, such as the OpenTelemetry exception.stacktrace attribute.
Avoid variable word order. Interpolated natural-language sentences that rearrange themselves become several patterns for one event.
Log the error class, not only the message. A class name is stable, whereas a message may contain locale-specific text, such as a Turkish or English exception string on different hosts, which turns one failure into two patterns.
Keep identifiers out of logs when they do not need to be there. National ID numbers, card numbers and tokens are not made safe by keeping logs on-premise. Pattern clustering may mask variable parts in its display, but you should not treat that as a redaction control.

That final point deserves emphasis. Having a log platform inside your own network removes the question of a third party holding those logs. It does not remove your obligations regarding what is in them, as discussed on the data residency page.

What pattern analysis will not do

It will not find a problem that writes nothing. A deadlocked process or a silently dropped message may leave no log line, new or otherwise. Baselines on throughput and latency are how you see those.
"New" needs history. In the first days after onboarding a service, many patterns are new simply because they have not been seen before. Expect more flags at the start, and don't judge the signal during that period.
Rare but regular events can look new. A monthly batch job may produce its normal output and still be flagged if your retained history is shorter than its cycle.
Log statements that vary wildly, such as dumps of whole payloads, can cluster poorly. The fix is on the application side.
Patterns do not replace search. Once a pattern points to an order ID or a pod, you will still filter specific lines.

Retention and storage sizing for logs depend on your volume. We do not publish a generic figure, so ask us with your daily volume in hand.

Frequently asked questions

What is log pattern clustering?

It groups log lines that came from the same statement by separating fixed text from variable values such as IDs and durations. Millions of lines become a much shorter list of templates with counts. That list is far easier to read and compare over time than the raw stream.

Do I need to install a separate log agent?

Pod and host logs come through the Wrentrix agent that you set up for a cluster or Linux host. OTLP logs can come from any OpenTelemetry SDK or collector, including one you already run. You can use both, depending on the workload.

How does Wrentrix detect new or rising log patterns?

Each line is assigned to a pattern, and patterns are compared with their earlier history. A pattern that has no earlier history is flagged as new, and a known pattern that is appearing more than usual is flagged as rising. This is a statistical signal, so it points at where to look and does not state a cause.

Can I correlate logs with traces?

Yes, when a log record carries trace context. OTLP logs from OpenTelemetry integrations normally include trace_id and span_id, while plain stdout logs only carry them if your format writes them. Test this on one service early in a rollout.

Is on-premise log storage enough to keep personal data safe?

No. Keeping logs inside your network removes third-party transfer, but personal data in a log is still personal data under rules such as KVKK or GDPR. Limit what you log, control who can read it, and get your own legal advice, since this page is not legal advice.