Log pattern clustering explained

Nobody reads ten million log lines, but ten million lines might contain only a few hundred distinct messages. Clustering recovers those messages, and the useful signal is which ones are new or suddenly more frequent.

The idea, in five lines

Log lines are mostly a fixed message filled in with variable values: 

Connection to 10.4.2.17:5432 timed out after 3000ms
Connection to 10.4.2.18:5432 timed out after 5000ms
User 8812 logged in from 10.1.1.9
User 1044 logged in from 10.1.1.4
Payment 7731 declined for reason insufficient_funds

Clustering these gives templates, also called patterns: 

Connection to <IP> timed out after <NUM>      (2)
User <NUM> logged in from <IP>                (2)
Payment <NUM> declined for reason insufficient_funds   (1)

A template is close to the format string in the source code, recovered from output alone. This is sometimes called log parsing or template mining. Why does it help? Volume stops being the obstacle. Instead of scrolling, you can ask which templates appeared for the first time today, which tripled after the deployment, and which disappeared.

A small working implementation

This Python sketch has the core of the technique: mask the obvious variables, group by a coarse key, then merge lines that are similar enough. It is deliberately simple, and is meant for learning rather than for production use. 

import re

MASKS = [
    (re.compile(r"\b[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\b"), "<UUID>"),
    (re.compile(r"\b\d{1,3}(?:\.\d{1,3}){3}(?::\d+)?\b"), "<IP>"),
    (re.compile(r"\b0x[0-9a-fA-F]+\b"), "<HEX>"),
    (re.compile(r"\b\d+(?:\.\d+)?(?:ms|s|MB|KB)?\b"), "<NUM>"),
]


def mask(line):
    for pattern, token in MASKS:
        line = pattern.sub(token, line)
    return line


class PatternMiner:
    def __init__(self, threshold=0.6):
        self.threshold = threshold
        self.groups = {}

    def add(self, line):
        tokens = mask(line).split()
        if not tokens:
            return
        key = (len(tokens), tokens[0])
        clusters = self.groups.setdefault(key, [])
        best, best_score = None, 0.0
        for cluster in clusters:
            same = sum(a == b for a, b in zip(cluster["tokens"], tokens))
            score = same / len(tokens)
            if score > best_score:
                best, best_score = cluster, score
        if best is None or best_score < self.threshold:
            clusters.append({"tokens": tokens, "count": 1})
            return
        best["tokens"] = [a if a == b else "<*>" for a, b in zip(best["tokens"], tokens)]
        best["count"] += 1

    def patterns(self):
        rows = [c for clusters in self.groups.values() for c in clusters]
        return sorted(rows, key=lambda c: -c["count"])


miner = PatternMiner()
for line in [
    "Connection to 10.4.2.17:5432 timed out after 3000ms",
    "Connection to 10.4.2.18:5432 timed out after 5000ms",
    "User 8812 logged in from 10.1.1.9",
    "User 1044 logged in from 10.1.1.4",
    "Payment 7731 declined for reason insufficient_funds",
    "Payment 7732 declined for reason card_expired",
]:
    miner.add(line)

for row in miner.patterns():
    print(row["count"], " ".join(row["tokens"]))

The output: 

2 Connection to <IP> timed out after <NUM>
2 User <NUM> logged in from <IP>
2 Payment <NUM> declined for reason <*>

The masks handled numbers and addresses. The merge handled insufficient_funds versus card_expired, which no generic regular expression could have known was a variable.

How the Drain approach scales this

The sketch above scans every cluster in a group. Drain, a well-known online parsing algorithm published in 2017, is built on similar ideas with an organised search structure. Its parse tree has a fixed depth:

  1. The first level splits by token count, since lines of one template usually have the same length.
  2. The next levels split by the leading tokens, on the assumption that the start of a line is more often constant than the end. Tokens containing digits route to a wildcard branch, so variables don't create endless branches.
  3. Leaves hold a short list of clusters, and a new line is compared with those, using the share of tokens in matching positions.
  4. If the best similarity exceeds a threshold, the line joins the cluster and differing positions become wildcards. If not, it starts a new cluster.

The fixed depth keeps the cost for each line low and predictable. It also means the system works as a stream, so it does not need a batch job or a training stage, and it can flag a template as new at the moment of its arrival.

Other published methods include Spell, based on longest common subsequence, and IPLoM, which partitions in steps. Benchmarks in academic literature compare these on labelled datasets, but your own logs matter more than any published ranking.

Where clustering goes wrong

Over-merging

A similarity threshold set too low combines different messages. Take Payment <NUM> declined for reason <*> and Payment <NUM> approved for reason <*>. At least four of six tokens match, so at a threshold of 0.6 they merge into Payment <NUM> <*> for reason <*>. Successful and failed payments now share one count, and a spike in declines becomes invisible.

Decide which words should never turn into wildcards. Severity levels, and words like failed, declined, timeout and denied, are good candidates. Keep severity as part of the cluster key, so an ERROR line never merges into an INFO template.

Over-splitting

With a threshold set too high or with weak masks, one message splits into thousands of patterns, each with a count of one. Typical causes:

  • Variable-length values: Failed for user John and Failed for user Mary Ann have different token counts
  • Identifiers that masks miss, such as short base64 tokens, session keys or file paths
  • Free-text content embedded in messages, like SQL statements or user input

A pattern count that keeps growing roughly in line with log volume is the main warning sign. In a stable service, it should level off. Treat a steadily growing count as a defect in your masks.

Structured logs

Running token clustering across raw JSON gives poor results, since field order and many variable fields defeat position matching. Cluster the message field, and keep other fields as attributes. With a structured logger, the message template may already be available directly: use it and skip the mining.

Multi-line stack traces

Unless they are joined at collection, each line of a stack trace becomes its own log record and its own pattern. Join them as close to the source as possible. In the OpenTelemetry Collector, the filelog receiver has a multiline setting for this: 

receivers:
  filelog:
    include: [/var/log/app/*.log]
    multiline:
      line_start_pattern: '^\d{4}-\d{2}-\d{2}'

After joining, cluster on the exception type and first line, not the whole trace. Deep frames vary with framework versions and would split one failure into many patterns.

Sensitive data

Masks serve privacy too. Templates and samples are often displayed to a wider audience than raw logs. A national ID or card number that misses the mask will end up in a pattern, in plain view. Add explicit masks for the identifiers relevant to your data, and test them against real samples. Remember that a stored raw sample line, kept next to each pattern, needs the same access controls as the logs themselves.

Detecting new and rising patterns

A list of patterns is only moderately interesting. Comparing patterns over time is where the value lies.

New patterns. A template never seen before is a strong lead right after a deployment. Cluster IDs need to stay stable for this to work. If your miner's state resets at each restart, then everything looks new. Persist its state, and key by service, because a message that's routine for one service may be new for another. Expect some noise after every release, as legitimate new messages appear. Rank new patterns by severity and by count.

Rising patterns. Ratios alone mislead at low counts, so compare the count to the normal range. As a worked example, suppose counts are treated as roughly Poisson, where the standard deviation is the square root of the mean:

Pattern Typical per hour Now Ratio (now − mean) ÷ √mean
Rare warning 3 9 3× (9 − 3) ÷ 1.73 = 3.5
Pool timeout 40 900 22.5× (900 − 40) ÷ 6.32 = 136
Routine info 50,000 51,000 1.02× (51,000 − 50,000) ÷ 223.6 = 4.5

Look at the last row. A 2% rise in routine traffic scores higher than the rare warning, so a statistical score by itself would rank it above something that may matter more. Use both: require a minimum absolute count and a minimum ratio, and only then rank by score. Real log counts are burstier than Poisson, so this is a heuristic, not a test of significance.

Seasonality. A nightly batch job's patterns rise at 02:00 every night. Compare with the same hour of the week so that regular jobs don't get flagged each time.

Vanishing patterns. A message that normally appears every minute and then stops may be a dead consumer. Few tools make this easy to see, but it's worth a check during an incident.

Using patterns during an incident

A short routine that works well:

  1. Filter to the affected service and time range.
  2. Sort patterns by change compared with the prior period, not by total volume. The largest pattern is usually routine noise.
  3. Look at new patterns, and check whether their start time matches the last change.
  4. Open a sample line. If it carries a trace_id, jump to the trace for the request involved.

That last step is why log and trace correlation matters. OpenTelemetry's log data model has trace_id and span_id fields, and its SDK log integrations can fill them in.

Patterns are also a far better input for a language model than raw lines, because they compress heavily and retain counts.

How Wrentrix handles patterns

Wrentrix's logs and patterns feature clusters pod, host and OTLP logs into patterns, and flags new and rising ones. Logs are linked to traces, so a rising pattern can lead you to an affected request. The processing runs on your own infrastructure, and nothing leaves the network. Ask us if you have specific masking requirements to check against.

Frequently asked questions

What is log pattern clustering?

It's the process of grouping log lines that come from the same message template, replacing the variable parts with placeholders. Millions of lines may reduce to a few hundred patterns with counts. This lets you compare how often each message occurs over time rather than reading raw lines.

What is the Drain log parsing algorithm?

Drain is an online log parsing method that uses a fixed-depth tree. It routes each line by token count and leading tokens, compares it with a small set of clusters by position-wise similarity, and either joins one or starts a new one. Because it processes a stream with a low per-line cost, it suits continuous operation.

Why do I get too many log patterns?

Usually variable values are not being recognised, so each unique ID or path creates its own pattern. Add masks for your identifier formats, and cluster the message field instead of the full JSON. If the pattern count grows in step with log volume, the masks need more work.

How do I detect anomalies in logs without machine learning?

Mine patterns, then count each per service over time. Flag templates never seen before, and flag those that exceed both a minimum count and a ratio above their same-hour-of-week history. These simple rules catch a large share of useful changes and are easy to explain to responders.