Baselines and anomaly detection learned from your own traffic

The problem with one number

A static threshold assumes that one value separates good from bad for all time. Set the p95 latency alert at 800 ms on an endpoint whose behaviour follows the working week, and you get two failures at once:

Raising the threshold quiets the first two rows and hides the last two even more. Lowering it does the opposite. No value fixes both, because the question is not "is this number high", it is "is this number high for now". That distinction also lies behind much of the alert fatigue that teams try to solve with routing rules.

Why hour of week

Traffic in business systems has a daily shape and a weekly shape, and they interact. Monday morning differs from Saturday morning. Friday afternoon differs from Tuesday afternoon. A baseline keyed on hour of day alone blends weekdays and weekends. A baseline keyed on the hour of the week keeps them apart.

The structure is simple arithmetic: 7 days times 24 hours gives 168 slots. Monday 09:00 to 10:00 is one slot, and Sunday 03:00 to 04:00 is another. Each slot learns what normal looks like for that hour only, based on your earlier weeks. The new value at 10:15 on a Monday is compared with earlier Mondays at 10:00, not with last night.

This has a direct consequence for how much history you need. Each slot gets one hour of fresh observation per week. Four weeks of data means each slot has seen its own hour four times, not 672 times. That is the main reason a baseline system cannot be fully trustworthy on its first day, and the honest expectation to set for your team is that it improves over several weeks.

What Wrentrix baselines

Wrentrix learns hour-of-week baselines from your own history, and that history stays in your installation. Baselines exist at two levels, which turn out to be useful for different reasons.

Endpoint baselines work at the level your users feel. They tell you POST /orders is outside its normal range right now. This is the level most useful for detection, because a change here is likely to matter to someone.

Method baselines work inside the process. They are built from 17-bucket histograms per method, and they answer the follow-up question: of the hundreds of methods behind this endpoint, which ones moved? They are mostly useful for diagnosis. A method that takes 700 ms every hour of every week is not interesting, however slow it looks in a sorted table. A method that normally takes 12 ms and now takes 90 ms is interesting, though it would never reach the top of a sorted table.

An anomaly score then expresses how far the current behaviour sits from the learned baseline. The score lets you compare and rank deviations. It is not a probability that something is broken.

How to reason about a deviation

A deviation can be measured several ways, and it helps to understand why they differ. Here is a simplified example, not a description of the internal formula. Suppose that for one slot, an endpoint's p95 over the past weeks was:

Week 1: 210 ms
Week 2: 190 ms
Week 3: 205 ms
Week 4: 195 ms
Mean: 200 ms

The current value is 260 ms. In absolute terms, that is 60 ms above normal. Relative to the mean, it is 30% above. Relative to the spread, it is far outside anything seen in this slot, since values only ranged from 190 to 210. Now consider an endpoint whose same slot historically ranged from 100 to 400 ms, with the same 200 ms mean. The same 260 ms is unremarkable there. That is why a useful score has to consider the normal variability of each slot, and not only a percentage from the mean.

Two further practical adjustments apply to any baseline system, and they are worth asking about in any tool:

• Low traffic. A slot with six requests produces a p95 that jumps around for no reason. An alert based on it would be noise, so low-volume data needs to count for less.
• Direction. A latency drop to a tenth of normal looks like an improvement, but it can mean requests are failing fast. The same applies to throughput: a sudden fall may matter more than a rise. Do not assume low is always good.

Situations where a baseline gets it wrong

Baselines fail in predictable ways. Knowing them is part of using the feature well.

• Annual and irregular events. A tax deadline, a public holiday that moves each year, a Ramadan schedule shift, or a sale event has no matching hour in the recent weekly pattern. An hour-of-week model sees it as anomalous, because in the terms it knows, it is. Expect those flags and plan for them.
• Slow degradation. If latency creeps up 2% a week for six months, each week looks close to the last. The baseline moves with it. This is the classic blind spot, and it is the reason baselines belong alongside fixed objectives. An SLO catches what a self-adjusting baseline absorbs.
• Legitimate shifts. After a real architecture change, last month's normal no longer applies. Expect a period of anomalies while new history builds.
• Baselines learned from an incident. A long incident becomes part of the history for those slots, which can make similar behaviour next week look less unusual.
• Time zone and daylight changes. Traffic follows people, so a clock shift can offset the pattern. Türkiye has stayed on one offset year-round since 2016, but a system serving users across Europe sees a seasonal shift, and it pays to keep this in mind.

Baselines and SLOs answer different questions

Teams sometimes ask which one to alert on. They do different jobs:

A workable default is to page on burn rate and use anomalies as context during investigation and as lower-urgency signals, then promote specific anomaly alerts to a page only once you have seen them be right several times. Anomalies also feed the next stage: the problem engine and root-cause analysis use topology to group anomalous signals into one incident so you don't receive forty separate notifications.

Using baselines during onboarding

Some advice for the first month, based on how the mathematics works and not on any claim about your environment:

1. Connect the services and let history accumulate before using anomaly alerts as a paging source. See Kubernetes and OpenShift monitoring for how quickly a cluster starts reporting.
2. Use the anomalies to learn during those weeks. Review them against what you know happened. Take note of which were real.
3. Write down your known irregular events in advance, particularly batch windows and month-end processing. Monthly cycles are not represented in an hour-of-week slot, so a month-end process will look unusual unless it happens to fall the same way each time.
4. Introduce SLOs in parallel, since they are useful from the first day and do not depend on learned history.

Questions about how long a specific baseline needs before it is usable for your traffic are best answered against your own data. The live demo shows the shape of the output using sample data, and we can talk through specifics if you ask us.

Frequently asked questions

What is an hour-of-week baseline?

It is a learned expectation for each of the 168 hours in a week, so Monday 10:00 and Sunday 03:00 each have their own idea of normal. The current value is compared with its own slot and not with a global average. This handles daily and weekly traffic patterns that a single threshold cannot.

How long does it take to learn a baseline?

Each hour-of-week slot only gets one new hour of data per week, so baselines improve over a period of weeks and not hours. The first week gives very little to compare with. Plan on several weeks before trusting anomaly signals enough to page on them, and use SLOs for paging in the meantime.

Does anomaly detection send my data to an external model?

No. Baselines are learned from your own history inside your own installation, and no telemetry leaves your network. Nothing about this feature depends on an internet connection.

Should I replace all static thresholds with anomaly detection?

No. Fixed limits still make sense for hard boundaries such as disk space, certificate expiry and contractual objectives. Baselines drift along with slow degradation, so they should work alongside fixed targets and not replace them.

Why did I get an anomaly during a planned campaign or holiday?

An hour-of-week model only knows the recent weekly pattern, so an event that moves through the year has no matching history. The system is correct that the traffic is unusual, even though you expected it. Telling your on-call team about those events in advance avoids wasted investigation.