Synthetic monitoring with HTTP(S), TCP and TLS expiry checks

The failure that leaves no evidence

Picture this invented scenario. A load balancer change at 01:40 sends all traffic for an internal portal to an empty pool. Every request fails before it reaches an application. The application's error rate is zero and its latency is excellent, since it is receiving no requests. Traces are empty. Logs are quiet. All the application dashboards are green until the first employee logs in at 08:30.

Telemetry from inside a service can only report on requests that arrived. For anything in front of it, such as DNS, routing, firewalls, load balancers, TLS termination and API gateways, you need a client that actively tries the path and reports what happened. That is the job of a synthetic probe.

Wrentrix provides HTTP(S) and TCP probes, with TLS certificate expiry and response-time alerts. That scope is deliberately simple. It answers "can this be reached, is it responding correctly, and how quickly". It does not script multi-step browser journeys. If you require that, ask us what fits. For measurements of real browser experience, use real user monitoring.

Which probe for which question

TCP probes are undervalued. Non-HTTP dependencies such as databases, message brokers and SMTP relays are often on the far side of a firewall rule that someone may change. A TCP connect from an application zone to the listener catches the "someone removed the firewall rule" class of outage, and catches it before the next connection pool refresh surfaces it as a mysterious application error.

Designing a useful health endpoint

An HTTP probe is only as honest as the URL it hits. Three levels are commonly used, and each has a different failure mode:

• **A static path or shallow /healthz.** It shows that the process and the path in front of it work. It stays green when the database is gone.
• A deep check that exercises dependencies. It detects more, but a probe that reaches into five systems can be noisy and can add load. It may also report a failure in an optional dependency as a full outage.
• A real read-only business endpoint. For instance, a public catalogue lookup. It is the closest to user experience, but needs a stable expected response.

For most services a shallow liveness path is the wrong choice for an external probe, since it is what your orchestrator already checks. A Kubernetes liveness probe restarts a container, and a readiness probe removes it from service endpoints. Neither of them checks the route from outside the cluster. An external synthetic check should go through the front door: same hostname, same TLS termination, same gateway, as your users.

Never use production credentials with broad access in a probe. If a check requires a login, create a dedicated read-only account with a narrow scope, and expect your auditors to ask about it. Probes that write data, such as creating a test order, need a very good reason, along with a way to clean up.

Getting the alert logic right

Network paths have brief glitches. A naive rule of "alert on one failed check" produces pages for nothing, and teams soon stop believing them. Some general tactics:

Require consecutive failures. Worked example: with a probe every 60 seconds and a rule needing 3 consecutive failures, you learn of a real outage roughly 3 minutes after it starts. That is your detection time, and it is a deliberate trade-off. Shorten the interval, and you get faster detection with more load and more transient noise.

Think about the probability of false alarms. As an illustration, if independent transient failures happen on 0.1% of checks, then one failed check occurs about 1.44 times a day on a 60-second probe, since 1,440 × 0.001 = 1.44. Requiring 2 in a row, under the independence assumption, brings the chance per check down to 0.000001, which is about one alarm every 694 days. Real failures aren't perfectly independent, so the real number will be worse, but the direction is clear. A single-failure rule pages daily, while a two-failure rule almost never does.

Set response-time alerts around what matters. Use an alert level with some margin over normal behaviour. A spike in a single probe tells you little about latency, whereas a sustained rise is meaningful. Use SLOs for the customer-facing promise, and probes for reachability.

Connect alerts to action. A TLS warning 30 days out should open a ticket and not wake anyone. A front-door failure should page. Wrentrix's alert wizard, channels and runbooks are where you route those differently, and the alert fatigue guide has the wider reasoning.

TLS certificate expiry deserves its own plan

Expired certificates are among the most preventable outages, yet they keep happening because the failure is silent until the moment it is total. Internal certificates, which issuing authorities and public transparency tooling can't help with, are particularly likely to be forgotten.

You can check a certificate by hand with standard tooling:

echo | openssl s_client -connect portal.example.internal:443 -servername portal.example.internal 2>/dev/null \
  | openssl x509 -noout -enddate -issuer -subject

And as a pass or fail check, for example whether it is still valid in 30 days (2,592,000 seconds):

echo | openssl s_client -connect portal.example.internal:443 -servername portal.example.internal 2>/dev/null \
  | openssl x509 -noout -checkend 2592000

Things that automated expiry checks should account for:

• Every instance, not just the name. A load-balanced hostname can serve an old certificate from one node out of six. A probe that hits the VIP sees whichever node answers.
• The chain, not just the leaf. An expired intermediate fails clients as well, though not all tools point this out.
• Mutual TLS and non-HTTP endpoints. Client certificates used for partner integrations are classic surprises, because nothing public depends on them and the owner may have changed roles. Inventory these separately.
• Tiered warning windows. 30 days for a ticket, 14 for a louder reminder, 7 for an escalation are reasonable starting points. Align them with how long your organisation's certificate request process actually takes. If it takes three weeks to get a certificate issued, then a 14-day warning is too late.

The TLS expiry guide goes into inventorying and renewal processes.

Where a probe runs changes what it tells you

A probe measures a path from one point to another, so placement decides what you learn. A probe inside the same subnet will prove that the service is up, and very little about the routes your users take. Probing from the application zone tests firewall paths between zones. Probing through the same front door that users use tests load balancing and TLS.

There is an honest limitation for any on-premise system here. A probe that runs inside your network cannot fully test the experience from the public internet. DNS as seen externally, your upstream provider and the edge path may all be broken while an inside view looks perfect. If you have public-facing services, you may want something outside your network to verify that path, and that is a choice to weigh with your security team, since it is one of the few cases in which an outside vantage point is the point. Details of where Wrentrix probes run and how many locations you can place them in are worth raising when you talk to us.

Using probes with the rest of the telemetry

A probe failure is a starting point. Some typical combinations and what they suggest:

The first row is the scenario from the top of this page, and a sudden fall to zero traffic combined with a failed probe is an unusually clear signal.

Frequently asked questions

What is the difference between synthetic monitoring and real user monitoring?

Synthetic monitoring sends scheduled test requests, so it works even with no users and gives consistent measurements. Real user monitoring collects what actual browsers experienced, across real devices and networks. Most teams want both, because each catches problems the other misses.

What can Wrentrix synthetic probes check?

They check HTTP(S) and TCP endpoints, with alerts for TLS certificate expiry and response time. Scripted multi-step browser journeys are not part of what we describe here, so ask us if you need them. A TCP probe is often the simplest way to verify a firewall path to a non-HTTP service.

How early should a TLS expiry alert fire?

A common pattern is about 30, 14 and 7 days with rising urgency. The earliest warning needs to be longer than your real certificate request process, so a slow internal process needs an earlier first alert. Short-lived certificates that renew automatically need tighter windows and an alert when renewal fails.

How often should an uptime probe run?

For important user-facing services, 30 to 60 seconds is common, with an alert after a few consecutive failures. Your detection time is about the interval multiplied by the number of failures required. Less important systems can use longer intervals, which reduces noise and load.

Can a probe inside my network replace external uptime checks?

Not for public services. An inside probe cannot see external DNS or your upstream provider, and it will keep passing while the public path is broken. For internal and closed-network systems, an inside probe is exactly the right vantage point.