Linux server monitoring for hosts outside the cluster

Why hosts still matter in a Kubernetes shop

Even in estates that have moved heavily to containers, the components with the most state tend to stay on virtual machines or bare metal: the database, the message broker, the file transfer server, the old application nobody wants to containerise. Those are often where the slow request eventually ends up. If your trace shows a 4-second database call, the next question concerns a machine, and the answer shouldn't require opening a separate tool with its own clock and its own login.

Wrentrix treats a Linux host as a first-class source. It collects CPU, memory, disk, network and journal logs, and discovers running processes by technology, so a new host tells you what is on it without someone maintaining an inventory by hand.

Adding a host

The flow mirrors adding a cluster:

1. Add a Linux host in the Wrentrix UI.
2. Download the install script. The token is already inside it.
3. Review the script, then run it on the host.
4. The host appears in about two minutes.

Because the script carries a token, handle it like a credential: don't paste it into a shared wiki, and if you distribute it with configuration management, store it the same way you store other secrets. Closed networks have an image-less agent install option, described on the air-gapped page. Which distributions and versions are supported, and how the agent is packaged for your hardening baseline, are questions to ask us before you plan a rollout.

Reading the four resources properly

Host graphs are simple to collect and easy to misread. Below is how to interpret each resource, along with a command to check the same fact from a shell when you are logged in. These commands are standard Linux tools, and they are worth knowing whichever monitoring system you run. Some, such as iostat, come from the sysstat package, which may not be installed by default.

CPU

High CPU is only a problem if work is waiting. Load average is not a CPU metric on Linux: it counts tasks that are runnable and tasks in uninterruptible sleep, usually waiting for disk. A load of 40 on a 16-core machine might mean a CPU shortage, or it might mean a storage problem with idle CPUs.

vmstat 1 5

Look at the r column (runnable tasks) compared with the core count, and at b (blocked tasks). In the CPU columns, wa is time spent idle while I/O is outstanding, and st is time stolen by the hypervisor. A virtual machine with high st is losing time to its neighbours, and nothing you do inside the guest will fix that.

Memory

"Free memory is low" is usually not a problem. Linux uses spare memory as page cache, and available is the column to read, since it estimates what could be given to applications without swapping.

free -h

The real signs of memory trouble are sustained swapping and the OOM killer. Swap activity shows in the si and so columns of vmstat, and swap in use with no activity is usually harmless. OOM kills are written to the kernel log, which brings us to the journal.

Disk

Two different failures live here: running out of space, and running out of performance. Space is simple until inodes run out while df -h still shows plenty of room, so check df -i too.

iostat -x 1 3

Read await (average I/O time including queueing) and the queue size. Be careful with %util. On a single spinning disk, 100% meant saturated. On SSDs, NVMe and RAID arrays that serve many requests in parallel, a device can show 100% while still having capacity. A latency increase is a better sign than the utilisation figure.

Network

Bandwidth graphs rarely explain application trouble on their own. Retransmissions and connection states tell you more.

ss -s
ss -tan state time-wait | wc -l

Rising retransmissions between an application and its database look like random slowness at the application layer. A large number of TIME-WAIT sockets on a client often means connections are being opened per request and not pooled, and that often connects to connection pool problems.

The framework behind this checking is the USE method: utilisation, saturation, errors, per resource. Golden signals, RED and USE explains when to use which.

Journal logs: where the machine tells you what happened

A surprising number of host incidents are explained by a single line in the journal that nobody looked at. These are the sort of lines that matter:

journalctl -k --since "2 hours ago" | grep -i -E "out of memory|oom-kill|i/o error|nvme|ext4|xfs"
journalctl -u myapp.service --since "2 hours ago" -p warning

Wrentrix collects journal logs and treats them the same way as other logs, clustering them into patterns and flagging new and rising ones. The practical gain is that a rare kernel message about a failing disk appears as a new pattern, instead of being buried in thousands of lines of routine cron and session messages.

If you run services with systemd, journal fields such as the unit name travel with each line, which makes per-service filtering possible without any log format work. A quick policy point: journald can be configured as volatile, in which case the log is lost on reboot. Check Storage= in journald.conf on hosts where the record matters, because the journal you most want after a crash is the one from before it. Shipping logs off the host also removes that dependency.

Process discovery by technology

On a long-lived server the question "what actually runs here?" can be hard to answer. Documentation drifts. Somebody installed a second application server for a migration in a previous year and it is still there.

Process discovery by technology lists what a host is running, grouped by what it is. The everyday uses are less glamorous than the name suggests:

• Finding what else lives on a server before maintenance, so that a reboot doesn't surprise a team you did not know about.
• Spotting a runtime that shouldn't be there at all, which interests security teams as well as operators.
• Deciding where to apply instrumentation next, because a host running a JVM or a .NET application is a candidate for tracing.
• Identifying which process is consuming a resource when a graph moves, in place of logging in and running top during the incident.

Which technologies are recognised is something to check against your own estate during a trial. Ask us if you depend on something specific.

A worked example across hosts and pods

This one is invented, but realistic. An SLO alert fires for POST /transfers, with latency climbing. Pods are healthy according to the Kubernetes view, with no change on the timeline. The trace shows time accumulating in one database call, which is normally 8 ms and is now 900 ms.

You look at the database host. CPU is at 25%, which by itself would suggest no problem. But iowait has climbed, the block device's await has gone from 2 ms to 180 ms, and a new log pattern in the journal reports I/O errors on one path of a multipath device. The database is fine. Storage has lost a path, and the remaining one is overloaded.

No application-side tool would have told you that, and no host-side tool alone would have connected it to the transfer endpoint. The joined view is what makes this a ten-minute investigation and not a meeting between three teams.

Limits to be honest about

• This page is about Linux. If you have other operating systems in your estate, ask us what applies to them. Don't assume coverage.
• Application-level method tracing is separate. The zero-code path for Java and .NET is the Kubernetes admission webhook. For runtimes directly on hosts, you can send spans with any OpenTelemetry SDK; distributed tracing shows how. Ask us about method-level data on hosts.
• Database internals. Host metrics tell you the machine is struggling. They won't tell you which query plan changed. You still need the database's own views, and traces show you which statements are slow from the caller's side.
• Hardware health. Fan, RAID controller and power supply monitoring generally belongs with your hardware management tools. Only what the operating system reports will appear.

Frequently asked questions

How do I add a Linux server to Wrentrix?

Add the host in the UI and download the install script, which has a token already inside it. Run it on the host, and the host appears in about two minutes. Review the script first, as you would any install script, and treat it as containing a credential.

What does the Linux agent collect?

It collects CPU, memory, disk, network and journal logs, and it discovers processes by technology. Journal logs go through the same pattern clustering as other logs, so new and rising messages get flagged. That means a single unusual kernel message is much easier to notice.

Does the host need internet access?

No. Wrentrix works fully air-gapped, with no license calls and no telemetry leaving your network. The host only needs to reach your own Wrentrix installation, and an image-less agent install exists for closed networks.

Why is my load average high when CPU usage is low?

On Linux, load average counts tasks waiting on uninterruptible I/O as well as runnable tasks, so a storage problem can raise load while CPUs sit idle. Check vmstat 1 for blocked tasks and I/O wait, then iostat -x 1 for device latency. If both are normal, look for tasks stuck on a network file system.

Is %util at 100% in iostat always a problem?

No. The figure shows that the device had at least one request outstanding for that share of time, which says little for SSDs and arrays that serve many requests in parallel. Rising await and queue size are better indicators that a device has really run out of capacity.