Java latency troubleshooting: from a p99 spike to the cause

Start from the shape of the problem

Before opening any tool, look at the latency over time and ask which of these it resembles. The pattern narrows the search far more quickly than the absolute numbers do.

Thread dumps: still the fastest test

A thread dump takes a moment and costs very little. Take three, ten seconds apart:

for i in 1 2 3; do
  jcmd 1 Thread.print > "/tmp/threads-$i.txt"
  sleep 10
done

A thread that appears in the same place across all three is genuinely stuck there. Group the dumps by top frame and look for these patterns:

• Many request threads in TIMED_WAITING inside your connection pool's getConnection means the connection pool is empty. See the connection pool exhaustion guide.
• Many threads BLOCKED on one monitor, with one thread holding it, means a lock has become a bottleneck. The dump names the owner. Look at what the owner is doing, since it may be sitting on network I/O inside a synchronized block.
• Threads in RUNNABLE inside socketRead are usually waiting on a remote peer. The state is misleading: for a thread blocked in native I/O, the JVM still reports RUNNABLE. So do not count these as using CPU.
• All threads named like http-nio-8080-exec-* occupied suggests the servlet container's worker pool is full. New requests queue, and that queue time is often missed by application-level timers.

Java Flight Recorder: the default deep dive

JFR is part of the OpenJDK distribution from JDK 11 and designed to run in production. You can start a recording on a running JVM without a restart:

jcmd 1 JFR.start name=incident settings=profile duration=120s filename=/tmp/incident.jfr
jcmd 1 JFR.check

The profile settings collect more detail than the default settings, at a little more overhead. For an incident you cannot predict, run a continuous recording and dump it when needed:

java -XX:StartFlightRecording=name=continuous,settings=default,maxage=1h,disk=true \
     -jar app.jar

jcmd 1 JFR.dump name=continuous filename=/tmp/last-hour.jfr

Open the file in a JFR viewer, or use the JDK's jfr command on the machine itself:

jfr summary /tmp/incident.jfr
jfr print --events jdk.JavaMonitorEnter /tmp/incident.jfr | head -60
jfr print --events jdk.GCPhasePause /tmp/incident.jfr | head -60

Events worth knowing:

jdk.SocketRead deserves special mention, because it links wait time directly to a remote address and the Java stack that made the call. It often answers "which dependency?" without any further tooling. Mind the recording thresholds: some events only record operations above a configured duration, so very short waits are not represented.

Garbage collection: prove it before tuning

Enable GC logging permanently. Its cost is small and you cannot recreate the data after the fact.

-Xlog:gc*,safepoint:file=/var/log/app/gc.log:time,uptime,level,tags:filecount=5,filesize=20m

On a read-only root filesystem, the path needs a writable volume mount. Then check the pauses against your latency spikes. If a spike at 14:32:10 has no pause near it, GC did not cause it, and the time spent tuning G1 would be wasted.

If GC really is responsible:

• Long young pauses often mean many objects survive each collection. That frequently comes from a heap too small for the live set or from large in-memory caches.
• Full GCs in G1 are a warning sign. Look for Pause Full and the log's stated reason, such as an allocation failure or a Metaspace limit. Fix the cause before making any flag changes.
• Humongous allocations in G1 are objects at least half a region in size. Frequent ones can cause early collections and fragmentation. Large buffers are the usual source.
• Pause goals: G1's default MaxGCPauseMillis is 200 ms. It is a goal, not a guarantee. If your p99 target is below this, consider ZGC, which aims for far shorter pauses at the cost of some throughput and memory. Generational ZGC is enabled with -XX:+UseZGC, plus -XX:+ZGenerational on JDK 21 and 22; it is the default ZGC mode from JDK 23.

A small container can catch you out too. If the JVM sees fewer than 2 CPUs or less than about 1792 MB of memory, it is not "server class", and ergonomics picks Serial GC. Check the log's first lines or run jcmd 1 VM.flags, because you may not be running the collector you think you are.

Safepoints: pauses that aren't GC

All stop-the-world pauses need a safepoint, and some of them are unrelated to GC: biased lock revocation on older JDKs, code deoptimisation, certain diagnostic operations. The log configuration above includes the safepoint tag. It reports both the operation time and the time to bring all threads to the safepoint. If threads take a long time to arrive, a single thread running a long uninterruptible loop can hold up the whole JVM, and this delay does not appear in GC pause time.

Containers and CPU limits

Modern JVMs read cgroup limits, and that creates its own issues.

CPU throttling. Under the Linux CFS scheduler, a container that uses up its quota is stopped for the rest of a 100 ms period. A JVM with a 2 CPU limit might run GC threads, JIT threads and request threads at the same time and exhaust the quota early. The result is stalls of tens of milliseconds with no matching event in any JVM log. Check it directly:

cat /sys/fs/cgroup/cpu.stat

The fields nr_throttled and throttled_usec appear on cgroup v2 hosts. A steadily increasing nr_throttled during your latency spikes means the JVM is being frozen by the kernel. The right-sizing guide covers what to do with requests and limits.

Memory. -XX:MaxRAMPercentage sets the heap relative to the container limit, and the default is 25%. That is conservative enough to leave many services needlessly short of heap. Do not push it near 100, however, because the JVM also needs memory for Metaspace, thread stacks, code cache, direct buffers and GC structures. An OOMKilled pod with a healthy heap usually points to off-heap use. Enable -XX:NativeMemoryTracking=summary and read it with jcmd 1 VM.native_memory summary to see the breakdown.

Wall-clock profiling

A CPU profile doesn't show waiting, and it's the waiting you are usually after. A wall-clock profiler samples all threads, regardless of state, so its flame graph shows where threads spend real time, including sleeping and blocking. Open source JVM sampling profilers offer this mode. Split the output by thread, which stops idle threads dominating the picture.

Check permissions first. Native CPU profiling on Linux often relies on perf_events, which locked-down containers may not be allowed to use, so look for a timer-based mode. Inside the JDK, JFR's jdk.ThreadPark, jdk.JavaMonitorEnter and jdk.SocketRead events cover much of the same ground.

Virtual threads

Virtual threads, final in JDK 21, change the diagnosis somewhat. A blocked virtual thread releases its carrier, so thread exhaustion gives way to other limits, most often the connection pool. A pool of 10 connections still serves only 10 queries at once, whatever the number of threads.

The other known issue is pinning, where a virtual thread can't unmount from its carrier. On JDK 21 through 23, blocking inside synchronized pins the carrier. JFR records this as jdk.VirtualThreadPinned. JEP 491 in JDK 24 removed this limit for synchronized, so your JDK version changes the advice.

A plain Thread.print omits virtual threads, too. Use jcmd 1 Thread.dump_to_file -format=json /tmp/threads.json to include them.

Warm-up after deployment

A new JVM interprets, then compiles the hot code progressively. Expect slower responses early on, and watch for these when p99 jumps with each rollout:

• Readiness passing before the service can really take full traffic
• Thundering connection set-up to the database and to downstream services
• Empty application caches, all populating at the same moment

Gradual rollouts, warm-up requests before readiness, and sensible pool minimumIdle values all help. Consider too whether your change correlation shows spikes that line up with every deploy, since that points to warm-up rather than to a bad release.

Coordinated omission: don't trust the wrong number

A load test client that waits for each response before sending the next one will send fewer requests during a stall. The stalled period then gets underrepresented, and your p99 appears better than users experienced. Use a load tool that keeps an open arrival rate, and be sceptical of percentiles taken from a closed-loop test.

Tracing to the method without touching code

The OpenTelemetry Java agent, attached with -javaagent:opentelemetry-javaagent.jar or through JAVA_TOOL_OPTIONS, instruments common libraries automatically. It tells you that a span was slow, but custom business methods usually remain invisible unless you add configuration.

Wrentrix method-level tracing adds this level for Java and .NET with zero code through admission webhook hooks. Each method gets a baseline built from 17-bucket histograms, which helps you tell a method that has become slower from one that has always been slow. It takes effect at the next restart. Wrentrix never restarts your workloads.

Frequently asked questions

How do I find what is causing Java latency spikes?

First check whether spikes affect all endpoints or just one. If all, compare timestamps with the GC log, the safepoint log and cgroup throttling statistics. If just one, use a trace and then JFR, looking at monitor, park and socket events for that code path.

What is the difference between CPU and wall-clock profiling?

A CPU profile samples only threads that are on CPU, so it shows what is consuming processor time. A wall-clock profile samples threads in any state, which includes blocking, sleeping and I/O. If the latency is high while CPU use is low, a CPU profile will show almost nothing useful, and wall-clock mode is the better tool.

Why does my Java pod get OOMKilled when heap usage looks normal?

The cgroup limit covers total process memory, including Metaspace, code cache, thread stacks, direct buffers and GC structures, as well as the heap. Enable Native Memory Tracking and compare it with the container's memory use. Thread count and direct buffers are common culprits.

Should I switch from G1 to ZGC to fix latency?

Only if GC logs show that pauses are behind your latency target. ZGC gives much shorter pauses, but it trades some throughput and usually needs more memory headroom. If your spikes don't line up with pauses, changing the collector will not help at all.