Troubleshooting .NET performance in production

An example incident: fine at 10:00, unusable at 10:05

This scenario is an illustration, built from a pattern that is very common in ASP.NET Core. An order API has a p99 of 180 ms. At 10:05 traffic rises moderately, and p99 climbs to 12 seconds within a couple of minutes. CPU sits at 25%. The database looks healthy. Liveness probes begin to fail and Kubernetes restarts pods, which makes things worse.

High latency with low CPU means requests are waiting, not working. Run the counters first because they are cheap:

dotnet-counters monitor -p 1 --counters System.Runtime

The output shows the thread pool queue length at several thousand and rising, with the thread count climbing slowly. That is thread pool starvation.

To find the cause, take a dump and look at where threads are stuck:

dotnet-dump collect -p 1 -o /tmp/orders.dmp
dotnet-dump analyze /tmp/orders.dmp
> threadpool
> pstacks

pstacks merges identical stacks, and a large group of threads sits in Task.Wait under a method called GetCustomerTier, which calls .Result on an async HTTP call. Each incoming request blocks a pool thread while waiting for an async completion, and that completion also needs a pool thread to run. Under light load this is merely inefficient. Under moderate load the pool runs out, and the pool's gradual thread injection cannot catch up.

The fix is to make the call path properly async. ThreadPool.SetMinThreads can act as a temporary mitigation by allowing a larger burst of threads, but it does not fix the cause.

The incident also shows a common secondary failure: the liveness probe was served by the same starved thread pool. Restarting a pod under this kind of load discards its warmed caches and pushes its traffic onto the other pods. Make liveness checks trivial, and put dependency checks in the readiness probe.

The toolset

All of these are separate .NET global tools, installed with dotnet tool install -g <name>. Standalone downloads also exist, which are useful for containers.

A dump contains process memory, which means it can hold connection strings, tokens and customer data. Treat it as sensitive, and follow your data handling rules for where it is stored and who can open it.

Running the tools in a container

The tools connect through a diagnostics IPC socket in /tmp. A minimal production image will not contain them, and adding them permanently is rarely welcome. Options:

• Use an ephemeral debug container that shares the target's process namespace: kubectl debug -it pod/orders-6f7d --image=<your tools image> --target=orders. The container needs to reach the target's /tmp (through /proc/1/root/tmp when the namespace is shared) and run as the same UID.
• Run a sidecar sharing an emptyDir mounted at /tmp, along with the same user ID. This works well under locked-down security contexts where ephemeral containers are not permitted.
• Copy a tool binary into the pod for a one-off session if policy allows.

Under OpenShift's restricted-v2 SCC you cannot run as root, so matching UIDs is the point most likely to trip you up. Note also that DOTNET_EnableDiagnostics=0 disables the IPC channel entirely, so check for it if the tools cannot find the process. The tools also need /tmp to be writable.

Pattern: thread pool starvation

Signs: latency rises while CPU stays moderate. Queue length grows and thread count creeps up.

Counters: in dotnet-counters, look for ThreadPool Queue Length and ThreadPool Thread Count. On .NET 9 and later, the System.Runtime meter uses names such as dotnet.thread_pool.queue.length and dotnet.thread_pool.thread.count. These same names are available to an OpenTelemetry metrics pipeline.

Causes: .Result, .Wait() and GetAwaiter().GetResult() on async work; synchronous I/O in a request path; long-running CPU work scheduled on the pool; lock contention around slow operations.

Find it: pstacks in a dump, or dotnet-stack report -p 1 for a live view. Run dumpasync in dotnet-dump analyze to inspect pending async state machines when threads look idle but requests are not completing.

Pattern: garbage collection pressure

Signs: latency spikes correlate with % Time in GC, frequent gen 2 collections, and memory that grows and falls in a sawtooth.

Collections in gen 0 and gen 1 are short. Gen 2 background collections are mostly concurrent, but blocking gen 2 collections pause threads for far longer. Large objects at or above 85,000 bytes go to the Large Object Heap, which is collected only with gen 2, and it is not compacted by default. Repeatedly allocating large buffers or strings is therefore a classic cause of trouble, and ArrayPool<T>.Shared is the usual fix.

Check that you are running the intended GC mode. ASP.NET Core projects enable Server GC by default, with one heap per logical core visible to the process. If a container is limited to a small CPU allocation, Server GC behaves differently from what you tested on a large host. Workstation GC may fit better for small replicas where memory density matters.

For retained memory, compare two heap snapshots:

dotnet-gcdump collect -p 1 -o before.gcdump
dotnet-gcdump collect -p 1 -o after.gcdump
dotnet-gcdump report after.gcdump

Or use a dump, which gives you more detail:

> dumpheap -stat
> dumpheap -mt <method table address>
> gcroot <object address>

gcroot answers the question that matters most: why is this object still alive? Static collections, event handlers never unsubscribed, and caches without eviction are the usual answers.

Pattern: container limits the runtime misjudges

.NET honours cgroup limits. By default, the GC sets a heap hard limit at 75% of the container's memory limit. That leaves room for native memory, but a process with heavy native use can still be OOM-killed while the managed heap appears healthy. DOTNET_GCHeapHardLimitPercent adjusts the ratio. Note that its value is parsed as hexadecimal, as with many DOTNET_ settings, so check the documented format before setting it.

CPU limits are subtler. A process limited to 2 CPUs sees that as its processor count, but CFS throttling can still freeze it for part of each 100 ms period when threads burst. Symptoms include periodic latency spikes that don't match CPU averages. Check container_cpu_cfs_throttled_periods_total against container_cpu_cfs_periods_total, and see the guide to Kubernetes right-sizing for choosing requests and limits.

Pattern: HttpClient and outbound connections

Creating a new HttpClient per request and disposing it leaves sockets in TIME_WAIT, and under load you run out of ephemeral ports. Errors then show up as SocketException on connect. A single static client avoids this, but it won't notice DNS changes.

Use IHttpClientFactory, or a shared client with a connection lifetime:

builder.Services.AddSingleton(new HttpClient(new SocketsHttpHandler
{
    PooledConnectionLifetime = TimeSpan.FromMinutes(5),
    ConnectTimeout = TimeSpan.FromSeconds(3),
}));

Set a deliberate timeout. The HttpClient default is 100 seconds, which is far longer than most request budgets and allows one slow dependency to tie up your whole service.

Pattern: the database is the bottleneck

Look for repeated near-identical queries in one trace (N+1 from lazy loading), large result sets materialised into memory, and pool timeouts. EF Core's AsNoTracking() reduces overhead for read-only queries, and AsSplitQuery() helps when a single join explodes row counts, at the cost of extra round trips.

This error message indicates a different issue:

Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool.

Do not raise Max Pool Size as a first response. Find out why connections are being held. Our guide on connection pool exhaustion walks through this properly.

Pattern: CPU actually is high

Collect a CPU trace:

dotnet-trace collect -p 1 --profile cpu-sampling --duration 00:00:30

Built-in profile names have changed across tool versions, so run dotnet-trace collect --help to see what yours offers. Output is a .nettrace file. Open it in a trace viewer, or use dotnet-trace convert to produce a format that flame graph viewers can read.

Keep an eye on JIT effects after a deployment: tiered compilation starts with quick, unoptimised code and promotes hot methods over time, so a new pod may be slower for its first minutes. ReadyToRun images and a gradual rollout both soften this.

From symptom to first command

Method-level visibility with no code change

Traces show that an endpoint is slow. Going further, into which method is slow, usually needs a profiler or manual spans. The .NET runtime's profiling API (CORECLR_ENABLE_PROFILING, CORECLR_PROFILER, CORECLR_PROFILER_PATH) lets auto-instrumentation run without source changes, and this is how the OpenTelemetry .NET auto-instrumentation works.

Wrentrix provides zero-code method-level tracing for .NET and Java through an admission webhook, with method baselines built from 17-bucket histograms, so a method that has become slower than its usual self stands out. Tracing starts on the workload's next restart, and Wrentrix never restarts anything by itself.

Frequently asked questions

How do I detect thread pool starvation in .NET?

Run dotnet-counters monitor and watch the thread pool queue length and thread count. A growing queue with a slowly rising thread count and moderate CPU is the signature. Confirm by taking a dump and running pstacks, which will usually show many threads blocked in Task.Wait or similar calls.

What is the difference between dotnet-dump and dotnet-gcdump?

dotnet-dump captures the whole process memory, so you can inspect threads, locks, objects and their contents, but the file is large and may contain sensitive data. dotnet-gcdump records only the structure of the managed heap, giving types, counts and references, and it triggers a full gen 2 collection when it runs. Use a gcdump for leak trends and a full dump for stuck-thread analysis.

Why does my .NET app get OOMKilled when the heap looks fine?

The container limit applies to the whole process, including native allocations, thread stacks and JIT code, not only the managed heap. The GC defaults to a hard limit of 75% of container memory, so large native use can push the process over while the heap looks healthy. Compare working set against GC heap size to see which side is growing.

Should I use Server GC or Workstation GC in Kubernetes?

Server GC gives better throughput and is the default for ASP.NET Core, but it uses more memory because each heap has its own allocation budget. On small replicas with a low CPU limit, Workstation GC or a capped heap count can be a better fit. Test under realistic load and compare pause time and memory, since the right choice depends on the workload.