Database connection pool exhaustion: diagnosis and fixes

What it looks like

Every driver words the failure differently. These are the messages people paste into a search box at 3 a.m.:

The first three are client-side: the app could not get a connection from its own pool within the wait time. The database may be nearly idle while these occur. The last two are server-side: the database itself refused a new connection, because the total across all clients hit max_connections. These are different problems, and they have different fixes.

The maths: Little's law

A pool's demand is simple to calculate. The average number of connections in use equals the arrival rate multiplied by the hold time:

connections in use = requests per second × seconds each request holds a connection

Here is a worked example. A service has 6 pods, each with a pool of 20, and receives 300 requests per second, so 50 per second per pod.

Normal day: each request holds a connection for 40 ms in total.

50 × 0.040 = 2 connections in use per pod (10% of the pool)

After one table starts seeing lock waits: hold time rises to 600 ms.

50 × 0.600 = 30 connections needed per pod, with only 20 available

Now the pool can serve at most 20 ÷ 0.6 = 33.3 requests per second per pod, against 50 arriving. About 16.7 requests per second pile up in the wait queue. With a 30-second acquisition timeout, that backlog could reach roughly 16.7 × 30 = 500 waiting requests per pod before the first timeouts fire. Request traffic did not change at all. A 15-fold increase in hold time did this.

This is why raising the pool limit is rarely a real fix. At 600 ms you would need 30 per pod, or 180 in total, and each extra connection pushes more concurrent queries into a database that was already slowing down.

Which of the five causes do you have?

1. Queries have become slower

A plan change, a missing index, table growth or heavy concurrent work slows queries. Hold time rises and the pool drains. Compare the query's duration against its usual range. In PostgreSQL, pg_stat_statements shows which queries' total time has grown, if the extension is enabled.

2. Lock waits in the database

Queries are cheap but stuck behind one blocker, perhaps a migration or a long-running update. This query shows who is waiting on whom:

SELECT a.pid,
       pg_blocking_pids(a.pid) AS blocked_by,
       now() - a.query_start AS waiting_for,
       a.wait_event_type,
       left(a.query, 80) AS query
FROM pg_stat_activity a
WHERE cardinality(pg_blocking_pids(a.pid)) > 0
ORDER BY waiting_for DESC;

Follow the blocked_by pids until you reach a session which isn't itself blocked. That session is the true cause. Quite often it is idle in a transaction, or a schema migration waiting for an ACCESS EXCLUSIVE lock that blocks everything queued after it.

3. Transactions held open across remote calls

This one is the most common design-level cause, and code review tends to miss it. A method opens a transaction, runs a query, calls a payment gateway over HTTP, then commits. When the gateway slows from 100 ms to 5 seconds, every database connection in the pool ends up waiting on an HTTP call.

SELECT state, wait_event_type, wait_event,
       count(*) AS sessions,
       max(now() - state_change) AS longest_in_state
FROM pg_stat_activity
WHERE backend_type = 'client backend'
GROUP BY 1, 2, 3
ORDER BY sessions DESC;

A large group in the state idle in transaction is the tell-tale sign. The database is doing nothing, and the application is holding connections open. The fix is to move the remote call outside the transaction. As a safety net, PostgreSQL's idle_in_transaction_session_timeout ends sessions left in this state. Set it thoughtfully, as it kills the session rather than merely warning about it.

4. Leaks

A code path borrows a connection and does not return it, usually an exception path with no finally or using. The signature is a pool that steadily loses capacity even at constant traffic and recovers on restart. Many Java pools offer a leak detection threshold that logs a stack trace when a connection is held beyond a set time. It can flag legitimately long operations too, so treat its output as a lead, not a verdict.

5. Total connections exceed what the server allows

Client pools multiply. Our example has 6 pods × 20 = 120 potential connections. PostgreSQL's default max_connections is 100, of which 3 are reserved for superusers by default (superuser_reserved_connections), leaving 97.

The trap is that everything works until load increases. Pools grow only when needed, so a quiet service stays well below the limit. Then a surge or slowdown makes every pool expand at once, and the server starts refusing connections to all of its clients, including services unrelated to the original problem.

Autoscaling makes this worse. If an HPA reacts to rising latency by scaling from 6 to 10 pods, the potential total jumps to 200. In a slowdown caused by the database, scaling out the app adds load exactly where it hurts. Rolling updates also add to the total temporarily, because surge pods run alongside the old ones.

Do this calculation for every service sharing a database, using maximum replica counts:

sum over services (max replicas + max surge) × pool maximum ≤ max_connections − reserved − admin margin

Triage order during an incident

1. Check if the error is client-side (pool wait timeout) or server-side (too many clients). This decides where to look.
2. Look at traces for slow requests and see where the time went. Long pauses before the first query mean waiting for the pool. A slow query span means the database. A long gap inside a span that holds a transaction points to cause 3.
3. Run the pg_stat_activity summary above. active with lock waits, idle in transaction, and many idle sessions each tell you something different.
4. Find and deal with any blocker. Terminating a session with pg_terminate_backend(pid) is sometimes the correct mitigation, though you should know what that session was doing first.
5. If a downstream call is the cause, shed load or trip a circuit breaker. A short timeout on that remote call often restores the pool immediately.
6. Stop retries from amplifying the problem. Each retry adds another request to the queue, so unbounded client retries can keep a pool saturated after the original cause has passed.

Sizing a pool

Make the pool small enough to protect the database, and size the wait timeout to fail quickly. A 30-second wait in a service with a 2-second request budget just means users wait 30 seconds to get an error.

For total database concurrency, the PostgreSQL wiki's starting formula is connections = (core_count * 2) + effective_spindle_count. It is a starting point for benchmarks, not a law. A database often delivers more throughput with fewer connections, because each backend adds contention for locks and memory.

When to add a connection pooler

If the sum of application pools exceeds what your server should run, put an external PostgreSQL connection pooler between them. In transaction pooling mode a server connection is assigned to a client only for the length of a transaction, so thousands of client connections can share a few dozen server connections.

Transaction pooling does not preserve session state. SET outside a transaction, advisory session locks, LISTEN/NOTIFY and temporary tables won't behave as they would with a dedicated session. Protocol-level prepared statements need explicit support from the pooler, or the driver has to be configured to avoid them. A pooler does not fix slow queries either. If hold time is the problem, the queue simply moves to the pooler.

What to monitor

Alert on the leading indicators, because the exception only appears after users are already affected.

• Pending requests. A non-zero wait queue that lasts for more than a moment is the earliest useful signal.
• Acquisition time. A rising p95 shows pressure before any timeout.
• Used versus maximum. A pool at 90% of capacity, sustained, leaves no headroom.
• Hold time. Hold time is the variable from Little's law, and it is the first to change.

OpenTelemetry's database client semantic conventions define db.client.connection.count (with a state of idle or used), db.client.connection.max, db.client.connection.pending_requests, db.client.connection.wait_time and db.client.connection.use_time. Library support varies, so check what your instrumentation really emits. These metrics map neatly to saturation in the golden signals.

Seeing the wait in traces

The hardest part of this failure is that the visible span is often the wrong one. A request shows 8 seconds, and the database span inside it shows 12 ms. The missing time was spent waiting for a connection, and auto-instrumentation may not create a span for that wait, leaving an unexplained gap.

Distributed tracing in Wrentrix shows the service map and endpoint latency, and method-level tracing for Java and .NET can show which method the time went to with no code change, so a gap like this one gets a name. Hour-of-week baselines make a rise in a pool-related method's duration stand out from normal peak-hour behaviour.

Frequently asked questions

What causes connection pool exhaustion?

Connections are being held longer than the pool can sustain at the current traffic. The usual reasons are slow queries, lock waits, transactions held open during remote calls, and leaked connections. Traffic growth is a less frequent cause than increased hold time.

How do I check for idle in transaction sessions in PostgreSQL?

Query pg_stat_activity for rows where state = 'idle in transaction', and order by now() - state_change to see the oldest. Those sessions hold locks and connections while doing no database work, which usually means the application is doing something else between statements. idle_in_transaction_session_timeout can terminate them automatically.

How many database connections should each pod have?

Multiply the peak request rate per pod by the average connection hold time in seconds, then add a margin. Then multiply the pool maximum by your maximum replica count, including rollout surge, and confirm the total across all services fits below max_connections. If it doesn't, either reduce pool sizes or add an external connection pooler.

Does an external connection pooler fix connection exhaustion?

It fixes the "too many clients" problem by letting many client connections share fewer server connections. It does not fix slow queries or long transactions, because those still occupy a server connection, and the queue simply forms in the pooler. Transaction pooling also restricts session features, so test your application against it first.