Why P99 Latency Hits 5 Seconds in a Connection Storm

ScyllaDB engineers traced a customer-visible incident where node restarts pushed p99 latency to 5 seconds for a full minute, even though steady-state latency on the same cluster sat at 4 milliseconds. The queries were not the problem — a connection storm was.

Quick take

A connection storm, not a query plan regression, drove ScyllaDB p99 latency from 4 milliseconds to 5 seconds for a full minute.

The cause was thousands of new connections competing for a small, fixed CPU admission slot, not disk or network capacity.

Aurora DSQL enforces a separate hard limit of 100 new connections per second with a burst of 1,000, and disconnects any connection after 1 hour.

Watch active and failed connection counts, not just query duration, when you suspect a connection storm.

What a Connection Storm Looks Like Before Query Latency Moves

On a cluster with 9 nodes and 64 cores (vCPUs), a single client process opens 500+ connections, one per vCPU, to reach every shard directly. In large Kubernetes deployments running tens of thousands of client processes, the effective connection count for the cluster can reach into the millions.

That fan-out is fine at steady state. It becomes dangerous during topology changes or a mass application restart, when every client tries to reconnect within the same few seconds.

A first glance at dashboards during this kind of event often shows query duration climbing, which pulls attention toward the query plan or the storage engine. Neither is where the time is going.

Why Throttling Queries Doesn’t Fix a Connection Storm

Slowing down or shedding read and write requests is a natural first response to a latency spike, and ScyllaDB already had request throttling before this incident. It did nothing for connection storms, because the bottleneck sits earlier in the pipeline, before any query is even parsed.

Each new connection becomes a task in the CPU scheduler’s queue. When thousands arrive at once, the queue bloats, most connections miss their setup timeout, and the client retries — adding still more connections on top of the ones still in flight.

That retry loop is what turns a brief surge into a full minute of degraded p99, and it explains why request-level throttling, which only governs already-established connections, left the problem untouched.

The Real Mechanism: CPU State vs Networking State

ScyllaDB’s fix separates every new connection into two states. A connection sits in the networking state while it is sending or receiving handshake packets, and moves to the CPU state once it needs scheduler time to finish setup.

Admission control now caps how many connections can be in the CPU state at once, with a default of 8, configurable through uninitialized_connections_semaphore_cpu_concurrency. Everything past that cap waits, instead of competing for the same scheduler queue as live traffic.

That single change, combined with caching, hashing adjustments, and a dedicated service level for background connection traffic, is what produced the 1000x cut in tail latency ScyllaDB reported for this workload.

Watch out

A p99 spike during a rolling restart or mass redeploy can look exactly like a query plan regression on a duration graph, but the query itself may never have changed.

Check connection counts and connection errors before touching indexes or query plans when the spike lines up with a deploy or restart window.

Checks That Separate a Connection Storm From a Query Regression

Azure Database for PostgreSQL flexible server exposes active_connections, connections_failed, and connections_succeeded as default metrics, each emitted at a one-minute interval with up to 93 days of retention. A spike in connections_failed alongside a duration spike points at admission, not the query planner.

Google Cloud’s Query insights for Datastore captures normalized query text, execution count, and average execution duration at 10 minute granularity for the last 4 days, extending to 1 hour granularity out to 30 days, though the data itself lags by one to two hours. If per-query duration is flat while overall latency climbs, the regression is not in the query plan.

AlloyDB’s advanced query insights add wait event telemetry and can trace a slow query back to the application route or ORM call that issued it, sampling up to 20 query plans per minute. When wait events point at connection setup rather than lock or I/O waits, that also rules out the query plan.

Fixing and Preventing the Next Connection Storm

The durable fix is architectural: cap and stagger connection admission rather than relying on clients to reconnect gracefully, since ScyllaDB explicitly designed its throttling to work even when a client’s own startup sequence fails or is missing.

Amazon Aurora DSQL applies a similar principle from the other direction, using transaction-level connection multiplexing instead of a 1:1 session-to-backend model, so a smaller pool of query processors absorbs a much larger number of client connections.

That architecture is also why AWS advises against running PgBouncer or pgpool-II in front of Aurora DSQL — the transaction-level pooling those tools provide is already built in, and layering another proxy on top is redundant.

For Aurora DSQL specifically, AWS recommends a HikariCP maximumPoolSize of 20, a 55 minutes max lifetime, a 10 minutes idle timeout, and a 30 seconds connection timeout, all set below the service’s 1 hour maximum connection age so the pool recycles connections before the platform forces a disconnect.

That same pool must also respect Aurora DSQL’s 100 new connections per second limit, with a burst allowance of 1,000, which is the same class of admission ceiling that a connection storm exceeds on any database that enforces one.

Before assuming a latency regression is a query plan problem, pull connection counts and connection errors for the same window and compare them against duration. If failed or new connections spike first, treat it as an admission problem, not a query problem, and size the pool’s max lifetime against the database’s own connection age limit.