How to survive 121 million concurrent gRPC connections

Picture this. It's the middle of a massive traffic surge. Fourteen million concurrent connections are hitting your infrastructure within seconds. Eighty percent of your requests are failing, returning HTTP 500 errors across the board. You do the logical, reflexive thing any infrastructure engineer would do. You spin up more capacity. You scale your Network Load Balancers from two, to four, and eventually to six.

You watch the metrics. The new capacity spins up, registers as healthy, and sits completely idle. Absolutely nothing changes. The incoming traffic keeps hammering a single load balancer, completely ignoring the new hardware, while that primary node melts into slag.

This is not a theoretical exercise. This happened to a mobile observability company named Bitdrift during the T20 World Cup cricket series. Their SDK was embedded in a customer's app to deliver live match telemetry. As matches kicked off, traffic surged from near-zero to over 110,000 requests per second in seconds. We are talking about a 100x traffic surge hitting the edge instantly. Millions of mobile devices established persistent connections simultaneously.

Their architecture was textbook. Amazon CloudFront sat at the edge to handle global ingress. Route 53 managed the DNS. Behind that, a fleet of multi-AZ Network Load Balancers routed traffic to EKS clusters running the workloads. It's a standard, highly scalable setup. But we're going to look at why that standard scaling failed. We're going to look at how a statistically sound DNS configuration for HTTP traffic becomes a catastrophic bottleneck when you mix it with the persistent, sticky nature of gRPC connections at scale.

Why Standard Scaling Fails with gRPC

To understand why adding load balancers didn't actually distribute the load, we first have to answer why this specific failure happens with gRPC, but almost never happens with standard, stateless HTTP traffic.

If you're serving standard web traffic, a request comes in, a response goes out, and the connection closes. The lifespan of the connection is measured in milliseconds. If your DNS resolution creates a slight imbalance in where those requests go, it usually doesn't matter. The sheer volume and brief lifespan of stateless HTTP requests act as a smoothing function. A mildly uneven distribution across your load balancers evens itself out over time.

But gRPC is a different beast entirely. gRPC connections are long-lived. They are sticky. When a client establishes a gRPC connection, it holds on. Those connections accumulate on whichever origin server was resolved at the exact moment the connection was made. The connection doesn't let go until the client disconnects or the server forces a termination.

Now, combine that stickiness with how Route 53 and CloudFront interact. Bitdrift was using Weighted routing in Route 53. Weighted routing works fine, but crucially, it returns a single IP address per DNS query. Just one. When a CloudFront edge node needs to reach your origin to pass along a connection, it queries Route 53. Route 53 hands back exactly one load balancer IP. CloudFront then caches that single IP for the duration of the DNS Time-To-Live window, which in this case was 60 seconds.

Think about what happens next. For the next 60 seconds, every single device connecting through that specific CloudFront edge node gets funneled to that one single load balancer IP.

The DNS Fix for a Connection Storm

When 14 million devices connect exactly as a cricket match starts, they are all hitting the same 60-second DNS cache windows across the edge nodes. All those persistent gRPC connections pile onto a single machine. Because it's gRPC, they don't close. They just accumulate. You can scale your Network Load Balancers from two to fifty. It won't matter. The DNS TTL cache acts as a massive funnel, pointing an entire region's worth of traffic at a single IP. It makes scaling your origins completely invisible to the incoming traffic.

The fix for this massive outage wasn't a rewrite of their connection handling. It was a zero-code dropdown change in the AWS Route 53 console. They migrated their DNS records from Weighted routing to Multi-Value Answer routing.

So, what is Multi-Value Answer routing, and how does it trick CloudFront into spreading the load? Instead of returning a single IP address per query, Multi-Value Answer routing returns up to eight IP addresses per DNS response. When CloudFront queries Route 53, it doesn't get a single funnel. It gets a list of up to eight load balancer IPs all at once. The CloudFront edge node can immediately spread the incoming connections across all of those IPs from the very first DNS resolution. It completely bypasses the single-IP caching trap.

CloudFront caches the list of eight IPs, not just one, and distributes the incoming gRPC connections across them. With this single configuration update, they went from an 80 percent failure rate to zero server-side errors. By the final match, they handled 121 million unique devices and 110,000 peak requests per second with zero origin connection errors. It's a great payoff. A configuration change saves the day and unlocks massive scale.

Architectural Coupling and Hidden Trade-offs

But we need to look at the architectural coupling this introduces. You can't just flip your routing policy to Multi-Value Answer and walk away. This routing type requires static IP address records. It fundamentally breaks compatibility with Application Load Balancers, because ALBs do not support static IPs. If you make this change, you are forced into using Layer 4 Network Load Balancers and assigning static Elastic IPs to each one, in every Availability Zone.

You also take on a heavy operational burden. You have to manually create and manage custom Route 53 health checks for every single Elastic IP. In a modern cloud environment, your infrastructure is supposed to be elastic. Instances and load balancers should be able to come and go automatically. But with this setup, you have just hardcoded static IPs and custom health checks into your routing layer. How are you automating the synchronization between NLB Elastic IPs and Route 53 if your infrastructure needs to be redeployed or expanded? It's a non-trivial automation problem.

There's also a hard mathematical limit at play. Multi-Value Answer routing returns a maximum of eight IPs per query. If your scale requires 50 load balancer IPs, you're still going to experience traffic clustering, just divided by eight. It mitigates the bottleneck. It doesn't mathematically eliminate it.

And distributing a 100x connection surge evenly at the load balancer level doesn't solve the underlying compute problem. Distributing the connections evenly stops the load balancer from falling over, but those EKS pods still have to handle an instant burst of tens of thousands of requests per second. Distributing connections evenly at the network layer doesn't magically spin up CPU and memory at the compute layer.

When (and When Not) to Worry

The cold-start problem remains entirely your responsibility. Which brings up the question: Do you need to worry about this if you aren't dealing with millions of concurrent users? No, you don't. A single AWS Network Load Balancer is fully capable of handling millions of connections on its own natively. You only hit this specific failure mode when you are forced to use CloudFront to route long-lived connections to multiple origin load balancers, and that DNS TTL caching creates a thundering herd.

If you're building for massive persistent connections right now, you should question whether CloudFront is even the right ingress service for the job. CloudFront is a caching Content Delivery Network. Using it to route real-time, long-lived gRPC telemetry is fighting the shape of the tool.

If you're designing this from scratch, look at AWS Global Accelerator. It is literally designed for global edge ingress of persistent TCP and UDP traffic. It gives you static anycast IPs that route directly to your regional endpoints, and it completely bypasses the DNS TTL caching issues that caused this outage in the first place. You don't have to trick the DNS layer into spreading the load with Multi-Value Answer routing, because the load balancing happens at the network edge natively.

What this outage teaches us is that the protocol you choose dictates the rules of your infrastructure. HTTP is forgiving. Its short lifespan hides slight imbalances in your routing layer. gRPC is unforgiving. A persistent connection magnifies any routing imbalance until it breaks the system. When you transition to long-lived connections, you can no longer trust the standard web traffic playbooks.

A DNS configuration that works flawlessly for REST APIs will tear down your servers when applied to WebSockets or gRPC at scale. You have to trace the exact path of a connection from the client, through the edge cache, through the DNS resolution window, down to the metal, and ask yourself what happens if that connection never closes.

This is TAKEYOURPILLS DOT TECH.

Go ship something.