How Netflix built its in-house LLM serving infrastructure

The Boring Java Architecture for Generative AI

Netflix decided their generative AI models would not get a shiny new infrastructure. They looked at massive, non-deterministic language models and forced them to conform to their existing, boring Java architecture. They didn't build an AI silo. They took large language models and shoved them behind a standard gRPC call on the JVM.

Why self-host LLMs at all when hosted APIs from OpenAI or Anthropic are so fast and reliable right now? For most teams, paying per-token is the right move. But Netflix operates an enormous, dedicated AI Platform organization. They already have a massive fleet of existing machine learning models: XGBoost, PyTorch, TensorFlow. These models share a highly optimized, cross-region GPU compute pool. Bringing LLMs in-house means they can multiplex generative models on the exact same shared hardware. It also gives them tight control over specialized constrained decoding at massive batch sizes.

A Unified, Mundane Architecture

They did not want two disparate systems. They wanted one unified routing layer. To make an LLM act like a standard microservice, Netflix routes all requests through a unified JVM-based serving system. This Java layer handles candidate generation, feature fetching, A/B testing, and local pre-processing. If the model is small, it runs right there in-process. If it requires a GPU, the Java system delegates the work over gRPC to a remote backend called the Model Scoring Service.

This is where the actual LLM lives. They wrapped the vLLM inference engine inside NVIDIA Triton Inference Server. Then, they put a custom Java control plane on top of Triton to handle health checks, zero-downtime upgrades, and multi-region rollouts.

To the product teams consuming these models, the experience is deliberately mundane. Netflix exposes an OpenAI-compatible API alongside their standard gRPC frontends. If a product team wants to graduate from a third-party hosted API to a fine-tuned, self-hosted open-source model, they simply change the endpoint URL. The request payload stays exactly the same.

The Platform Tax

But that frictionless transition is an illusion. It is only easy for the product developers because the platform engineering team absorbed a crushing amount of complexity to make it work. Take the OpenAI-compatible API. Netflix used the off-the-shelf Triton frontend to serve it. In testing, they realized the frontend was silently dropping critical parameters, specifically the response_format schema. A caller would request strictly formatted JSON, Triton would drop the parameter, and the inference engine would proceed without guided decoding constraints. The caller would get malformed garbage back, and the platform would log zero errors. Netflix had to git-subtree and patch NVIDIA’s frontend code just to translate the response format into vLLM’s guided decoding parameters.

You see this platform tax everywhere in the architecture. Downloading massive model weights directly from S3 at startup is slow. It inflates cold-start latency well past what their orchestration schedulers will tolerate. So Netflix built a system to materialize models on Amazon FSx at the time of the model announcement. Warm starts hit a high-performance file system instead of object storage.

Metrics were another casualty of wrapping engines inside engines. Triton reports metrics through its own HTTP endpoint. vLLM writes metrics to a local disk directory as database files. Neither knows the other exists. Out of the box, Triton’s bridge only surfaces nine of vLLM’s forty-plus metrics. You lose critical visibility into token throughput and cache hit rates. Netflix had to write a custom HTTP proxy just to merge the two sources together, reading from HTTP and disk simultaneously, so their existing alerting dashboards wouldn't break.

Zero-Downtime Deployments and Their Brutal Cost

Treating models like normal software also means demanding zero-downtime deployments. Netflix relies heavily on standard red-black deployments. You spin up the new version, health check it, shift traffic, and scale down the old one. But what happens when a model update changes the input and output tensor shapes? Standard red-black rollouts have a fatal flaw here. The upstream calling application cannot update its configuration until the new model is fully live. During the migration window, the caller inevitably sends old tensor schemas to the new deployment. Those requests instantly fail.

To fix this, Netflix uses versioned deployments for breaking schema changes. They run an independent deployment for every model version simultaneously. The consumer waits for the new version to be perfectly ready, switches its configuration, and the old version continues serving legacy traffic. The cost is brutal. You have to temporarily double your GPU quota during the transition window to keep both versions alive.

The Illusion Shatters: Constrained Decoding at Scale

Picture this. You swap out your OpenAI API call for a self-hosted open-source model. The latency looks incredible in your local testing. But the minute you deploy to production and hit it with real concurrency, you ask it to format outputs as JSON. Suddenly, your service grinds to a halt. Your massive, expensive GPU cluster is sitting completely idle, because the entire system is bottlenecked on a single Python CPU thread. This is exactly where the normal software illusion shattered for Netflix.

It brings us to the reality of constrained decoding at scale. Some Netflix production workloads require fine-grained control over token generation. Instead of generating text, parsing it, finding out the JSON is malformed, and paying to retry the inference, they push constraints directly into the decode loop. The model generates compliant data by construction. It evaluates a state machine at every single token to emit eligibility masks. It physically cannot output an invalid token.

When Netflix initially deployed this using vLLM version zero, they wrote the constraint logic in pure Python. Functionally, it worked perfectly. Under production load, it blew up. In that older version of the engine, custom logits processors ran per-request. The GPU would generate logits for a huge batch of requests, copy them over to the CPU, and then wait. The Python Global Interpreter Lock, the infamous GIL, forced the CPU to run the constraint logic sequentially for every single request in the batch. CPU time grew linearly with the batch size. A service backed by top-tier hardware became completely CPU-bound. The tail latency was disastrous.

Rewriting the Hot Path

Fixing this required a massive engineering lift. They migrated to vLLM version one, which moved logits processing to the batch level. But they couldn't just write better Python. They rewrote their custom processor to operate on batch-level data structures, and they built the hot path in multi-threaded C++ to bypass the Python interpreter completely. Performance recovered. The CPUs could finally keep up with the GPUs.

But pushing stateful logic inside an LLM decode loop introduced completely alien failure modes. You are no longer tracking stateless web requests. You have to track partial prefills. Modern inference engines chunk prompts over multiple steps to maximize throughput. A request might only be partially prefilled in a given batch. Then you have to handle preemption. If the system hits memory pressure, it might evict a partially completed request’s KV cache and reschedule it later. When that happens, the output token list actually shrinks. It reverses. The C++ state machine has to detect that the generated sequence went backwards, reset its internal state, and re-evaluate the new prompt from scratch.

The Verdict: A Blueprint and a Warning

Netflix proved that you can force generative AI to conform to existing infrastructure. You can hide the massive complexity of an LLM behind a standard gRPC interface, unifying it with your legacy machine learning stack. The engineering work here is phenomenal. The way they dynamically generate I/O tensor specs at deployment time using Triton's vLLM backend, rather than freezing them in Python, is a masterclass in decoupling models from frontends.

But looking at the effort required, you have to measure the cost. They are managing custom HTTP proxies for metrics. They are maintaining internal patches of NVIDIA frontend code. They are writing multi-threaded C++ state machines to survive Python concurrency limits. If you are at member scale, multiplexing thousands of models on a shared hardware pool, this architecture is an incredible blueprint for pragmatism. But if you are anywhere else, this is a massive warning label. Unless you absolutely must self-host for strict data privacy or highly specialized constrained decoding, stick to hosted APIs. The baseline cost of maintaining idle GPU capacity and the operational overhead of a custom control plane will eat your team alive.

And if you do decide to self-host, run a standard open-source vLLM Docker container behind a boring HTTP load balancer. Do not attempt to introduce Triton and build a unified inference mesh unless you have the platform engineering army to absorb the fallout.

TAKEYOURPILLS.TECH. Go ship something.