Netflix is replacing traditional recommendation models with LLMs

The Prompt is the New Feature Vector

The prompt is becoming the new feature vector. In recommendation systems, we are used to pipelines that look like massive plumbing projects. You extract thousands of dense features. You build embeddings for users and items. You pump them into a two-tower model or a massive gradient-boosted tree. It is complex, expensive, and heavily engineered. Adding a new feature—say, tracking podcast listens alongside movie views—usually requires re-architecting the feature pipelines, updating schemas, and retraining models.

But Netflix just did something entirely different with a new system they call GenRec. They took a user's entire watch history, compressed it into a block of text, fed that text to a Large Language Model, and let the LLM score the catalog. They replaced thousands of hand-crafted features with a single string of natural language. And it is beating their legacy production stack.

This is a deep structural shift from feature engineering to context engineering. Instead of wrangling distributed systems to calculate moving averages of a user's engagement, Netflix turns raw interaction logs into text. A prompt might read: a user watched a specific sci-fi series for three hours, thumbed-up a standup special, and abandoned a romantic comedy after ten minutes. By verbalizing the history, they bypass the traditional feature store entirely. The heavy lifting moves to deciding what raw data goes into the prompt, how far back in time to go, and how to format it.

Context Compaction: The New Feature Budget

Passing a user's entire multi-year interaction history into an LLM context window sounds incredibly expensive. If you naively dump every play, pause, and skip into a prompt, you blow out your token budget and your compute costs skyrocket. The token window is your new feature budget.

You have to spend those tokens on the highest signal data. Netflix handles this with aggressive context compaction. They do not just dump the raw logs. They drop low-signal events like quick hovers or two-minute abandons. They summarize repetitive behavior. If a user binge-watched ten episodes of a show, that gets compressed into a single, high-signal sentence rather than ten separate log entries. They found they could cut the context tokens down to roughly a third of the original size with zero drop in offline ranking quality. Serving cost is directly proportional to context length, so optimizing the text directly optimizes the infrastructure bill.

The Prefill-Only Trick: Scoring Without Generating

But here is the real engineering trick. If you have ever used an LLM, you know they generate text one token at a time. It is slow. If you ask an LLM to generate a list of movie recommendations for a user, you run into two massive problems. First, it takes seconds to print out the titles, which is entirely unacceptable for a homepage loading in milliseconds. Second, the model will hallucinate. It will recommend a movie Netflix does not have the rights to, or it will invent a title that does not exist.

GenRec solves this by never generating text. It does not decode. It runs in what is called prefill-only mode. When you send a prompt to an LLM, the first thing the model does is read the whole prompt in parallel to build an internal representation. That is the prefill phase. The slow part is the decoding phase, where it spits out the answer token by token. Netflix completely skips the decoding phase. They take the pooled hidden state from the prefill phase—essentially the LLM's deep, mathematical understanding of the user's text history—and pass it into a custom, catalog-aware scoring head.

Every item in the Netflix catalog has a learned embedding. The scoring head takes the user's state, compares it to the item embeddings, and outputs a probability score for every item in a single forward pass. There is no text generation. There is no autoregressive loop. There is no token-by-token delay. It is just one massively parallel pass through the GPU that scores the entire candidate set instantly.

Because the scoring head only knows about in-catalog item embeddings, it is physically impossible for the model to hallucinate a movie that isn't on the platform. The LLM is strictly a reasoning engine for the user's context. The scoring head grounds it to reality.

A Two-Phase Training Approach

Training this system happens in two phases. They do not just grab an off-the-shelf open-source model. Phase one takes a foundational LLM and adapts it strictly on Netflix data. The model learns what shows are, how members behave, and how to understand general language in the context of streaming. Phase two is the actual ranking post-training. This is done frequently to track new content and shifting user tastes. The model learns how a verbalized user history maps to actual future engagement.

And because the phase one model already deeply understands the domain, phase two requires ten to forty times fewer labeled examples than their old production stack. That is a massive win for data efficiency. They also use a reward-weighted loss function to align the model with business goals. If you just train on raw clicks, you get a model that heavily favors clickbait or pure binge-watching. Netflix weights the training examples using separate reward signals. A long-term satisfaction event—like a user exploring a new genre and coming back the next day—gets a higher scalar weight. A quick click and abandon gets down-weighted. This aligns the LLM with long-term retention, skipping the massive complexity of full reinforcement learning.

The Hardware Reality Check

This architecture is undeniably clever. But there is a massive hardware reality check we need to run here. Does this mean the era of traditional two-tower recommendation models and complex feature engineering is over? For Netflix, maybe. For most engineering teams, absolutely not.

Look at the actual infrastructure required to pull this off. You are not just calling an external API. Netflix is maintaining in-house foundational LLMs ranging from one billion to ten billion parameters. They are constantly post-training these models on proprietary data. They are running this in the critical path of user requests on a dedicated GPU serving stack using vLLM. The operational overhead of running LLM inference for ranking is immense.

Replacing dense machine learning models with ten-billion parameter LLMs for ranking requires a massive shift in GPU provisioning. The maintenance burden does not disappear; it just shifts from maintaining complex feature pipelines to managing complex prompt pipelines, continuous model post-training, and keeping a fleet of low-latency GPUs running hot. If you are at a normal scale, this is severe overkill. The boring alternative is a standard two-tower model or gradient-boosted trees. They are vastly cheaper to train. They run incredibly efficiently on cheap CPUs. They deliver reliable sub-millisecond latency without needing a fleet of GPUs.

Also, GenRec is scoring a candidate set. It is a ranker. The architecture conveniently leaves out how those candidates are retrieved before the LLM scores them. You still need an efficient, traditional system to retrieve the top few hundred items out of a massive catalog before the LLM ever sees them. The old stack isn't dead; it is just being pushed further down the funnel.

Beyond Chat: The LLM as Infrastructure

So why care about GenRec if you aren't a hyperscaler? Because it proves a fundamental shift in how we can use large language models in production systems. We have spent the last two years treating LLMs as conversational agents. We put them in chat boxes. We wait for them to type out answers. Netflix is treating the LLM as a core piece of routing and ranking infrastructure. By stripping away the text generation, they use the language model's deep semantic understanding of human behavior without paying the latency tax of generation. They proved that natural language is a perfectly viable substrate for machine learning pipelines.

When you verbalize a user's history, you tap into the model's pre-trained understanding of the world. The model already knows that someone who watches intense political thrillers might also like a specific type of historical documentary, because it read the internet. You do not have to build complex cross-features to teach it that relationship. Writing logic to decide what events to keep, summarize, or drop from a prompt is the new feature selection.

We are moving toward a future where many specialized machine learning tasks will share a single, foundational language backbone. It might be a hyperscaler pattern today. But the underlying technique—using prefill-only inference to score known items instead of generating text—is a pattern you can use right now. You can use it for classification, routing, or small-scale ranking tasks. You do not have to decode. You can just score.

This is TAKEYOURPILLS.TECH. Go ship something.