Netflix's personalization system treated images as opaque IDs across five siloed canvas models, causing a severe cold-start problem where new titles or UI formats defaulted to generic, unpersonalized artwork.
- Multimodal Embeddings: Encode images using CLIP and audio/video using MediaFM, then concatenate them with ID embeddings so the model understands semantic themes.
- Unified Architecture: Collapse five canvas-specific models into one, pooling interaction signals and using reward-based weighting to prevent high-volume canvases from dominating the loss.
- Unbiased Evaluation: Use inverse propensity scoring on randomized exploration traffic to accurately evaluate offline metrics before triggering expensive A/B tests.
Pooling diverse canvases risks high-volume areas dominating the training loss, and the architecture requires dedicated exploration traffic and a strict centralized embedding store to prevent training-serving skew.
Script
Netflix personalizes every thumbnail you see. Two hundred million members, billions of impressions, and for years the system treated every image as an opaque ID. It knew you clicked asset seven-four-three-nine-one, but it had no idea what was in the picture. That worked fine for established titles with lots of history. It failed completely the moment a new show dropped. That is the cold-start problem. And Netflix just published how they solved it at scale.
The Cold-Start Problem and The Multimodal Fix
The old system trained separate models for each artwork canvas: Billboard, vertical box, horizontal panel, short panel, landscape panel. Five models, five silos. A new image on the short-panel canvas had zero history, so the model fell back to popularity heuristics that ignored your taste. You would see generic artwork until enough people clicked to teach the model otherwise.
The fix is multimodal embeddings. They let the model actually look at the image. They encode every artwork with CLIP, the open image-text model. Each image becomes a seven-hundred-sixty-eight-dimensional vector. They concatenate that CLIP embedding with the asset's learned ID embedding, run both through a small MLP, and use the result as the asset representation the model scores against your profile.
Suddenly the model understands what it is showing you. If you consistently engage with artwork featuring a particular comedian, the model carries that preference to their new special. It recognizes the face, the color palette, the visual theme. It transfers taste across titles because the signal lives in image space, not asset ID space. The new asset arrives with an embedding the model already understands.
Unifying Models and Managing Training Data
That same trick collapses five models into one. CLIP embeddings stay nearly identical under crop, resize, and aspect ratio changes. So the billboard version and the short-panel version of the same scene map to almost the same vector. One unified model pools interaction signal across every canvas. Your clicks on the high-traffic billboard inform the artwork you see on the sparsest panel.
But pooling five canvases creates a training problem. Raw impression counts would let the highest-volume canvas dominate the loss. And not every click is worth the same to Netflix. They solve this with reward-based weighting. Every training example is weighted by its interaction type's long-term reward score, not by how many times it happened. The model optimizes for member satisfaction, not raw frequency.
The Evaluation Problem
Then there is the evaluation problem. Judging a new model on old logs is biased, because the old system showed some assets far more often than others. The logs describe what the old policy preferred, not what members would have chosen freely. Netflix runs a small slice of exploration traffic with a randomized policy. They log the exact propensity of showing each asset. Then they reweight observations with inverse propensity scoring. This gives an unbiased offline estimate. In their experience, having propensities known by construction, rather than modeled after the fact, is the single biggest reason their offline numbers predict online outcomes.
Testing the New System
They ran three variants against the old five-model baseline. V1 added image embeddings but kept five separate models. V2 used one unified model with ID embeddings alone. V3 combined both: one unified model with image embeddings.
Offline, V1 and V2 each helped on data-starved canvases like short-panel and landscape, but mostly inside the noise band. V3 won clearly, with a five-point-six-nine percent lift on short-panel.
Online, the story got sharper. Neither V1 nor V2 moved core member metrics alone. Only V3 produced a statistically significant lift. The two ingredients need each other. Image embeddings give the unified model something to learn from on sparse canvases. The unified model gives those embeddings enough data to learn how they map to preference. The effects compound rather than add.
A Real-World Test: The Homepage Redesign
Then Netflix launched its biggest TV home-screen redesign in a decade. Short-panel became the dominant canvas overnight. This was cold-start in its sharpest form: the canvas receiving the most impressions had the least historical data. Waiting for interactions to accumulate would have degraded the experience. They shipped V3 ahead of the launch. In a month-long holdback test, it absorbed the shift immediately, with gains on both discovery and streaming hours larger than in steady state.
Expanding the Architecture
Query-Aware Artwork
The same CLIP infrastructure handed them query-aware artwork almost for free. When you search for an actor, Netflix blends the usual personalization score with the cosine similarity between the CLIP text embedding of your query and the CLIP image embedding of the asset. That alpha mixing weight is tuned through live A/B testing. The result is artwork that surfaces the actor you searched for, even if your general taste says otherwise. One similarity term at scoring time. No new model.
Upgrading to Video Previews
Video previews raised the bar. A static image is one thing. A preview has motion, dialogue, music, pacing. Their first attempt, SeqCLIP, averaged CLIP embeddings across frames. It captured what the preview looked like, but not what it sounded like. So they turned to MediaFM, an in-house multimodal foundation model trained on eighty million shots. It fuses visual SeqCLIP embeddings, audio embeddings, and caption text into a single shot embedding.
Adopting it required no new infrastructure. They plugged MediaFM into the same asset representation slot that CLIP used for artwork. Offline and online, MediaFM beat SeqCLIP, which beat ID-only. The audio and text signals mattered, with gains largest on TV. MediaFM is now the default for video previews across all platforms.
The Foundation: The Netflix Embedding Store
None of this moves without the Netflix Embedding Store. CLIP, SeqCLIP, MediaFM, they all live there. A foundation model encodes raw content into a dense vector once. Every downstream system—artwork, query-aware ranker, video preview model—pulls the same vector through the same interface. Crucially, training and inference see the exact same embedding. No skew.
The store decouples foundation model updates from personalization deployments. A new embedding version gets registered, backfilled across the catalog, and validated on its own. Downstream models consume it through configuration alone. No coordinated release, no downstream code changes.
Gating New Embeddings with a Proxy Task
New embeddings arrive constantly, but end-to-end trials are expensive. Netflix gates them with a cheap proxy task. From the embedding alone, can a linear probe predict which asset wins under an unpersonalized policy? They find the debiased popularity winner per title using their exploration data, label it, and train a linear classifier on the embedding alone, with no title or cast metadata.
They keep the probe linear intentionally. A complex model might memorize metadata. A linear probe isolates whether the embedding itself carries the semantic signal. If the embedding actually captures what makes an asset appealing, the probe beats random chance. If not, the embedding fails before it wastes a full A/B test. The probe, offline IPS lift, and online A/B results all agreed on ranking MediaFM above SeqCLIP. That alignment is why the linear probe now gates every new MediaFM version.
What You Should Actually Borrow
Now, here is the counterweight. Netflix has billions of impressions, a dedicated exploration slice, and an in-house foundation model trained on eighty million shots. If your catalog has thousands of items, not millions, the exact same infrastructure is overkill. But the pattern is not. The real insight is not that CLIP is magic. It is that content-aware representations let you transfer signal across items that look or sound alike, and that transfer is what defangs cold-start.
What should you actually borrow?
- Separate your foundation model from your ranker. Build an embedding store that serves identical vectors at training and inference time. If your training pipeline sees a different vector than your serving stack, you are debugging a ghost.
- Do not trust your logged metrics blindly. You need exploration data or you are grading your new model against your old model's homework. Propensity scoring is not a nice-to-have. It is the difference between an offline metric that predicts reality and one that lies to you.
- Ablate properly. Netflix's V1 and V2 both looked plausible and both failed in production. If they had shipped either one, they would have concluded that multimodal embeddings do not work. The combination is what won. Look for a second blocking factor before you give up on content features.
If you are still treating every item in your catalog as an opaque ID, you are leaving engagement on the table the day anything new launches. Start with pretrained image or text embeddings, concatenate them with your existing ID embeddings, and measure. But measure honestly. Use exploration traffic, use propensity scoring, and do not declare victory until the online A/B test confirms it.
This is TAKEYOURPILLS.TECH. Go ship something.
References
- MAPS: Netflix’s Multimodal Asset Personalization at Scale - Netflix Technology Blog