How Netflix solved the recommendation cold-start problem using multimodal embeddings

The Cold-Start Problem with Artwork

Netflix's artwork models used to be blind. They knew you clicked asset four-seven-two-one, but they had no idea if it showed a comedian, a sunset, or a jump scare. Every image was just an opaque ID in a database. When a new title launched, its artwork carried zero interaction history, so the system fell back to popularity heuristics that ignored your taste completely. Only after enough clicks piled up could personalization kick in.

That's the classic cold-start problem, and every recommender faces it. But Netflix just published how they solved it in production, and the solution contains a harder lesson than the headline suggests. It isn't simply that multimodal embeddings fix cold-start. It's that they fix it only when you pair them with a structural change to how your models consume data. One without the other flat-out fails. And Netflix has the A/B test results to prove it.

The Setup: From Data Silos to a Unified Model

Here's the setup. Netflix generates a portfolio of artworks for every title. Billboards for TV. Vertical boxes for mobile. Horizontal panels, short panels, landscape panels. Historically, each canvas had its own dedicated model. Five models in total. The reason wasn't clever engineering. It was a limitation. An ID-based model can't tell that a cropped billboard and a vertical thumbnail came from the same source image. To the model, they're unrelated IDs. So signal couldn't flow between canvases. Each lived in its own data silo, and each suffered its own cold-start every time a new show dropped.

The pressure to fix this came from a real product change. Netflix was preparing its largest TV home-screen redesign in a decade, which would make short-panel the dominant artwork canvas effectively overnight. The canvas about to receive the most impressions had the least historical data. Waiting for short-panel interactions to accumulate would have degraded the experience for millions of users. They needed a system that could absorb that shift immediately, not after months of exploration.

The Solution: Giving the Model Eyes

The new approach gives the model eyes. Every artwork gets encoded by CLIP, the pretrained image-text model from OpenAI. That produces a 768-dimensional vector representing what's actually in the picture. Netflix concatenates this CLIP embedding with the old learned ID embedding, passes the combined vector through a small MLP, and uses that as the asset representation to score against a member.

The change is surgical, but the effect is structural. Now, when a brand-new artwork enters the system, it arrives with a semantic signature the model already understands. If you've consistently engaged with artwork featuring a particular actor, that preference lives in the embedding space, not tied to a specific asset ID. When that actor appears in a new title, the model spots them immediately and prioritizes the artwork that puts them front and center. It has never seen that exact image before, but the embedding space carries your taste across titles. Cold-start stops being a blind spot and becomes something the model has an informed opinion about.

The A/B Test: Proving the Whole is Greater Than the Sum of its Parts

That alone sounds like the win. But Netflix ran a proper ablation, and the results are humbling. They tested three variants against the old five-model baseline.

• Variant one: Kept the five per-canvas models but added image embeddings to each.
• Variant two: Unified everything into a single model across all five canvases, but used only ID embeddings, no image content.
• Variant three: Combined both: one unified model plus image embeddings.

Offline, using inverse propensity scoring on exploration traffic, variant one and variant two each helped slightly on data-starved canvases like short-panel and landscape. But most of their lifts sat inside the noise band. Variant three, however, jumped over five percent on short-panel.

Online, across a four-week A/B test on every device platform, the story got sharper. Variant one was flat and non-significant. Variant two was flat and non-significant. Only variant three produced a statistically significant lift in core discovery metrics and streaming hours.

The two ingredients need each other. Image embeddings give the unified model a way to understand new assets. The unified model gives the embeddings enough cross-canvas data to learn how those visual features map to member preference. Mature canvases teach the shared model, and that knowledge transfers straight to the sparse ones. The gains compound rather than add. V3's short-panel lift exceeded what you'd get from adding V1 and V2 together.

This is where a staff engineer would tap the table and tell you to write this down. Most teams trying to improve a recommender would ship exactly variant one. They'd sprinkle content embeddings into their existing pipeline, run the A/B test, see flat metrics, and conclude that image features don't work for their problem. They'd kill the project and move on. Netflix's real insight wasn't the CLIP architecture. It was recognizing that image embeddings and model consolidation were blocking factors for each other. Neither makes sense without the other. That's a systems-level diagnosis, not a model-level one.

If you're running a team that's one A/B test away from abandoning a promising embedding project, ask whether you're actually testing the embedding, or whether you're testing it inside a data architecture that's too small to use it.

A Freebie: Query-Aware Ranking

The same CLIP vectors handed Netflix a second feature almost for free. When you search for something specific, your intent is explicit. If you type Adam Sandler, you want results that show him. Because CLIP embeds both text and images into the same space, Netflix can measure query-artwork alignment with cosine similarity between the text embedding of your query and the image embedding of each candidate. They blend that alignment term with the personalization score, tuned through online A/B testing. The first term says what you generally like. The second says what you just asked for.

Crucially, this required no new model training. The embeddings were already sitting in the asset representation. Query-aware ranking became a scoring-time addition, not a months-long model project.

Beyond Static Images: Video Previews and MediaFM

Video previews raised the bar further. A still image is static, but a preview unfolds over time. Its appeal comes from motion, pacing, dialogue, music. Netflix's older video models treated previews as opaque IDs, just like the artwork models did. Their first content-aware attempt, called SeqCLIP, encoded each frame with CLIP and averaged the vectors. That captured what a preview looked like, but missed what it sounded like. The dialogue and soundtrack that carry so much tone were invisible.

So they built MediaFM, an in-house tri-modal foundation model trained on eighty million shots. It fuses three signals per shot:

• Visual embeddings from SeqCLIP,
• Audio embeddings from a pretrained speech and audio model,
• And text from captions encoded through a large text model.

The result is a single embedding that represents what a preview actually feels like. Offline and online, MediaFM beat SeqCLIP, which beat the ID-only baseline. The audio and timed-text signals weren't marginal improvements. They were the decisive factor. MediaFM is now the default video preview embedding across all platforms.

The Architectural Keystone: The Embedding Store

Now, you don't have Netflix's budget, and you probably don't have an eighty-million-shot foundation model sitting on your cluster. But the pattern underneath is something you can steal. All of these embeddings—CLIP, SeqCLIP, MediaFM—live in the Netflix Embedding Store. It's a shared component of their AI platform that hosts dense vectors for titles, games, member profiles, and multimedia assets.

A foundation model encodes raw content once, and the Embedding Store serves that exact same vector to every downstream system at training time and at inference time. No skew. No drift between what the model learned from and what it sees in production. The key property is decoupling. A new embedding, or a new version, gets registered and backfilled across the catalog without touching any consumer model's training or serving code. Once it's in the store, downstream rankers pick it up through configuration alone. That's how they swapped CLIP into the artwork model, stood up query-aware search on the same vectors, and rolled MediaFM through video previews, each as an independent change rather than a cross-team migration.

You don't need Netflix's scale to copy that boundary. You need a single place where embeddings are produced, versioned, and served identically to training and inference. Treat your embeddings as a stable data product, not as a byproduct of your training pipeline.

How to Evaluate: Avoiding Biased Logs and Wasted Effort

Inverse Propensity Scoring

There's another lesson hiding in how they evaluate. Judging a new model on old logs is biased, because your production policy shows some assets far more often than others. The logs describe what the system preferred, not what members would have chosen if everything had been shown equally. Netflix handles this with inverse propensity scoring on a dedicated slice of exploration traffic. A small fraction of users sees truly randomized asset selection, and the system logs the exact probability of each impression. Reweighting every observation by the inverse of that probability makes the offline estimator unbiased. Netflix reports that having propensities known by construction, rather than modeled after the fact, is the single biggest reason their offline numbers track online outcomes. It's a prerequisite, not a luxury. If your offline metrics don't correlate with your A/B results, check whether your logs are lying about what members actually prefer.

The Linear Probe Gatekeeper

Finally, there's the cheap trick that now gates every new embedding before it sees a full A/B test. End-to-end trials are expensive. They cost data engineering, model retraining, and weeks of traffic. So Netflix asks a blunt question first. From the embedding alone, can a linear probe predict which asset wins under a plain unpersonalized popularity policy? They use exploration data to find the debiased popularity winner for a fixed set of titles, label it, and train a linear classifier to recover that label from the embedding with no other features allowed.

If the embedding truly captures the semantic drivers of popularity, the probe finds them. If it doesn't, the probe performs no better than a coin flip, and the embedding dies before any production code is written. They used this exact screen to prune candidates down to SeqCLIP and MediaFM. The linear probe accuracy, offline IPS lift, and online A/B results all agreed on the same ranking. That alignment is why the probe now gates every MediaFM release. It's a few hours of work that saves weeks of misplaced effort.

The Verdict and Takeaways

So what's the verdict? If you're running a recommender system that faces cold-start, pretrained multimodal embeddings are almost certainly worth exploring. But drop them into your existing architecture and you'll likely get variant one: flat metrics and a canceled initiative. Look for the second blocking factor. Maybe your categories are too granular. Maybe your data is siloed by surface or locale. Maybe your training and serving embeddings diverge. The embedding is only half the prescription.

And if you're not running a recommender at all, take the meta-lessons.

• Separate your foundation model pipeline from your consumer models with a stable embedding contract.
• Run a small slice of true exploration traffic so your offline metrics aren't fiction.
• And before you commit to a full integration, build a cheap linear probe that answers whether your new signal actually encodes the outcome you care about.

Netflix spent enormous resources learning that the combination matters more than any single ingredient. You can get the same insight for a lot less if you know what to look for.

TAKEYOURPILLS.TECH. Go ship something.