> ## Content Index
> Fetch the complete content index at: https://www.takeyourpills.tech/llms.txt
> Use this file to discover other available public pages before exploring further.

# Treating recommendation engines like LLMs just doubled Meta's training efficiency
- URL: https://www.takeyourpills.tech/treating-recommendation-engines-like-llms-just-doubled-metas-training-efficiency/
- Published: 2026-09-08T22:44:13.000Z
- Updated: 2026-09-08T22:44:13.000Z
- Description: Meta deployed GEM, a generative foundation model for ad recommendations trained across thousands of GPUs. By rethinking their synchronization architecture and applying LLM scaling laws, they overcame massive infrastructure bottlenecks to double their training efficiency....
- Author: Youre Pena
- Tags: AI Infrastructure & FinOps

![audio-thumbnail](https://www.takeyourpills.tech/content/images/2026/09/cover.png)

Treating recommendation engines like LLMs just doubled Meta's training efficiency

0:00

/0

1×

Clinical Summary

Diagnosis

Standard LLM infrastructure fails when applied to recommendation systems due to sparse data, variable-length sequences, and extreme numerical sensitivity. Relying on standard attention kernels and padding can waste up to 50% of available compute.

Prescription

- **Custom Kernels:** Replace standard attention with **Jagged Flash Attention** and GDPA to natively handle unpadded, variable-length tensors.
- **Smart Quantization:** Implement a full numerical stack using **MXFP8** block-wise quantization and stochastic rounding to preserve model accuracy.
- **5D Parallelism:** Overlap sparse embedding communication with dense compute by mapping parallelization strategies directly to **RoCE** and **NVLink** network topologies using custom **NCCLX** extensions.

Side Effects

Replicating these optimizations requires complete vertical control over hardware, custom kernels, and network architectures, rendering it unfeasible for teams without hyperscaler resources.

Potency

Achieved 25% Model FLOPs Utilization (MFU) and quadrupled training FLOPs in 12 months.

#### Script

Meta wants you to know they doubled the training efficiency of their ads recommendation model by treating it like an LLM. Twenty to twenty-five percent Model FLOPs Utilization. Four times the training FLOPs in twelve months. Those are the numbers in the headline.

Now here's the contrarian read. This isn't a story about how LLM techniques gracefully ported over to save a recommendation engine. It's a story about how **every piece of standard LLM infrastructure failed** when it touched rec-system data properties — and Meta had to rebuild the stack from CUDA kernels to network topology to make it work.

GEM is the central brain behind ads on Instagram and Facebook. It’s a hybrid architecture with trillions of sparse embedding parameters and billions of dense parameters. It trains on your activity history plus ad content using custom attention modules that don't look anything like standard transformer blocks. Self-attention on long but windowed sequences. Cross-attention between users and ads with asymmetric query and key lengths. Pooled attention that compresses history into short queries against long keys. This is not a language model wearing an ads costume. *It's a different species.*

## Where Standard LLM Tech Broke

And that's the gap between the framing and the reality. When Meta says they trained this at LLM scale, they don't mean they dropped a standard checkpoint into an ads pipeline and watched it fly. They mean standard FlashAttention, standard FSDP, and standard FP8 training all broke — *badly* — and they spent eighteen months writing replacements.

### Jagged Flash Attention

Let's walk through the breakage. Standard FlashAttention assumes dense, fixed-length sequences. In GEM, user histories vary from hundreds to tens of thousands of tokens. Pad everything to the max length and you throw away fifty percent of your FLOPs.

Meta's first major fix is Jagged Flash Attention. It's a custom kernel that operates directly on variable-length tensors without padding, supporting asymmetric queries and keys, custom attention biases, and efficient backward passes. After four generations of refinement, the latest version hits forty to one-hundred-forty percent TFLOPS improvement over their v2, contributing an eighteen-point-five percent relative local MFU gain.

### Generalized Dot-Product Attention (GDPA)

Then there's the attention diversity problem. FlashAttention is built for the uniform dense shapes of language modeling. On GEM's real production traffic — short asymmetric KV sequences, jagged batches, large batch sizes — standard kernels show a 2.6x forward performance gap and up to a 4x backward gap versus synthetic benchmarks.

Meta's fix is GDPA, Generalized Dot-Product Attention. It unifies self-attention, pooled attention, and cross-attention under one kernel by replacing the softmax stage with GELU or SiLU activations, then redesigning the pipeline for short loops and zigzag tile scheduling across streaming multiprocessors. Under production settings, it's up to 3.5x faster than FlashAttention 4 on short KV sequences. Across the full model, that’s over thirty percent end-to-end training throughput improvement.

For the long-sequence self-attention, they moved from full quadratic attention to sliding window, then pushed further to block-aligned attention with fixed sixty-four-token blocks. Complexity drops from order L squared to order L. They also fused rotary backward into the attention epilogue to kill another memory-bound kernel, gaining a thirty percent MFU improvement in that layer alone.

### Low-Precision Training

Low precision is another LLM hand-me-down that doesn't stick. FP8 delivers double the peak FLOPs over FP16 on the latest GPUs. But click-through rate and conversion rate predictions are numerically sensitive. Naive quantization regresses model quality.

Meta built MXFP8 attention and MLP training with block-wise quantization, random Hadamard transforms to spread outliers, stochastic rounding to kill deterministic bias, and selective higher-precision weight gradients where activations get spicy. They fused quantization into upstream normalization and projection kernels so the casting and scaling doesn't introduce extra HBM round-trips. Even with all that, they still fall back to BF16 in later layers where quantization error propagates too aggressively. This is not a datatype swap. It's a full numerical stack.

### Scaling & Parallelism

Now scale this across thousands of GPUs. Standard LLM parallelism assumes mostly dense parameters. GEM has trillions of sparse embeddings. Standard FSDP across the full job degrades because effective bandwidth drops as group size grows, especially across Meta's oversubscribed cross-zone RoCE.

Their solution is topology-aware 5D parallelism.

- **For dense parameters:** Expert Parallelism on intra-node NVLink, plus 2D FSDP with small shard groups inside an AI zone, plus DDP replicas across zones. Each dimension's collective volume is matched to the bandwidth available at its tier.
- **For sparse parameters:** Fully Sharded 2D Model Parallelism that eliminates the memory overhead of keeping full embedding table copies on every replica. The extra communication lands on fast NVLink and gets overlapped with dense compute.

They also built SM-free collectives through NCCLX. Standard all-gather kernels occupy around twenty-four streaming multiprocessors. NCCLX offloads data movement to copy engines and RDMA, cutting SM usage to one. That reclaims twenty-three SMs for actual compute, resulting in a five percent QPS gain at full training scale.

And because jagged sequences create data-driven load imbalance — some ranks finish in half the time, others drag — they developed Base Batch Shuffling. Distributed readers generate small sub-batches of 128 samples, sort them by total sequence length, then interleave heaviest with lightest when merging into full training batches. This requires zero cross-rank communication and immediately drops the max-over-average workload gap, yielding a four percent efficiency gain.

## The Staff Engineer Check

Picture this. It's Tuesday. You're staring at your own training pipeline. Maybe a few hundred million parameters. Your MFU is sitting at six or eight percent. You read Meta's blog post. Twenty-five percent. Custom kernels. Five-dimensional parallelism. Topology-aware collectives.

You start asking yourself whether you need Jagged Flash Attention. Whether you should shard your embeddings across a custom network hierarchy. Whether your batching logic is leaving FLOPs on the floor.

Here's the staff engineer check. Meta hit these numbers because they control the entire vertical. The GPU generation. The three-tier network with NVLink inside the host, RoCE inside the AI zone, and oversubscribed RoCE between zones. The trillion-parameter embedding tables. The compiler graphs. The in-house NCCL extensions. You don't have most of that stack. And if you're not operating at that scale, the engineering cost of co-designing kernels, precision recipes, and parallelism strategies for your specific workload will dwarf any efficiency gain you might squeeze out.

## A Market Signal, Not an Engineering Blueprint

The real question isn't whether you can replicate Meta's setup. **You can't**, not without their hardware and their data scale. The question is whether any of these techniques leak out in a form you can adopt. Jagged Flash Attention and GDPA are described in enough detail that similar implementations could surface in PyTorch or Triton. The MXFP8 stability tricks and SM-free collective ideas are conceptually portable.

But the full integration — mapping parallelism dimensions to network topology tiers, eliminating memory overhead on trillion-parameter sparse tables, hiding every collective behind compute — that's hyperscaler infrastructure. It's not a library you pip install.

If you're building recommendation models, even at one percent of GEM's scale, the lesson is cautionary. Recommendation workloads are memory-bound, jagged, numerically sensitive, and communication-heavy in ways that dense LLM training simply isn't. Don't assume the next release of your favorite LLM training framework will handle your rec system gracefully. **It won't.** Meta just proved you need a dedicated kernel library and a network-aware parallelism strategy to even get close.

For the rest of us — which is almost all of us — this matters as market signal, not engineering blueprint. Meta's ads model trains faster and cheaper now. That means more iterations, more experiments, tighter feedback loops on ad targeting. The downstream effect lands on the product experience and the business model. It does not land on your git repo. Don't rewrite your data loader. Don't hunt for jagged attention implementations unless you know padding is literally burning half your compute. Don't reconfigure your stack because a hyperscaler published their internal CUDA recipes.

Meta's efficiency doubling is serious infrastructure work. It's also a reminder that when you hybridize two hard problems — recommendation systems and LLM-scale distributed training — the solution isn't a port. It's a full-stack, hardware-aware rebuild that most teams should observe from a distance.

[TAKEYOURPILLS.TECH](https://takeyourpills.tech/?ref=takeyourpills.tech). Go ship something.

## References

- [GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model](https://engineering.fb.com/2026/08/03/ml-applications/training-gem-at-llm-scale-meta-ads-recommendation-foundation-model/?ref=takeyourpills.tech) \- Engineering at Meta