Datadog Reveals How It Built A/B Testing: Hint, Your Percentiles are Lying (Without CUPED)

You're running an A/B test on a checkout flow. The conversion rate looks flat. Your p95 page load time looks flat too. You ship it. Three days later your support queue explodes because latency didn't spike for the average user, but it doubled for your highest-value customers on the treatment. And you missed it because your percentiles were lying to you.

Not maliciously. Statistically. They had so much variance that your confidence intervals were wider than a barn door, and you didn't have enough traffic to wait for them to tighten. This is the problem Datadog is solving with its rebuilt Experiments platform.

They acquired Eppo last year, tore the experimentation engine down to the studs, and shipped four changes that actually matter. But here's the adoption question up front. This isn't a generic "experimentation is good" pitch. This is about whether you should run your next test inside Datadog's stack, or keep doing what you're already doing in your warehouse.

The Headline Feature: Unbiased CUPED

Let's talk about the headline feature. CUPED. Controlled-experiment Using Pre-Experiment Data. It's a variance reduction technique. You use data about each user from before the experiment to tighten your confidence intervals. The classic example is using someone's pre-experiment spending to reduce noise in a revenue metric. Less noise means smaller confidence intervals. Smaller intervals mean you can call a result faster or with less traffic.

Eppo did this with a regression adjustment approach. They fit a separate subject-level model for each variant. That worked for the overall population, but it had a flaw. The model was fitted on the whole population, so if you sliced into segments, like mobile versus desktop or new versus returning users, the adjustment became biased.

You couldn't reliably look at subgroups and trust the CUPED numbers. Each segment would need its own refitted model, and practically nobody does that.

Datadog rebuilt this as an augmentation method inspired by the original CUPED authors. Instead of predicting outcomes per subject, they add adjustment terms that average to zero across randomizations. One model run. One set of global coefficients. Because those terms are zero in expectation, they add no bias to any segment. You get unbiased CUPED-adjusted results for mobile users, for desktop users, for any slice you want, and those segment results stay consistent with the overall result.

In their example, a raw lift of 16% gets walked down to a 13% CUPED-adjusted lift, and the estimated runtime drops by 52%. That's not a marginal improvement. That's the difference between running a test for two weeks versus four. If you're running a high-velocity experimentation program, that compression changes how you schedule releases.

But here's where a staff engineer should pump the brakes. CUPED is statistical machinery. It's not a button you press and trust blindly. The augmentation method is sound on paper, but you're now relying on Datadog's implementation of a pre-experiment covariate model. If the model misfires on a segment with sparse data, or if your pre-experiment data has drifted, the variance reduction can turn into variance inflation. You still need to know what you're looking at. Don't let a pretty waterfall chart replace your own sense check.

Solving the Percentile Problem

And then there's the percentile problem. Percentile metrics like p90 or p95 page load time are exactly what you want for performance guardrails. They're also traditionally immune to CUPED because standard regression needs a subject-level outcome to predict, and percentiles don't have one.

Datadog's augmentation approach side-steps this. They apply adjustment terms directly to the percentile metric itself. A p90 can now inform your experiment with less traffic than an unadjusted percentile would need. This is genuinely new for commercial platforms.

But a staff engineer will immediately ask: what happens when your RUM definition of a user doesn't match your warehouse definition? If the session split logic differs, your percentile isn't measuring what you think. And if your traffic volume is low, even with CUPED, percentile estimates can remain noisy. The tool helps. It doesn't perform miracles.

Making Global Lift Auditable

Datadog also rebuilt global lift. Global lift answers a specific business question. If I roll this treatment out to everyone eligible, how much does my company-wide metric actually move? At Eppo, global lift and coverage were reported separately, and they didn't multiply cleanly. You couldn't take coverage times local lift and get the reported global lift. Global lift also didn't work on windowed metrics, which meant analysts had to maintain duplicate metric definitions, one windowed and one not, and hope they behaved similarly.

Now the formula is simple. Global lift equals coverage times local lift. If your experiment-eligible users are half your revenue base, coverage is 50%. Local lift is 10%. Global lift is 5%. Every number in the metrics table checks out because it comes from that multiplication. You can audit it with a calculator. Windowed metrics work because coverage is computed independently of the lift calculation. You don't need to maintain a second metric definition just to get a global number.

This is good hygiene. It turns a black-box number into something you can reconstruct. But it assumes your warehouse metric is defined correctly in the first place. Datadog added Copy SQL to help here. For any warehouse-native metric, you can copy the exact query the platform used and run it in your own warehouse. The copied query lands very close to the dashboard number. Small differences can appear because the pipeline is incremental and late-arriving data might not be fully reflected yet. Still, this is the right move. It shifts the burden of proof from "trust us" to "here's the query, check it yourself."

Real-time Signals and Diagnostics

There's also a diagnostic layer that flags traffic imbalance or missing metric data before you waste time interpreting a broken result. These are table stakes for a serious platform, but they're welcome.

Finally, they plugged in RUM and Product Analytics. These update every fifteen minutes. Warehouse metrics often lag for hours. Early in an experiment, before your revenue or retention numbers have landed, you can see behavioral signals. Did pageviews drop? Are error rates up? Are users clicking the new button or ignoring it? You can even watch Session Replays from the first exposed users.

Confidence intervals will still be wide. You shouldn't call the test yet. But you can spot a disaster unfolding and kill it before your Monday morning warehouse job finishes.

So, Who Is This For?

If you're already a Datadog shop with Feature Flags, RUM, and warehouse integrations wired up, Experiments is now a credible place to run tests without exporting data to a separate vendor. The CUPED extensions, especially on percentiles, are a real capability gap compared to most self-serve tools. The fact that global lift is auditable and Copy SQL exists means the trust model is better than most closed platforms.

If you're not in Datadog's ecosystem, the pitch is harder. You're looking at wiring your warehouse into another pipeline, trusting their statistical models, and paying for another product when you might already have dbt, a Jupyter notebook, and a homegrown dashboard that gets you 80% of the way there. The 15-minute RUM metrics only matter if you're already using Datadog's RUM product.

For teams under a thousand users or with low experiment velocity, this is probably overkill. The gains from variance reduction only matter if you're running tests frequently enough that shaving a week off runtime changes your roadmap.

For teams at scale, with performance-sensitive features and real money on the line, the percentile CUPED and real-time behavioral metrics are worth a hard look.

The Scenario and The Real Risk

Picture this. It's Tuesday afternoon. You kick off an experiment on a new recommendation algorithm. The warehouse revenue metric won't be reliable until tomorrow morning. But your RUM metrics update in fifteen minutes. The p95 latency for the recommendation card lookup looks fine in the unadjusted view. Then you apply the CUPED adjustment and the interval tightens. It's still fine. You check the Session Replay and users are interacting normally. You don't call the test yet, but you sleep better. That's the scenario where this platform earns its keep.

The risk isn't that the math is wrong. The risk is that you'll treat the platform like an oracle. You'll kick off a test Friday afternoon, glance at the green confidence intervals Monday morning, and ship. Without verifying the warehouse query. Without checking that your coverage denominator makes sense. Without watching those early Session Replays to see if users are rage-clicking through a broken flow that your p95 hasn't caught yet.

Experimentation platforms don't replace judgment. They compress the time to get signals. What you do with those signals, and how fast you act when they're wrong, is still on you.

TAKEYOURPILLS.TECH. Go ship something.