Datadog built a tiny state-space model to triage security events on the cheap

A language model trained to predict the next token in a sequence inherently assigns a probability to every single token it sees. It builds a mathematical expectation of what should happen next. When it encounters a token—or in this case, a system event—that it did not expect based on the preceding context, that low probability is quantifiable. The model is surprised. That surprise is represented mathematically as negative log-likelihood. Datadog recently realized that this simple byproduct of next-token prediction is actually a perfectly functional, completely label-free anomaly detector for enterprise security logs.

The Problem: The Prohibitive Cost of Reasoning at Scale

Picture this. It is Friday afternoon. You are auditing the cloud bill for your enterprise security operations center. You recently made the decision to run a frontier reasoning model, something on the level of GPT-4, over every single security log your infrastructure generates. You are feeding it every syslog entry, every network flow, every authentication attempt across the entire company. Your systems generate ten billion events a day. You do the math on the invoice. You realize this setup costs tens of millions of dollars. Every single day. You immediately shut it off.

Reasoning models are incredibly powerful. Given a suspicious event, an agent can retrieve a user's recent activity, check patch levels, consult threat intelligence feeds, and explain exactly why a threat is genuine. But applying that level of expensive reasoning to a raw event stream is fundamentally impossible at enterprise scale.

You might think you can just give the agent interactive search capabilities. Let it query the logs when it needs to. But interactive search only finds what you explicitly know to look for.

Spotting one anomalous event hidden among ten billion normal ones, especially when no static rule describes it in advance, is not a search problem. It is a ranking problem. Something has to score every piece of hay to find the needle.

Datadog's Solution: A Two-Stage Detection Pipeline

Datadog solved this by building a two-stage detection pipeline borrowed directly from information retrieval. You pair a fast, cheap retriever model that narrows the field with a slower, expensive reader model that investigates the shortlist.

The retriever in this architecture is called Mambark. It is a tiny, ninety-six point nine million parameter selective state-space model. It was pretrained on hundreds of billions of security events just to predict the next event.

This is where the elegance of the architecture clicks into place. Mambark scores every single event in your firehose in a single forward pass. It uses that negative log-likelihood as an anomaly score. If the model did not expect the log line given the thousands of log lines that came before it, the event gets flagged. It requires absolutely no labeled training data to do this. The model just learns the baseline geometry of what normal activity looks like in your systems, and flags whatever deviates from that geometry.

Why Mamba? The Advantage of State-Space Models

But why a state-space model? Why did they build Mambark on the Mamba architecture instead of using a standard Transformer? The answer comes down to how real attacks operate and how memory scales.

Real security intrusions rarely happen in a burst of ten log lines. They unfold slowly. An attacker might poke around your active directory, wait three days, move laterally to a test server, wait another week, and then exfiltrate data. A single session can span thousands of events. A slow-and-low intrusion generates even more.

If you use a Transformer to track this, you run into the quadratic scaling problem. The attention mechanism in a Transformer requires every token to look back at every previous token. As the context window grows to capture a slow-and-low attack, the memory and compute costs explode quadratically until the model physically cannot fit on a single GPU.

A state-space model changes the math. Mamba runs in linear time. It maintains a compressed, rolling state of the sequence without needing to attend to every historical token simultaneously. Because of this linear scaling, Mambark can condition its predictions on contexts spanning tens of thousands of events, entirely on a single commodity GPU. It holds massive context windows open for pennies. It tracks the slow-and-low attacks that Transformers simply cannot afford to watch.

The Data Engineering Hurdle: Unifying Messy Telemetry

There is a significant data engineering hurdle here. Enterprise security telemetry is incredibly messy. You are dealing with wildly different formats. How do you feed syslog text, structured network flows, and raw authentication records into one tiny ninety-seven million parameter model without hand-engineering a custom feature set for every log type?

Datadog uses a highly uniform data representation. They take every raw event from every source, group them by the entity involved, and flatten them into a fixed field structure. Then, they hash every single field into a shared vocabulary. The model never actually knows which data source produced a token. The architecture, the tokenizer, and the training recipe remain identical regardless of the input.

Datadog claims this process requires no hand-engineered features. That is a bit of a stretch.

Flattening every chaotic log into a fixed structure and hashing them into a shared vocabulary is absolutely feature engineering. It is just highly generalized feature engineering. But it works beautifully. It allows one tiny backbone to process an entire diverse telemetry stream without needing specialized parsers for every vendor tool in your stack.

The Payoff: Four Orders of Magnitude Cheaper

The scale transition of this pipeline is where the real value lies. Let us look at the actual math. Your enterprise generates ten billion raw events a day. Stage one is Mambark. It runs its linear-time sweep over all ten billion events. It strips out the noise and promotes only the most surprising candidates. That drops the volume from ten billion down to about ten thousand events a day. Running that initial sweep on commodity GPUs costs on the order of tens of dollars per day.

Stage two is the frontier AI agent. It receives that prescreened shortlist of ten thousand events. It runs a full multi-step investigation on each candidate and generates an actionable, explained verdict. Because the expensive reasoning model is only looking at ten thousand events instead of ten billion, its operating cost also stays in the tens of dollars per day.

Datadog points out that this two-stage pipeline is roughly four orders of magnitude cheaper than applying an agent to the raw event stream.

That specific comparison is a bit of a strawman. Nobody in the security industry actually attempts to run frontier models on billions of raw logs. Everyone uses some form of pre-filter. Teams traditionally rely on deterministic SIEM tools paired with rigid playbooks to filter the noise. But deterministic rules only catch known patterns. They catch the threats you have already seen. Mambark provides recall at scale for the unknown unknowns.

Operational Realities and Frictions

There are operational realities you need to accept if you adopt this kind of probabilistic pipeline over static rules.

Debugging False Negatives

The first is debugging. If Mambark silently ignores a real intrusion because the specific sequence of tokens happened to have a high probability, you have a false negative.

With a traditional rule-based system, a false negative is auditable. You check the rule, find the logic error, fix it, and move on. With a state-space model, there is no signature to inspect. There is no threshold dial to turn. The model simply believed the malicious sequence was normal.

The False Positive Problem

The second reality is the false positive rate of the agent. Even with a massive reduction in volume, Mambark is still passing ten thousand anomalous events a day to the stage two LLM. If that frontier agent has a highly impressive ninety-nine percent accuracy rate, a one percent error rate on ten thousand events still generates one hundred wild goose chases a day.

Alert fatigue is the primary enemy of a security operations center. One hundred hallucinated threats a day requires human analysts to investigate and clear them. Replacing deterministic rule fatigue with AI hallucination fatigue does not solve the human bottleneck.

The Commercial Reality

And finally, there is the commercial reality. Datadog highlights their internal compute cost of tens of dollars per day. Historically, internal compute costs for managed software providers have zero correlation with the final pricing tiers they charge customers. The technical efficiency is real, but the final invoice will reflect enterprise pricing models, not raw GPU hours.

A Masterclass in System Design

Despite those operational frictions, the architecture itself is a masterclass in system design. This is how you deploy AI at fleet scale. You do not point a massive reasoning engine at a firehose.

You pair a tiny, highly efficient probabilistic filter with a slow, heavy reader. You use the inherent mathematical byproduct of a next-token predictor—its own measurable surprise—to isolate signal from noise without needing a single label. You use linear-scaling architectures like Mamba to hold massive context windows open, tracking threats across time without bankrupting your compute budget.

You let the small model do the reading, and you save the large model for the thinking.

This is TAKEYOURPILLS.TECH. Go ship something.