Providing raw database schemas to an LLM is not enough for accurate SQL generation because crucial business logic lives in external codebase rules, not table definitions. Scaling this context dynamically across thousands of tables overwhelms context windows, degrades latency, and leads to hallucinations.
- Offline Context Crawling: Automate daily jobs to parse Airflow and Spark scripts to extract true table lineage and downstream usage patterns.
- Iterative Execution: Utilize the Model Context Protocol to allow the agent to autonomously write queries, check execution results, and debug failed joins.
- AST Evaluation: Compare Abstract Syntax Trees instead of raw text strings to bypass formatting differences, pairing it with an LLM judge for regression testing.
Permitting an autonomous agent to execute unmonitored, trial-and-error queries against live data incurs astronomical compute costs and demands constant human babysitting to curate semantic memory.
Script
OpenAI built an internal AI agent called Kepler to answer data questions across six hundred petabytes of information and seventy thousand tables. But the secret to making this actually work isn't a massive, million-token context window. The secret is an unsexy, offline job that crawls Airflow scripts every single night.
The Problem with Raw Schemas
If you want an LLM to reliably write SQL, dumping a basic database schema into the prompt is not enough. Schemas tell you column names and data types. They tell you that a column is a string or an integer. They do not tell you that a table is pre-filtered by user feedback. They do not tell you that certain IDs are encrypted while others are plain text, meaning joining them will return zero rows. They do not tell you that a specific column adjusts for fraud rates, or the infuriating detail that Trino arrays are one-indexed.
Nuance lives outside the database. Missing just one of these details can lead to an answer that is wrong by an order of magnitude.
When you have seventy thousand datasets, manually updating table descriptions to capture this logic is impossible. Human data catalogs rot. You also can't shove seventy thousand schemas and their query histories into a model's context window at inference time. It is too much data, it destroys latency, and models get lost in the noise.
Automating Context with Offline Jobs
So OpenAI automated the context generation completely offline. Every day, they launch parallel jobs using Codex tasks. These tasks crawl the company's own codebase. They read Airflow folders, Spark observability scripts, and agent files. They analyze the code to extract a table's actual purpose. They figure out downstream usage patterns, the exact grain of the data, the freshness constraints, and rules for when to use one table over another.
Instead of a generic label, the agent learns that a specific table contains first-party traffic, not third-party traffic, and that some fields will be null if they fall outside an hourly window. All of this extracted lineage is embedded and stored.
They do the exact same thing for human context. Internal knowledge from Slack threads, Notion documents, and Google Drive files is chunked, embedded, and dropped into blob storage with a retrieval service enforcing permissions.
By the time a user asks a question, the agent searches this preprocessed knowledge base. It retrieves only the highly enriched metadata for the exact tables that matter.
Iterative Execution and Tool Use
Armed with this context, Kepler uses the Model Context Protocol to execute tools iteratively. It writes a query, runs it, checks the results, and loops back if it gets an empty set. If it picks two tables to join but uses the wrong key and gets zero rows, it knows to back up, re-read the schema, and try a different join. It explores the data autonomously.
Interestingly, OpenAI found that giving the model too many specific tool calls actually degraded performance. When tools had overlapping capabilities, the model got confused by the subtle differences. They also found that providing overly prescriptive instructions hurt accuracy. There are too many branches in logic for analytical questions. General prompts that allowed the model to use its own reasoning outperformed rigid instruction sets.
The Reality of Production Costs
This interactive, iterative execution is mechanically impressive. But pause and consider the reality of running this in production. Letting an agent run an unmonitored trial-and-error loop of SQL queries against live, petabyte-scale warehouse data is a recipe for astronomical compute costs.
A bad, unindexed cross-join generated during an LLM's reasoning phase will burn through resources instantly. OpenAI enforces pass-through authentication, meaning Kepler uses the requester's credentials directly. It will not grant extra access to sensitive tables you aren't supposed to see. But access control does not prevent inefficient, expensive queries from hitting the data warehouse while the bot is guessing and checking its way to an answer.
Solving the Regression Testing Dilemma
The other massive engineering challenge OpenAI had to solve was regression testing. When you update the system, how do you automatically evaluate an LLM's SQL output? Exact string equality is a terrible metric. A developer can write a date filter in ten different ways, and all of them are valid.
To solve this, OpenAI built an Abstract Syntax Tree based evaluation pipeline. They maintain a curated set of question-and-answer pairs, mapped to expected, ground-truth SQL queries for their most important metrics. During an eval run, they hit the agent's endpoint to generate SQL. Then, they take both the generated query and the manually curated expected query, and they convert both into Abstract Syntax Trees.
By comparing the AST representations instead of the raw text strings, the grading system bypasses minor syntax formatting entirely. They also execute both queries. The result sets are fed into an LLM acting as a judge. This judge allows for a little wiggle room on things that don't meaningfully impact the answer. If a generated query returns a float and the expected query returns an integer, the judge can evaluate if that precision actually matters for the specific metric being tested.
They also expose the agent's chain of thought in the eval outputs, which makes debugging failures much easier. If the model picks a curated dashboard table instead of a raw logs table, the chain of thought shows exactly why the reasoning branched.
The Immense Operational Overhead
This AST parser approach is incredibly sharp. But the operational overhead backing it is immense. They rely on manually curated expected SQL for these evaluations. Maintaining a library of ground-truth SQL queries across thousands of constantly drifting schemas is a massive, unending data engineering burden. Who updates the eval pipeline when a downstream table is deprecated?
On top of that, they built a scoped semantic memory system so the agent can learn from user corrections. If a user tells the agent to always use Pacific Time for a certain metric, that gets embedded and injected into future queries. But memory is just a stateful distributed cache. Generated memories frequently conflict or are simply incorrect. OpenAI has to run offline pruning jobs, and users have to manually curate their own team and global scopes to stop bad instructions from poisoning the well. It requires constant human babysitting.
What Should a Normal Company Steal?
So, which of these architectural patterns actually apply to a normal company? OpenAI operates at a scale most teams will never see. Managing seventy thousand datasets is a distributed systems problem. At that scale, you need automated codebase crawling and semantic search just to find the right table. But for ninety-nine percent of organizations, dealing with maybe a few hundred core tables in Snowflake or BigQuery, this architecture is severe premature optimization.
If you have a hundred tables, you do not need to build an AI agent to parse your Airflow code to figure out what your data means. You need a strongly typed semantic layer like dbt metrics or Cube. You need to fix your data modeling, enforce naming conventions, and write documentation. Do not build an LLM to navigate a messy warehouse. Clean up your warehouse and use a standard BI tool.
However, the offline evaluation pipeline is a pattern you absolutely should steal. If you are building any agentic workflow that generates code or queries, exact text matching will fail you. Building an AST comparison to verify structural intent, pairing it with an LLM judge to evaluate the actual execution results, and exposing the reasoning chain for debugging, is exactly how you prevent a model from quietly regressing.
The real intelligence of an AI agent is rarely in the prompt. It is in the unglamorous, offline engineering that happens long before the user asks a question.
This is TAKEYOURPILLS.TECH. Go ship something.