Stop asking AI to draw charts and make it write SVG instead

The Architecture of Constraint

Picture this. Your product manager wants a Slackbot that can answer customer questions with charts. Your brain immediately paints a nightmare scenario. You envision deploying a bloated two-gigabyte headless Chrome Docker image just to take screenshots of web pages. Worse, you picture a chatbot confidently hallucinating a bar chart with entirely fake revenue numbers, dropping it in an executive channel, and ruining your weekend.

The secret to building this feature reliably does not involve finding a better multimodal image model. It requires exactly the opposite. You lock the language model out of the data tier entirely. You keep it away from rendering pixels. You force it to write Vega-Lite JSON instead.

The architecture of constraint starts with a simple rule. You never ask the language model to draw a picture. You ask it to describe a picture using a strict declarative specification.

Vega-Lite is a JSON grammar built exactly for this. You do not prompt the model for an image. You tell it to construct a JSON object where the x-axis maps to a temporal field called "month", the y-axis maps to a quantitative field called "review_count", and the visual mark is a bar. That is the entire chart. The language model operates entirely within a text-based schema.

You might ask why you would not just have the language model output Python code and run Matplotlib. Generating Python used to be the default way to handle dynamic AI data visualization. But executing arbitrary, LLM-generated Python code in your backend is a security and maintenance liability.

• You have to manage a sandbox.
• You have to maintain a runtime.
• You have to handle execution timeouts.

Generating a declarative JSON schema bypasses all of that. You do not evaluate code. You parse text.

A Safe Data Visualization Pipeline

Forcing an LLM to output Vega-Lite is rapidly becoming the established baseline for AI data visualization. It restricts the language model to a task it is actually good at doing. It fills in a well-defined schema and chooses between a line chart or a bar chart, without having to calculate the exact coordinates of a pixel.

In a safe pipeline, the language model never sees the actual numbers before the chart is rendered. It acts only as a translation layer. The model writes a SQL query based on the user's prompt. Your backend runs that query against the database. Your backend then takes the raw results and stitches them into the data.values array of the Vega-Lite JSON payload. The model gets creative control over the presentation layer. It gets zero creative license over the math.

This requires feeding the language model an understanding of your database schema so it can write the SQL. If you have fifty or sixty tables, dumping the entire data definition language (DDL) into every prompt wastes tokens. You can use schema-narrowing techniques to filter the schema down. Tools exist to map natural language questions to relevant tables, cutting a fifty-table schema down to twenty-five tables before the model writes a single character.

A quick check on that specific optimization. With modern prompt caching from Anthropic or OpenAI, passing a filtered schema dump is often incredibly cheap and fast. Maintaining a dedicated semantic search layer just for schema retrieval might be unnecessary overhead for smaller databases. You can often just cache the entire DDL dump and let the model sort it out.

Locking Down SQL Execution

The SQL execution itself is where the actual danger lives. If the language model is writing SQL to fetch chart data, preventing data leaks between tenants is your biggest operational risk.

A confidently hallucinated query is one WHERE clause away from joining Organization A's private data into Organization B's chat window.

The engineering team behind a tool called LiveReview recently published their pipeline for this exact problem. They run an application-level SQL guard. Their middleware parses the generated query.

• It rejects non-read operations.
• It checks tables against a denylist.
• It specifically looks for the shapes of tenant-isolation bypasses, like a constant-versus-constant comparison such as OR 1=1.

Parsing arbitrary SQL in application code to catch logical bypasses is notoriously fragile. If you are relying on application-level syntax checks to catch LLM-generated cross-tenant joins, you are walking on thin ice.

The boring, secure way to solve this is database-level Row Level Security. You enforce tenant isolation in PostgreSQL directly. You bind the query execution to a scoped database role representing the active user. If the language model writes a malicious or hallucinated join, the database engine simply returns no rows. The database must guarantee tenant isolation, not a middleware string check.

Assuming your query execution is locked down at the database level, a reliable charting pipeline implements another smart constraint: a two-step query process. Before fetching the actual data, the model writes a query to count the results. If a user asks a question that returns four thousand rows, a bar chart with four thousand bars is completely useless. The pipeline checks the size first. If the result set is too massive to visualize, it bails out and hands the user a CSV file instead.

Rendering: From Web to Chat

Once you have a sensible amount of data stitched into your Vega-Lite JSON, you have to render it.

If the user is on your web application, the path is straightforward. You pass the JSON spec directly to a frontend library like react-vega. The user's browser takes the JSON and renders an interactive SVG. They get hover tooltips. They get smooth resizing. Your backend does zero image processing. It just ships JSON over the wire.

But chat environments break that model. Rendering these charts in Slack or Discord requires an actual image file. Slack cannot interpret a Vega-Lite spec.

This brings us back to the two-gigabyte headless Chrome container. The legacy way to render a chart server-side is to spin up a headless browser instance, load an HTML page with a JavaScript charting library, take a screenshot, and extract the image. It is slow, memory-intensive, and prone to hanging.

You avoid this entirely by using a tool called vl-convert. It is a standalone Rust binary that you run on your backend. It takes your Vega-Lite JSON as an input and rasterizes it straight to flat PNG bytes. No headless browser is involved. No massive memory footprint. It is fast, deterministic, and tiny.

The exact same chart spec is routed to the web frontend as JSON, or piped through a fast Rust CLI to become a static PNG for a chat thread. The destination dictates the rendering method. The language model does not know or care where the chart is going.

A Pattern for Safe AI

This entire pattern teaches us how to safely deploy generative AI for data analysis. You do not treat the model as an analyst. You treat it as a constrained interface between human intent and a strict technical contract. By forcing the model to output SQL for data retrieval and Vega-Lite JSON for presentation, you completely isolate the unpredictable system from both your raw data and your pixel rendering.

Adopting this architecture means integrating a few moving parts.

• You need a safe execution sandbox for dynamic SQL.
• You need Row Level Security implemented at the database layer.
• You need an external Rust binary integrated into your backend for static rendering.

It is not as simple as dropping a new API key into a script. The tradeoff is absolute control. You build a pipeline where the AI provides the chart taste, and your deterministic code provides the facts. Trusting an LLM with actual numbers is a risk you never have to take.

This is TAKEYOURPILLS.TECH. Go ship something.