Output executed as server-side code
The highest-severity output-handling bugs feed the model's reply into a sink that runs it as code on the server: a Python eval/exec, an OS shell, an unsafe YAML or deserialization loader, or a template engine. The app asked the model to compose a receipt, a manifest, a computed column, or a diagnostic command - and the interpreter executes what it composed, turning improper output handling into server-side code execution or the disclosure of a secret only the app process can see.
When the server runs what the model wrote
The earlier lessons in this path dealt with output that runs somewhere else: a browser renders it, or a database executes it as a query. This lesson is about the worst place it can run, which is your own server. The application asks a model to compose something, then feeds the reply to an interpreter that runs it as code inside the app's process. Composing the content and authoring the code become one step, and the interpreter has no way to tell the two apart.
A sink is the place where output leaves your code and enters something that acts on it. An execution sink runs its input. Four kinds show up in AI apps: a template engine that evaluates the expressions in a draft, a Python evaluation call that runs a string as code, an operating-system shell that runs a string as a command, and an unsafe loader that builds live objects from a document as it reads it. Each one acts with the reach of the process it runs in, so the process environment, the filesystem and the application config are all in range.
The reason this is a security problem and not just a design smell is that model output is untrusted. A model writes what it is steered to write, and the request that steers it often carries attacker-controlled text: a customer memo copied onto a receipt, a column description shared across a workspace, a template line a contributor authored. Treat every one of these the way you already treat a form field. The model did not add trust to the text by passing it through.
How a tester spots an execution sink
You find this class by asking one question at every point where model output leaves the model: does the application run this, or only display it? If it runs it, the interpreter is your server, and you map what that interpreter can reach.
Some signs that output is being run rather than shown:
- The feature turns a request into a program and acts on the result: a calculator that returns a number, a diagnostics panel that streams command output, a report that fills in computed cells.
- The output is a format with an interpreter behind it: a template, a configuration document, a spec that gets compiled.
- The app shows you both the text the model produced and the effect it had, and the two differ. If a draft contains an expression and the rendered version shows a value instead of the expression, the engine evaluated it.
- A value the model should only quote comes back resolved, computed or replaced. That is the engine acting on it.
The test for the whole class is the same. Put a harmless computed value where attacker-influenced text reaches the sink, and see whether the result shows the value you wrote or the answer to it. If it shows the answer, the sink is running your input, and the question is only how far that reach goes.
Template engines that run the draft
A template engine fills placeholders in a template with values. The safe way to use one is to write the template yourself and pass the model's text in as a value for a slot. The unsafe way is to take the model's whole reply and treat it as the template. When model text becomes the template, any expression inside it is evaluated, because evaluating expressions is what the engine is for. This is server-side template injection, or SSTI.
The Server-Side Template Injection lab (Intermediate) gives you Ledgerly, a billing tool at ledgerly.internal.example. When an order closes, the backend asks a model to compose the receipt body from the order record, then renders that body through the same template machinery the production mailer uses. One of the order fields is a customer memo, and Ledgerly reproduces the memo on the receipt exactly as the customer wrote it. A per-session marker lives only in the app's server config object, nowhere in the model's context or on the page. The class of move is to recognise that the customer-controlled memo rides into a body that is rendered as a template, so a template expression placed in the memo is evaluated rather than printed, and a default engine that exposes the config puts a server secret one expression away. Your job is to show the receipt surfacing that secret. The lab's own hints walk you there if the channel is hard to see.
The Template Injection to RCE lab (Advanced) takes the same sink further. Placard, at placard.internal.example, drafts a customer notice from an engineer's description and builds a live preview by compiling the draft through the engine. The model is told to mark any computed value as a double-brace field and to reproduce an exact expression the engineer designates. The class of move is to get a computed field placed in the draft whose expression is not a field name but one that walks from an ordinary template object out to imported modules, which is how a template expression reaches code execution on a default engine. The session marker here lives in the process environment, so the preview surfaces it only when an executed expression reads it. This lesson does not sketch the expression. The lab's staged hints do, if you stall.
Telling the model to avoid template syntax does not fix this. The sink still compiles whatever body the model returns, and the attacker's text is reproduced verbatim on its way there. A prompt reminder cannot un-compile a template. The repair is at the sink: stop rendering model output as a template at all.
Output run as Python or as a shell command
The next three labs ask the model for code or a command and then run it directly, with no sandbox and no separation of code from data. The model is doing exactly what it was told. The flaw is that its answer is executed.
The Eval Sink Runs Model Python lab (Advanced) gives you CalcPilot, a natural-language calculator that asks a model to return the Python that computes an answer and then runs that text in-process. Builtins are not disabled, so imports and file access are reachable. The result panel shows you both the Python the model wrote and the value it returned, which is the tell that the text was run. The class of move is to phrase a request whose computed value is not arithmetic but reads something the process can see, such as an environment variable or a server-side file. The marker lives in the environment and in a file, nowhere the model can read without executing.
The Computed-Column Expression Eval lab (Advanced) is the same sink in a spreadsheet tool. Sheetsmith turns a plain-English column description into a per-row Python expression and evaluates it against every row. Column specs are shared across the workspace, so a spec one user authors runs for another, which makes this a stored, second-order version: the hostile input is saved first and executed later. The class of move is to author a column spec that steers the model into an expression that reads a secret from the environment or a protected key file instead of computing a margin.
The Model Output Run as a Shell Command lab (Advanced) swaps the interpreter for an operating-system shell. ForgeRunner AI is a diagnostics console that asks a model to turn an infrastructure request into a shell command and runs it, streaming output back to the panel. There is no allowlist and no argument structuring. The model never touches the secret; it only writes text, and the application runs that text. The class of move is to work out what a command running as the app process could read that appears nowhere in the UI, then phrase a diagnostic whose satisfying command prints it.
Unsafe loaders that build live objects
Deserialization is turning a document back into objects in memory. A safe loader produces plain data: strings, numbers, lists and maps. An unsafe loader supports typed values, which means a tag in the document can name a constructor and the loader will call it as it reads. If a construction tag reaches an unsafe loader, code runs at load time, before anything looks at the result.
The Unsafe YAML Deserialization lab (Advanced) gives you Latch, a configuration-manifest composer at latch.internal.example. An engineer describes settings; the model drafts a YAML manifest, writing ordinary settings as plain values and any designated custom typed value as a tag; the backend loads the draft with the full loader to preview the effective configuration. The loader was chosen because it resolves those typed values into live values at load time rather than treating them as inert data. The class of move is to designate a typed value that is a construction tag naming a Python constructor, so that loading the manifest runs it server-side and reads the session marker out of the load process.
The YAML Gadget in a Generated Config lab (Advanced) is the same loader with one extra lesson. Provisio fills a catalog template with a couple of form values and loads the generated config to validate it. The template is authored by a contributor, not by the engineer running the generate, so the untrusted content arrives upstream of the person using the tool. The model reproduces every template line, including a custom typed value, verbatim. The class of move is the same construction tag, carried this time through the template rather than a direct request. Hardening the generator prompt to avoid typed values changes nothing, because the loader still constructs whatever tag reaches it.
The same bug in shipped software
This is not a lab-only concern. In April 2023, LangChain's LLMMathChain fed the model's arithmetic answer to a Python evaluation call and ran it verbatim, so a crafted prompt could make the model emit code that ran on the host. It was tracked as CVE-2023-29374, rated 9.8 critical, and the fix was to switch evaluation to a numeric-only evaluator.
In May 2024, JFrog disclosed the same class in Vanna.AI, a text-to-SQL library that also generated and ran visualisation code by default. A prompt injection in the user's question rewrote the generated code, which the library then executed, giving arbitrary Python on the host. See the Vanna.AI prompt injection to RCE, CVE-2024-5565, affecting versions up to and including 0.5.5 with no patched release, only guidance to run the function in a sandbox. Both incidents are the defining improper-output-handling mistake: model output treated as trusted, executable code.
The fix that holds
The control has one shape at every sink: do not hand model output to the interpreter as code. The specific move depends on the interpreter.
A system-prompt rule telling the model to return safe content. A word blocklist on the generated code or command. Both run after the model and leave the sink intact, so attacker-influenced output still reaches it, and a different expression or a different gadget root carries none of the blocked words and still executes.
Replace the executing sink with a non-executing one, so model output is a value the interpreter reads, never code it runs. The decision is made in your code, where a persuasive request cannot argue with it.
- Templates. Author the template yourself and bind the model's text into one slot as data, rendered with a sandboxed engine such as
SandboxedEnvironmentthat denies attribute access, or a logic-less engine with no expressions to evaluate. An expression in the bound value then prints as literal text. Keep the config and secrets out of the render context. - Python evaluation. Compute the result with a non-executing evaluator: an abstract-syntax-tree allowlist of the numeric and operator nodes the feature needs, or a math-only library such as
numexpr, with builtins disabled, no imports and no attribute access. A secret-reading expression is then a parse error, not an exploit. Removing builtins from an ordinary evaluation call is not enough, because reachable objects can climb back to dangerous operations. - Shell. Constrain the model to a structured intent chosen from a fixed allowlist, map that intent to a hardcoded argument array, and run it with
subprocess.runas a list with the shell disabled, in a process whose environment does not hold the secret. Model output becomes display and routing only, never the command. - Loaders. Parse with
yaml.safe_load, which refuses construction tags, and validate the plain result against a schema. Model output becomes settings, never a live object graph. - Everywhere. Keep secrets out of the process the sink can reach, so a sink that slips has nothing worth taking.
What you should be able to do now. At any point where model output leaves the model, decide whether the application runs it or only displays it, and if it runs it, name the interpreter and what it can reach. Recognise the four execution sinks in this lesson and the single control that answers each. In the labs, probe the channel with a harmless computed value first, read both the output and its effect, and submit what each lab asks for. In a design review, say why a prompt rule and a word blocklist leave the sink in place, and move the decision into code.
Decide whether a sink runs the model's output or only displays it. If it runs it, the interpreter is your own server process.
A template engine, an eval or exec call, a shell, and an unsafe loader run model output as code once it reaches them.
A word blocklist or a prompt rule leaves other routes to the same capability. The durable fix is to replace the sink with a non-executing one.
Keep secrets out of the process the sink can reach, so execution on the server surfaces nothing worth taking.
What is server-side template injection in an AI application?
Server-side template injection happens when an application takes a language model's reply and treats the whole reply as a template, instead of writing the template itself and passing the model's text in as a value. A template engine evaluates the expressions inside a template, so any expression the model was steered to include runs on the server. On a default engine this can reach application config or imported modules, turning a composed document into code execution.
Why is it dangerous to run code that a language model generates?
Model output is untrusted, because a model writes what it is steered to write and the request often carries attacker-influenced text. If the application runs that output through a Python evaluation call, an operating-system shell, a template engine, or an unsafe loader, the interpreter executes it with the reach of the server process. A request that was meant to ask for a calculation or a config can then read secrets or run commands on the host.
How do you prevent model output from being executed on the server?
Replace the executing sink with a non-executing one, so model output is data the interpreter reads rather than code it runs. Compute arithmetic with a restricted evaluator instead of a general one. Parse configuration with a safe loader that refuses construction tags. Map an infrastructure request to a fixed command allowlist run as an argument array with the shell disabled. Render model text as a bound value in a sandboxed template. Keep secrets out of the process the sink can reach.
Why doesn't a blocklist of dangerous words stop a code-execution sink?
A blocklist runs after the model and leaves the sink in place, so attacker-influenced output still reaches an interpreter that runs it. Blocklists test spelling, and the same capability is reachable by many expressions: a different object to start from, index-based attribute access, or a filter that carries none of the blocked words. Prompt rules fail for the same reason. The durable fix is to stop running model output at all, which is a change in your code, not in the wording.
© 2026 GenAI Security Lab. All rights reserved. You may read, quote, and link to this material with attribution. Copying, republishing, redistribution, resale, or use to train models or build competing products is prohibited without prior written permission.