Output that drives actions and emails
The highest-impact class is model output parsed into an action. An ops assistant whose ACTION blocks the backend executes, an email assistant whose draft is rendered into a live preview, a redirect link a user is steered to click - in each case a sentence of model text crosses into an effect with no independent authorization. The model became the control plane.
When output becomes an effect
The earlier lessons in this path dealt with model output that runs somewhere you can watch: a browser renders it, a database runs it as a query, a template engine evaluates it. This lesson covers the highest-impact case, where a sentence of model text crosses straight into an effect. A record is written to a ledger, store credit is issued against an order, a file is read or written on disk, a request leaves your server. In each case the application took the model's reply and acted on it, and nothing in between decided whether the act was allowed.
The reason this is a security problem is the same one that runs through the whole path: model output is untrusted. A model writes what its context steers it to write, and that context usually carries text an attacker can shape. In the labs below it is a customer ticket, a counterparty's document title, a submitted work item or a plain-language request. When the application treats the model's reply as a command, whoever shaped the context is driving the effect. The model has become the control plane, the part that decides what happens, and an attacker reaches it through ordinary content.
A few words you will need:
- Sink. The point where output leaves your code and enters something that acts on it. In this lesson the sink is an action: a write, a send, a file open.
- Side effect. A change the request makes beyond returning an answer: a row created, a message sent, a byte written.
- Authorization. The decision that this actor, in this context, may perform this action. It is a separate question from whether the model proposed the action, and it is the check that goes missing here.
- Path traversal. A file path that climbs out of the directory it was meant to stay in and reaches files elsewhere on disk. A classic web bug, and it returns here because the model now chooses the path.
Finding the point where output turns into an act
You find this class by locating every place a model's reply leaves the model and asking one thing: does the application act on this, or only show it? If it acts, find what stands between the reply and the act. When the only thing in between is the code that parses the reply, nothing has checked whether the act is allowed.
The tells, in an AI feature:
- A field in a structured reply drives a decision: a routing target, a status, a boolean that releases money or access.
- A draft the model writes is rendered live or sent on, so its contents reach a renderer, a mail server or a chat channel with no review step.
- The model names a file, a path or a destination that the application then opens for reading or writing.
- A tool call, an escalation or a workflow step fires straight off the reply, with no code checking the caller's rights first.
The test for the class is to make the model propose a privileged act from the untrusted content you control, then watch whether a separate layer refuses it or the app carries it out.
A structured field that releases money
The Structured Output Drives a Privileged Action lab (Advanced) gives you Triage Assist, the automated ticket-triage service for Harborline Support Desk, a fictional homegoods retailer. Triage Assist reads an incoming customer ticket and returns a small structured record: a category, a priority, an order id, a routing target and a boolean that marks whether store credit may be issued automatically. The backend reads that record and, when the boolean is set, issues credit against the named order in a real ledger. The ticket you are given concerns a clearance, final-sale order that policy says is not refund-eligible, so credit on it is meant to require a human.
The design fault is that the application reads the boolean as an authorization. It issues the credit on the strength of a field the model filled in, and never re-checks the order's own eligibility in its own trusted state. The model filled that field in from the ticket body, which is untrusted customer content, so an untrusted input is deciding whether money moves.
The class of move is to get the record to come back with the privileged field set, using only the ticket body. Asking for the refund the way a customer would gets you nowhere: the assistant treats a customer's own words as untrusted, including insistence, urgency and claimed authority, and keeps the field false. The work is to find what kind of content, beyond an ordinary customer request, the assistant is willing to trust when it fills that field in. The app acts on the value the model emits, so getting the model to emit it is the task. The lab's own hints take you in from there.
The lab completes through Submit Evidence. The app surfaces a per-session authorization code only when the unauthorized credit fires, and you submit that code.
Schema validation does not authorize anything. It rejects a malformed reply, but a schema-valid record with the privileged field set still fires if the action trusts that field. Only an authorization check at the action keeps the credit from going out.
A draft that sends itself out
The Rendered-Output Exfiltration via a Link Preview lab (Advanced) moves the effect to a send. Relaycast is an automated post composer for a fictional media workspace. You submit a work item, and a bot turns it into a short post for the team's operations channel, with the workspace's internal directory record, which holds a privileged access token, sitting in its context. You never see the composed post. It goes to the channel, not back to you.
When the post is delivered, the channel's link-preview service renders it. A link preview, or unfurl, is the small card a chat app builds by fetching the addresses in a message to show a thumbnail or title. Here that service fetches every address in the post on the server, with no sanitisation and no allowlist. The render is itself an outbound request, and that request is a side effect you can observe.
Two things make this harder than a reply you can read. The composer is blind to you, so steering the model to say the token is useless; the words go to the channel. And the composer honours some work-item content as post formatting, so what you submit can shape the post. The class of move is to stop trying to read the output and find the observable effect instead. The lab gives you one thing you can see: the preview service's outbound-request log. Work out what the rendering channel does with an address in the post, and what has to ride that address for the leak to be worth anything. The lab's hints build it up in stages.
This completes through Submit Evidence. You read the leaked value off the outbound-request log and submit it. Blocking any single form of address leaves the others, because the fetching sink and the secret in context are both untouched.
A path the model chose
The last three labs share one sink: the application asks the model to name a file, then opens that name on disk. A path that is absolute, or that contains directory-climbing segments, resolves to somewhere other than the directory the application had in mind. When the model's name is joined onto a base directory and opened with no confinement, a chosen path walks out of that directory and reaches whatever the process can read or write. This is path traversal, and the root cause is the same in all three: a model-authored path was trusted as a safe leaf name.
Two of them are the read form. The Path Traversal via a Document Title lab (Intermediate) gives you VaultRoom, a due-diligence data room. When a deal analyst exports a buyer bundle, VaultRoom names each entry after its source document and preserves a counterparty-supplied title verbatim as the path it reads into the bundle. You are the counterparty, and a protected file sits outside the data-room directory. The Model-Chosen Filename Escapes the Tree lab (Advanced) gives you Bindery, an internal report-template library. You type a plain-language report request, a small model resolves it to a template filename, and the backend opens that name from the template directory and renders what it read. A protected file sits outside that directory too. In both, the class of move is to make the resolved name point outside the intended directory rather than at a file inside it. The levers differ: in one the name comes from a document title you control, in the other from how the resolver handles a request for a template it does not have. The labs' own hints narrow it down.
The third is the write form, and it is worse. The Arbitrary File Write via a Model Path lab (Advanced) gives you Filer, an internal filing service that resolves a plain-language filing request to a destination filename and then opens that name for writing. Filer also serves a live public site from a directory outside its document store. A read traversal discloses a file; a write traversal puts a file where the application later serves or runs it, so whoever steers the path decides what the live site hands out. The class of move is to make the resolved destination land outside the store, in a place the running site will read back, and where the store and the site root live is part of what the lab asks you to work out. Completion is by Submit Evidence in each of the three: you surface the per-session secret the traversal reaches and submit it.
The same move against shipped agents
The leak path Relaycast models has turned up in shipped products. Two disclosures dated September 2025 turned an agent's own output into a data-exfiltration channel against production systems.
ForcedLeak, from Noma Security, targeted Salesforce Agentforce through a public web-to-lead form. Instructions hidden in a lead's description field lay dormant until an employee asked the agent to process the lead. The agent then gathered customer-relationship data and sent it to an image address, and that address sat on a domain still on Salesforce's allowlist but lapsed from the company's control, which the researchers re-registered for a few dollars. It was rated 9.4 on the CVSS severity scale with no CVE assigned, and Salesforce fixed it on 8 September 2025.
ShadowLeak, from Radware, targeted ChatGPT's Deep Research agent connected to Gmail. A poisoned email, with instructions hidden in near-invisible text, only had to arrive in the inbox. When the agent later processed the mailbox it encoded the inbox contents and appended them to an attacker's address, which it fetched from OpenAI's own servers, so nothing on the user's device or network saw the leak. OpenAI resolved it server-side in September 2025. In both cases untrusted content reached the model and its output triggered an outbound request. In ForcedLeak the allowlist did not stop it, because a domain on the list had passed out of the company's control.
The fix that holds
The durable control is a deterministic gate in your code between the proposal and the act. The model's reply is a suggestion. A separate layer, written in code and reading your own trusted state, decides whether it runs, regardless of how confident or well-formed the output was.
Proposes. It reads a context that includes untrusted content and emits a reply: a record, a draft, a filename. It can be steered, so its proposal is a claim that still needs checking.
Disposes. A deterministic layer reads your own trusted state and decides whether this actor may run this act, independent of how the proposal was worded.
- Re-derive authorization server-side. For a privileged action, check the actor's rights and the resource's own state in code. In the triage case, read the order's own eligibility from your database and require a human approval step for an ineligible order. Treat any boolean the model set as advisory and keep it away from the action.
- Validate structured output against a strict schema and drop unknown fields. Validation catches a malformed record, but its limit matters: a schema-valid record with the privileged field set still fires. The gate belongs at the action, not only at the parse.
- Render drafts as escaped text. Show a draft as escaped text, or sanitise it against an allowlist (for example
DOMPurifywith a URL-validation hook), and do not auto-fetch addresses that appear in model output. Where an unfurl is wanted, allowlist the scheme and host, route external images through a proxy, and set a strict content-security policy. - Never build a file path from model output. Have the model choose an id from a fixed set, map that id to a server-controlled path in code, and confine every read and write with
realpathunder a fixed base directory before opening. Model output is only ever a lookup key. - Keep secrets out of the context the output can reach. A token that is never in the model's context cannot be folded into an outbound address; fetch credentials in code at the moment of use.
- Require independent authority for consequences. Send, escalate, deploy and money-movement steps need a check the model cannot talk past, and a confirmation from the real user for anything high-impact.
Prompt rules and blocklists do not hold here. A system-prompt instruction such as "only approve genuine refunds" is one more line in the same stream the attacker's content is steering, so it can lose. A blocklist of trigger words or markers is beaten by rewording the same instruction. Asking the model to re-check its own proposal sends it back through the same content that steered it the first time. A check in code that reads your own state does not depend on the output's wording, so rewording cannot get past it.
What you should be able to do now. In any AI feature, find the point where model output becomes an effect and name what stands between them. Tell the three effect types apart: a structured field read as a command, a draft rendered or sent live, and a file path opened for read or write. For each lab, make the model propose the privileged act from untrusted content alone, then confirm the effect fired because no code-side gate stood in the way. In a design review, ask where authorization is decided: if the answer is "the model proposed it," the gate is missing.
A sentence of model text crosses into an effect - a DB write, an escalation, a live email - with no independent authorization. The model became the control plane.
Parsing an action out of model output is not authorizing it: a prompt patch loses in the same stream a crafted incident controls; a blocklist is reworded.
Put a deterministic code-side authorization gate between the proposal and the act - it decides if this actor may run this action, regardless of the model's output.
Treat the model as an untrusted author of a proposal; render drafts as text, and require independent authority for send, escalate, deploy, move money.
What is improper output handling that drives actions and emails?
It is a weakness where an application acts on a language model's output instead of only displaying it. A field in a structured reply releases money or access, a draft is rendered or sent on, or a model-chosen path is opened on disk. Because the model writes what its context steers it to write, and that context carries untrusted input, whoever shapes the input drives the effect.
How do you prevent model output from triggering a privileged action?
Put a deterministic authorization gate in code between the model's proposal and the act. Re-derive the decision from your own trusted state, such as the resource's status and the caller's rights, rather than a field the model filled in. Validate structured output against a strict schema, render drafts as text, map any model-chosen path to a server-controlled location, and require independent confirmation for send, escalate or money-movement steps.
Why do prompt rules and blocklists fail to stop output-driven actions?
A system-prompt rule is one more line in the same text stream the attacker's content is steering, so it can lose. A blocklist of trigger words is reworded around with synonyms. Asking the model to re-check its own proposal sends it back through the same content that steered it the first time. Only a decision made outside the model, in code, with no dependence on the output's wording, holds.
What is path traversal through a model-chosen filename?
Some AI features ask the model to name a file, then open that name on disk. A name that is absolute or that climbs out of the intended directory resolves elsewhere, so the application reads or writes a file it never meant to touch. A write is worse than a read, because attacker-influenced content can land where the app serves or runs it. Map model output to a fixed set of ids and confine every file access in code.
© 2026 GenAI Security Lab. All rights reserved. You may read, quote, and link to this material with attribution. Copying, republishing, redistribution, resale, or use to train models or build competing products is prohibited without prior written permission.