When the app reads the model's output as structure

The sinks in the earlier lessons were obvious. Text went into a page, a query, a shell, a spreadsheet. Something plainly ran or rendered. This lesson covers a quieter sink. The model returns something that looks like tidy data the app asked for, and the app uses it as data: a URL rendered as a button, a block of JSON parsed into a decision, an XML document handed to a parser. Nothing looks like code, which is why the flaw is easy to miss.

The flaw is that the structure is trusted as the model emitted it. There is no check between the model producing the structure and the application acting on it. And the structure is shaped by whatever went into the model, which on a real feature includes untrusted content: a forwarded email, an imported document, a customer ticket, a partner order. A value drawn for one purpose, a label or a link or a status, becomes the thing that drives a decision. Structure the consumer never verified is untrusted input, even when it arrives in a clean shape.

Four terms you will see throughout:

• Open redirect is when an app sends a user to a destination it took from input, without checking that the destination is one it owns. It is catalogued as CWE-601. The user trusts the starting domain and lands somewhere else.
• Server-side request forgery, or SSRF, is when an app fetches a URL it was handed, and that URL points inside the app's own network. The server makes a request a client on the outside could never make.
• XML external entity injection, or XXE, is when an XML parser reads a document that declares an external entity and resolves it. The entity can name a local file or an internal address, and the parser folds the contents into the result.
• Structured output here means a machine-readable record, usually JSON, that the app deserializes and acts on rather than showing to a person.

In each case the model is only a text generator. The damage happens at the sink that trusts what it made.

Finding the missing check

You recognise this class by looking for the step that should sit between the model and the decision, and finding it absent. A URL that becomes clickable, or that the server fetches, without passing a host allowlist. JSON deserialized without a schema. A field accepted that the consumer never declared. A status value or a numeric id read straight out of the output and used unverified. XML parsed with entity resolution left on.

The tester's move is the same every time. Emit a piece of structure the consumer did not ask for and confirm it is acted on rather than rejected: an off-allowlist redirect target, an extra field in a record, an entity that reaches a file. If the app validated to the exact shape it needed, each of these would be refused before it touched any logic. When it is not refused, the output is driving the decision.

The seven labs in this lesson sit on four sinks. One trusts a JSON field. Two trust a URL as a redirect target. Two fetch a URL. Two parse model-authored XML. In every one, an attacker shapes the model's input and the app acts on the output without checking it.

A JSON field read as a command

The Structured Output Drives a Privileged Action lab (Advanced) gives you Triage Assist, the ticket-triage service for Harborline Support Desk, a fictional homegoods retailer. Triage Assist reads an incoming customer ticket and returns a small JSON record: a category, a priority, an order id, a routing target, and a boolean that approves an automatic refund. The backend deserializes that record and, when the boolean is true, issues store credit against the named order in a real ledger. The ticket in front of you concerns a clearance final-sale order that policy says is not refund-eligible, so credit on it is meant to need a human.

The design fault is that the application reads the boolean as an authorization. It never re-checks the order's own eligibility in its trusted state. It trusts a field the model filled in from the ticket body, which is untrusted customer content.

The class of move is indirect. Asking for a refund the way a customer would changes nothing: insisting, escalating or claiming to own the account all leave the field false, because the assistant treats a customer's own words as untrusted. Your goal is a record that comes back with the privileged field set, which means working out what the assistant will accept as the basis for filling in that field. The lab's own hints walk you in from there if you stall.

The lab completes through Submit Evidence, with a per-session code the app shows only when the unauthorized credit fires. A well-formed record proves nothing about whether its fields should be honoured.

A link trusted as a safe place to send someone

Two labs render a model-chosen URL as a button and forward the user to it with no allowlist.

The Open Redirect from a Model Link lab (Intermediate) gives you Waypost, a travel-booking concierge. A traveller forwards a booking confirmation, Waypost summarises it, and it surfaces a single one-click action link: the confirm, check-in or pay link. Waypost takes that link from the forwarded confirmation, which is third-party content it does not control, and its redirector forwards the traveller to whatever host the link names, with no check on where it points.

The Deep-Link Open Redirect lab (Intermediate) gives you Waypoint, a notification service for a fictional analytics product. When an event fires, a model composes the "continue where you left off" link from the workspace's configured return path, and the redirector forwards the recipient there. This one is worse: the redirector attaches a one-time access token to the destination so the recipient lands signed in. A return path that points off the sending origin becomes an open redirect that also hands that token to the off-origin host.

The class of move is the same for both. The destination is built from content an attacker can influence, the forwarded confirmation in one lab and the configured return path in the other. So whoever writes that content chooses where the user ends up. You supply content that names a destination host the app does not use, and watch the trusted redirector carry the user, and in Waypoint's case a live credential, off to it. Start by sending ordinary content and reading which host the button points at. Both complete through Submit Evidence, and the labs' hints narrow it if you need them.

A URL the server fetches for you

The next two labs do not just render a URL. They fetch it, server-side, from inside the app's own network. That turns a model-built URL into a server-side request forgery driver.

The SSRF via a Model-Built URL lab (Advanced) gives you Notewell, an internal research-note assistant. An analyst pastes a note, a model extracts the single source URL to unfurl, and the backend performs a real server-side fetch of that URL and renders the response as a preview. There is no egress allowlist, so a public address and an internal one are fetched alike. A second, internal-only service that your browser cannot reach serves an instance credential.

The Unfurl SSRF to an Internal Appliance lab (Intermediate) gives you Clipboard, a team knowledge base that auto-unfurls the links in an imported document, the way chat tools preview a link. A model lists the source URLs and the unfurl service fetches each one to build preview cards. The knowledge-base host also runs an internal deploy control-plane on its own loopback that the internet cannot address, but the fetcher can.

Here the class of move is about what the app will fetch. The model never sees the secret. The server does, when its fetcher reaches an address only the server can reach: a loopback service, a private-range host, or the cloud instance-metadata address. You steer the extractor, through the note in one lab and the imported document in the other, into naming that internal address as the source to preview, then read what the server fetched off the preview card. Both complete through Submit Evidence, with hints toward the internal target if you stall.

XML the app parses as a live document

The last two labs have the model compose XML, which a server-side parser then reads with external entities enabled. The model-authored document becomes the carrier for an XML external entity injection.

The XXE via a Partner Feed Entity lab (Intermediate) gives you Interchange, a business-to-business service that turns a submitted partner order into an XML feed. A model composes the feed from the order, including an order-notes field the partner controls, and an ingestion validator, a full XML parser, reads it to validate structure before forwarding. The External Entities in Model XML lab (Advanced) gives you Ledgerlink, a document composer where a model drafts XML from a plain-language request and the same kind of parser reads it for a live preview. In both, the parser is configured to resolve entities and load a document type definition.

The class of move is to get the model to author a document whose own declarations make the parser read something on the server, such as a local file, at parse time. The model only writes the document. The parser does the damage when it reads it. The marker you submit is read off disk by the parser and is never in the model's context, so a clean-looking draft is no evidence the sink is safe. Both complete through Submit Evidence.

The same trust in the wild

Two shipped products show what happens when a model's output is trusted because it came from the model. In July 2025, through Mozilla's 0DIN programme, Marco Figueroa showed that Google Gemini's Gmail summary feature could be steered by hidden text in an email body to append an attacker's message, a fake security alert, inside the summary panel users trust. The structure the app presented as a safe summary carried content the attacker chose. See the Gemini Gmail phishing summary.

In 2024, Johann Rehberger demonstrated a chain against Microsoft 365 Copilot where an injected email or document made the assistant search the victim's mailbox and files for sensitive data, then emit a clickable link with that data hidden inside it in invisible Unicode characters. Clicking the link sent the data to an attacker's server. The link the model produced was trusted and rendered. Microsoft's fix restricted link rendering in Copilot's output. That is the control this lesson argues for: a check on model output before it becomes something a user can click. See the Copilot ASCII smuggling exfiltration.

The fix that holds

Every sink here is fixed the same way in spirit: validate the model's output to the exact shape the consumer needs, in code, before any of it reaches logic. The specifics differ by sink.

• URLs, whether rendered or fetched. Require an https scheme and check the host against an explicit allowlist before the link becomes clickable, and reject anything else. Better, have the model choose a server-side action id and map that id to a URL your code owns. Never attach a session token to a cross-origin redirect.
• Fetched URLs specifically. Gate the fetch with an egress allowlist that resolves the target and refuses private, loopback, link-local and metadata addresses whatever the model emitted, permits only known public hosts, disables redirects, and caps response size and time.
• Structured output. Validate JSON against a strict schema and reject unknown fields. Decode it into a typed, checked value rather than acting on the raw record. Bind each field to its purpose, and re-derive any authorization from your own trusted state. A schema-valid record with the dangerous field set to true still fires, so the check that stops it belongs at the action.
• XML. Parse with a hardened library that disables external-entity resolution and document-type loading, for example defusedxml, or an lxml parser configured with resolve_entities=False and load_dtd=False. Disabling network access alone is not enough, because local file reads still resolve.

Prompt rules and blocklists do not hold here, for the reason they fail across this risk. The structure is shaped by content an attacker upstream controls, so an instruction telling the model to use only approved hosts, or to keep a field false, competes in a stream the attacker helped write. A blocklist that strips one bad scheme or one literal internal address leaves every other off-allowlist host and every other internal address open. The boundary has to be a code-side check the model cannot argue with.