Raw HTML and cross-site scripting
The most direct improper-output bug is handing the model's text to an HTML sink - innerHTML, dangerouslySetInnerHTML, a template that interpolates without escaping. The frontend wanted 'rich formatting,' but it just promoted model output to executable markup. Any tag the model emits - a script, an onerror handler, a tracking element - runs in the user's browser with the app's origin.
Model output lands in an HTML sink
The first lesson in this path followed model output into a markdown renderer, where an image or a link could make the browser fetch a URL on its own. This lesson follows the output one step further, into a sink that does more than fetch. An HTML sink is any place in your code that hands a string to the browser to be parsed as markup: an innerHTML assignment, React's dangerouslySetInnerHTML, Vue's v-html, or a template that drops a value in without escaping it.
The browser reads that string as HTML and builds whatever it describes. Plain words become text. A tag becomes a real element. An element that can run script runs it. The feature was meant to give replies some formatting, bold text and a working link. What it built is a path from model output to code that runs in the reader's browser with your application's identity.
Model output reaching that sink is untrusted input. The developer trusted it because the AI produced it, but the model's words are shaped by whatever the model read: a product review, a support ticket, a retrieved document, a file someone uploaded. Anyone who can influence that source can influence the output. Treat the model's text the way you treat a form field or a query parameter. This weakness is cross-site scripting, catalogued in the Common Weakness Enumeration as CWE-79, with the model as the source of the markup.
Some words you will need. Cross-site scripting, or XSS, means running attacker-controlled script in someone else's browser under your application's origin. An origin is the site's security identity, roughly its scheme and domain together. Script that runs at your origin can read the signed-in session, call your API as that user, and change what the page shows. A sink is the point where data crosses into a context that interprets it, and here it is the point where a string becomes live HTML.
Reflected, stored, and who the victim is
Two things decide how much damage this bug does. The first is whether the output is reflected or stored.
The output runs once, in the response that carried it, and is gone when that response is gone. The attacker has to get the victim to trigger that one response.
The output is saved and re-rendered on every load. It runs for every viewer, every time, and outlives the attacker's session. One write reaches many people.
The second is that the victim need not be the person who supplied the input. Suppose an internal console shows an AI summary of each incoming support ticket. Customers write the tickets. No customer ever sees the summary. The operator reading the console does. If the summary is assigned to an HTML sink, a ticket written to steer the model into emitting markup runs in the operator's browser. The attacker is upstream in the ticket, the victim is the staff member, and placing the console on a trusted internal network does not change that, because the dangerous input rode in through customer content.
How a tester finds this sink
You find this class by asking one question at every place model output appears on screen: is the text being shown as text, or parsed as markup? Read the frontend code and the render path. The tells are an assignment of model output to innerHTML, a dangerouslySetInnerHTML prop fed from model text, a v-html binding, an unescaped value interpolated into a template, or a markdown-to-HTML step configured to let raw HTML through.
To confirm it safely, a builder seeds the model's output with a harmless marker that would be visible only if it rendered as live markup, and checks which happens. If the marker shows up as text, the sink escapes. If it renders as an element, the sink parses. Then test the stored path: seed it once, reload, and confirm the marker is still inert after it has been saved and read back through whatever code draws the feed. A reflected path and a stored path can give different answers.
Check the stored path, not just the first render. A sink can escape model text on the screen where it was created and parse it on the feed that replays it later, because a different component renders each one. A test that only checks the view where the content was entered passes while the stored path still executes for every viewer.
Lab: stored XSS from an AI moderation digest
The Stored XSS from an AI Moderation Digest (Unescaped HTML Render) lab (Intermediate) puts you in the role of a security tester reviewing a trust-and-safety tool. Vigil is the AI moderation assistant inside Larkspur's moderator console. Moderators work a queue of abuse reports that users file, so every report is untrusted content written by whoever submitted it. For each report Vigil drafts a short case digest summarising the complaint, and the console renders that digest as HTML with innerHTML so links are clickable and basic formatting shows.
The design choice that breaks it has two parts. The console renders Vigil's digest as trusted HTML without escaping it, and Vigil is told to preserve the reporter's exact wording and reproduce any markup they put in their report, so moderators see the report as it was submitted. Nothing escapes or sanitises the digest between the model and the sink.
An ordinary report produces a safe prose digest. Your job is to show that a report can carry an executable element through the digest to the render sink, where it would run in a moderator's browser. The class of move is to get the model to reproduce markup you placed in the report into its digest, rather than describing it in words. The move uses a feature the model was told to provide: faithful reproduction of reporter content. A well-aligned model may refuse markup that plainly reads as script, so expect more than one attempt. The lab's own hints take you there in stages if you stall.
This lab completes through Submit Evidence. You watch the Case Digest panel: an element that renders live rather than as inert text has reached the sink. The grade reads what physically reached the render sink on the trusted side, not what the reply said.
Lab: build the guardrail at the render sink
The second lab moves you to the defending side. Build the Guardrail: Safe Output Rendering at the HTML Sink (Intermediate) is a blue-team build lab. The app, HelpScribe-Guard, wraps HelpScribe, the support-reply assistant in Northwind's support-agent console. An agent opens a customer ticket, HelpScribe drafts the reply, and the console renders it as HTML. Out of the box the console renders whatever the model produced, so a ticket can drive it to reproduce an executable badge from the customer's text, and a recovery request can make it emit a link whose scheme runs script instead of opening a page.
You cannot rewrite the renderer or re-align the model. What you author is the output-rendering policy the render layer enforces before any reply reaches the sink. A passing control has to do two independent things. It has to keep raw model markup from reaching the sink as live HTML, so a reproduced tag shows as inert text. And it has to restrict which URL schemes the renderer will make clickable, so a dangerous-scheme link is dropped before it becomes an anchor. Escaping the HTML does nothing to a live dangerous-scheme link, and an allowlist of schemes does nothing to a raw script, so one control without the other leaves a class open.
The lab also measures whether you broke the product. A legitimate reply still has to render, including its ordinary link to the self-service reset portal. A policy that strips every link stops the attack and fails the job. When you submit the policy the platform pushes it into the running app, replays an attack suite and a benign suite, and grades on what reached the render sink. Security and utility both have to hold. The lab shows you the policy schema, and its hints narrow it down if you stall.
The same bug in shipped products
Two shipped products had this bug in 2026.
In March 2026, Discourse's AI triage automation passed the raw text the triage model returned into the moderator Review Queue and rendered it with a trusted-HTML helper. An attacker could post forum content that, through prompt injection, steered the triage model into emitting markup. When a moderator or admin later opened that item in the queue, the script ran in their privileged session. That is stored XSS, and the victim was staff rather than the poster. It was assigned CVE-2026-27740 in the Common Vulnerabilities and Exposures list and fixed by HTML-escaping all model-generated content. See the Discourse AI triage stored XSS.
In July 2026, the open-source note app NoteGen rendered AI chat responses by running them through a markdown library set to allow raw HTML and injecting the result with dangerouslySetInnerHTML, with no sanitiser and the content security policy disabled. Content that reached the prompt could make the model emit active markup that executed inside the app's privileged desktop webview. Chained with a shell-execute permission, that script reached host commands, turning cross-site scripting into remote code execution. The XSS was CVE-2026-17496, and both bugs were fixed in a later release. See the NoteGen AI-output XSS to RCE.
The fix that holds
The control is contextual output encoding, applied at the sink and applied every time. For an HTML context that means one of two things.
- Render model text as text. Assign it to
textContent, bind it as text rather than HTML, or let an auto-escaping template escape it. A tag then shows as the characters a reader typed, and nothing runs. This is the default you want almost everywhere. - If the product has to render formatting, sanitise against an allowlist before the string reaches the page. A library such as
DOMPurifyparses the HTML and keeps only the tags and attributes you permit, dropping script elements, event-handler attributes, and dangerous URL schemes. Sanitise before rendering, never after.
Stored output is rendered again on every load, often through more than one code path, so encode or sanitise at each render path. Cleaning the text once at save time is a useful second layer and does not replace encoding at the sink. The day another screen reads the raw value into a sink, the save-time step is bypassed.
Two things that look like fixes do not hold. A prompt rule telling the model not to produce HTML runs before the sink and controls the wrong component; the markup still renders if it slips through, and attacker-influenced input often talks the model into producing it anyway. A blocklist that strips one tag, usually the script element, leaves every other way to run script untouched, and there are many. The control that holds sits in the code that consumes the output.
What to take into the lab
What you should be able to do now. Point to the HTML sink in a render path and say whether model text reaches it as text or as markup. Explain why reproducing untrusted markup makes AI output a stored-XSS source, and why the operator reading a console can be the victim of a report a customer filed. In the Vigil lab, drive an executable element through the digest to the render sink and submit the evidence. In the build lab, author a rendering policy that closes both the executable-markup class and the dangerous-scheme class while the legitimate reply and its link still render. In a review, find where the decision is made: if the answer is a line in the prompt, the sink is still open.
innerHTML, dangerouslySetInnerHTML, and unescaped templates parse model output as markup - any tag it emits runs at your origin (CWE-79).
Reflected XSS runs once; stored XSS re-renders on every load, hits every viewer, and outlives the attacker's session.
The victim need not be the attacker - an agent console rendering an untrusted retrieved ticket executes markup the ticket steered the model into emitting.
Encode for the sink at every render: assign to textContent, or allowlist-sanitize with DOMPurify. Sanitizing at save time is a second layer.
What is cross-site scripting in an AI application?
It is the same bug web developers know as XSS, with a new source. Model output reaches an HTML sink, a place that parses a string as markup, so any tag the model emits becomes a live element and can run script in the reader's browser under your application's origin. The model's text is shaped by whatever it read, such as a review or a ticket, so attacker-influenced input turns into executable markup. It is catalogued as CWE-79.
Why is rendering AI output with innerHTML dangerous?
An innerHTML assignment, a dangerouslySetInnerHTML prop, or an unescaped template tells the browser to parse the string as HTML, not to show it as text. A script element or an element with an event handler then runs code at your origin, where it can read the session, call your API as the signed-in user, and change the page. Because the model's output is influenced by untrusted content it processed, treating it as trusted HTML is the whole mistake.
What is the difference between reflected and stored XSS from AI output?
Reflected output runs once, in the response that carried it, and disappears when that response is gone, so the attacker has to make the victim trigger it. Stored output is saved and re-rendered on every load, so it runs for every viewer, every time, and outlives the attacker's session. A model summary written once and replayed through an HTML sink on each feed load is persistent stored XSS, which is the more damaging case.
How do you prevent XSS when displaying AI-generated content?
Encode model output for the context at the sink, every time. In an HTML context, render it as text with textContent or an auto-escaping template so tags show as characters, or, where you must allow formatting, sanitise against an allowlist with a library such as DOMPurify before it reaches the page. Apply this at every render path, because stored output renders again on each load. A prompt rule asking the model to avoid HTML and a blocklist that strips one tag do not hold.
© 2026 GenAI Security Lab. All rights reserved. You may read, quote, and link to this material with attribution. Copying, republishing, redistribution, resale, or use to train models or build competing products is prohibited without prior written permission.