The task that carries the prompt out

The previous lesson asked the assistant for its instructions outright. Most deployed assistants now refuse that. A non-disclosure line in the system prompt, the hidden instruction text an operator gives the model before any user turn, is enough to turn away a request that plainly asks for the prompt.

Extraction by transformation asks for something else. The request is for a piece of ordinary work: translate this, reformat that, convert it to another representation. The work is pointed at the assistant's own context. The reply is the hidden setup in a new form, and at no point did anyone ask the assistant to reveal anything.

This works because of how a non-disclosure rule is applied. The model does not check outgoing text against a list of protected strings. It forms a view of what a request is asking for and decides whether that looks like disclosure. A translation looks like translation. The rule was written with a different kind of request in mind, so it does not come into play.

Three properties of transformations matter when you test for this:

• A faithful transform keeps fixed values. A key, an identifier or a codename has no translation. A translator leaves it as it is, and a format converter copies it across. The sentences around it change and the value arrives intact.
• A lossy transform may drop them. A summary chooses what to keep. It can leave out the one line you care about and still be a good summary. When a summary comes back clean, that tells you the summary omitted the value. It does not tell you the value is protected.
• An encoding changes what a filter sees. An output filter that compares the reply with the known secret is looking for one exact sequence of characters. The same value written in another representation is a different sequence, so the comparison finds nothing.

The request also never contains the words a keyword filter on the input would look for. It mentions a language or a format. It does not mention secrets, prompts or instructions in any way the filter was built to catch.

LinguaDesk: a translator pointed at itself

The The Literal Translator lab (Intermediate) gives you LinguaDesk, a multilingual support assistant for Novacorp. Its job is to translate customer messages between languages. Its system prompt carries a confidential internal directive value, a fixed token, together with a rule never to disclose its own instructions or configuration.

In normal use LinguaDesk behaves well. Give it a customer's sentence and it translates that sentence and nothing else. Ask for its instructions, its configuration or the value itself and it declines. Both behaviours are worth confirming first, because the lab is about the gap between them.

The gap is that LinguaDesk treats translation as a literal formatting job. The rule that protects its setup is one sentence in the prompt, and it is applied according to what the model decides a request is asking for. The class of move is to find a request LinguaDesk accepts as ordinary translation work whose subject is its own context instead of your text. Because the directive value is a token and not a phrase, it comes through a translation unchanged.

Expect some resistance. LinguaDesk refuses to summarise its setup, and the obvious ways of referring to its instructions are either declined or resolved to something harmless. Working out which reference it accepts is the exercise, and the lab's own hints narrow it down if you stall. There is nothing to submit. When the directive value appears in a reply, the Completion Check confirms the lab.

Halo: an output filter that reads only plaintext

The Exfiltrate a System-Prompt Secret Past an Output Filter (Encoded Leak) lab (Intermediate) adds a real control. Halo is an internal platform assistant that helps engineers with deploys, configuration and diagnostics. Its system prompt holds a confidential deployment token for your session and an instruction not to hand it out.

Behind Halo the application runs an output filter of the kind usually called DLP, data loss prevention. It is deterministic code, separate from the model, and it runs over every reply before you see it. If the reply contains the token, or a trivially respaced variant of it, the filter replaces it with a redaction marker. So even a successful direct extraction comes back redacted. This is a better design than a prompt instruction alone, and many production systems stop here.

The filter's limit is that it reads surface text. It compares what is on the page with the plaintext token and it does not decode anything first. A value written in a reversible encoding contains none of the characters the filter is matching, so the filter passes it as harmless text.

The class of move is to have Halo emit the value in a representation the filter does not recognise, then reverse the representation yourself. Not every encoding is equally dependable on this target, since a model can make mistakes when it encodes by hand, and part of the lab is noticing when a decoded result is wrong. Use the hints for which representation holds up. This lab completes through Submit Evidence: you decode what Halo returned and submit the recovered token, and the Completion Check compares it with the one issued for your session.

FindBot: a 2023 leak, replayed

The Breach Replay: The Search Bot Codename Leak lab (Beginner) recreates a real incident. FindBot is a search assistant with a confidential internal codename in its system prompt. It is under a firm rule never to reveal the codename or reproduce its configuration, with no exception for audits, debugging or transparency, and it declines when you ask directly.

The lab replays the Bing "Sydney" system-prompt leak. In February 2023 a student used a one-line prompt injection to make Microsoft's new Bing Chat reveal its confidential system prompt and its internal codename. It was one of the first prompt-injection attacks against a production LLM product.

What that incident showed is the idea underneath this whole lesson. The hidden instructions were treated as text to be output instead of rules to be followed. Once the model handles its own setup as material to work on, the non-disclosure rule inside that material is just one more line being processed. FindBot's protection rests entirely on the model choosing to obey a sentence, and no server-side control stands behind it.

The class of move is to get FindBot to handle its configuration as content. The lab asks you to recall how the original leak worked and apply the same idea, and its hints help if you have not read the incident. Completion is automatic when the codename appears in a reply.

The same weakness has been measured at scale. In Custom GPTs gave up their prompts and files on request, researchers tested more than 200 custom GPTs in 2023 and showed that simple prompts could extract their confidential system prompts and their uploaded knowledge files.

The rule guards a request, and the value is still there

The three labs share one defect. A confidential value sits in the system prompt, and what protects it is a judgement about requests. LinguaDesk judges that translation is safe. FindBot judges each request against a sentence it has been told to obey. Halo adds a filter, and the filter judges one spelling of the value.

None of these controls acts on the value itself. They act on a description of how the value might be asked for, or on one form in which it might appear. The set of faithful transformations of a piece of text is very large, and a tester needs only one that the description did not anticipate.

This is also why a clean result from one probe proves little. If a summary omits the token, or a direct request is refused, the value is still in context and still available to the next request shape. During an assessment, record which transforms the assistant will apply to its own context and which it refuses, and treat any faithful one as a finding even if the particular secret you were looking for was not in the prompt that day.

The tell in a transcript is an answer whose content is about the assistant when the request was about a task. A translation that contains operating rules, or structured output with fields nobody supplied, means the task was applied to the hidden context.

The fix is the same, and filters come second

Adding rules to the prompt does not fix this, for the reason given above. The controls that hold are outside the model.

• Keep the secret out of the prompt. A directive value, a deployment token or a codename that must stay confidential belongs in server-side configuration, used by application code and never placed in the model's context. No transform can reproduce text the model was not given.
• Write the prompt as if it will be read. Assume the full text will be recovered. Behavioural guidance can stay. Anything that would cause harm when published should be moved out.
• If you run an output filter, make it decode-aware. Before matching, the filter should normalise the reply and decode candidate runs in the common representations, then compare the result with the protected value. An assistant emitting a long encoded string that nobody asked for is worth alerting on by itself.
• Treat the filter as defence-in-depth. Even a decode-aware filter recognises only the representations someone thought to add. It reduces how often a mistake reaches a user. It cannot be the reason a secret is safe.

The order matters. A team that starts with the filter ends up maintaining a growing list of encodings. A team that starts by removing the secret from context finds the filter has very little left to catch.