Custom GPTs Gave Up Their Prompts and Files on Request
Whatever a GPT builder hid in the prompt or uploaded as a knowledge file, a visitor could ask for — and get.
| When | November 2023 |
|---|---|
| Target | Custom GPTs (OpenAI GPT Store ecosystem) |
| Researchers | Yu et al. (Northwestern) |
| Finding | ~97% system-prompt extraction (216 GPTs); 100% file leakage (all 24 with uploaded files) |
| Impact | Proprietary instructions, business logic, and private files exposed |
- 1A builder embeds proprietary instructions + files in a custom GPT's hidden context
- 2A visitor sends a simple extraction prompt
- 3The GPT prints its system prompt and dumps its uploaded files
What happened
In 2023, Northwestern researchers tested 216 custom GPTs (200 of them user-built) and showed simple prompt-injection queries could reliably force them to disclose their confidential configuration — a roughly 97% success rate for extracting the system prompt and 100% for exfiltrating uploaded knowledge files from the 24 GPTs that had them, without special skill. Follow-on studies found the leakage pervasive across the GPT Store.
How it worked
A custom GPT's “secret sauce” lives in its hidden instructions and attached files. Adversarial queries induce the model to print those instructions and dump the files — the hidden context is right there in the model's reach.
Root cause
Relying on the system prompt and knowledge files to keep IP and secrets confidential, when both are recoverable through the model itself.
What this shows
Wording can't reliably hide context from the model that holds it. If disclosure would harm you, the data shouldn't be in the prompt or knowledge base.
How to prevent it
- Keep secrets/IP out of prompts and knowledge files; assume extraction.
- Serve sensitive data via a gated backend, not the model's context.
- Don't treat “please don't reveal this” as a control.
Feel it yourselfThe replay lab primes an assistant into disclosing its hidden configuration — the hidden-context class.
FAQ
What could be extracted?
The customised system prompt (proprietary instructions and business logic) and the uploaded knowledge-base files — in the study, ~97% and 100% success respectively, with no special skill required.
Why is this LLM08?
Builders embed their differentiating IP, private documents, and sometimes API details in a GPT's hidden context. Extraction exposes exactly that — the confidential context the product relies on.
Can it be prevented?
Not fully by wording. Keep genuine secrets out of the prompt/knowledge base entirely; treat anything you put in a GPT's context as potentially extractable.