Many-Shot Jailbreaking: Drowning Safety in a Long Context
You don't out-argue the model's safety training — you bury it under a hundred examples of the model ignoring it.
| When | April 2024 |
|---|---|
| Target | Long-context aligned LLMs (Claude and peer models) |
| Researchers | Anthropic (Anil, Durmus, Panickssery, Sharma et al.) |
| Technique | Long-context in-context jailbreak |
| Status | Research demonstration; later NeurIPS 2024 |
- 1An attacker prepends up to hundreds of faux Q&A turns where the assistant answers harmful requests
- 2Long-context in-context learning makes the model continue the demonstrated pattern
- 3The final, real question gets a compliant, unsafe answer
What happened
In April 2024, Anthropic published many-shot jailbreaking: a technique that defeats a model's safety training by prepending a large number of fabricated dialogues in which the assistant happily answers harmful questions. The more of these 'shots' you include, the more reliably the model complies with a final harmful request — and it worked across Anthropic's own and other vendors' models.
How the attack worked
Large context windows let an attacker pack in hundreds of demonstration turns. In-context learning then treats those turns as the pattern to continue, so the model 'learns', within the single prompt, that its role here is to answer anything — overriding the refusals instilled by training.
Root cause
Safety alignment is a learned tendency, not a hard rule, and in-context examples can outweigh it. The very feature that makes long context useful also gives an attacker room to demonstrate the behaviour they want.
Why it matters
As context windows grow, so does this hole. A defence that only checks the final instruction misses an attack whose power lives in the hundreds of turns before it.
How to prevent it
- Don't rely on prompt-level refusals alone; add classification/filtering before the model.
- Inspect the whole context, not just the last message, for demonstration-style priming.
- Enforce policy outside the model on both request and response.
FAQ
What makes many-shot different from a normal jailbreak?
It doesn't rely on a clever single instruction. It floods the context with many examples of the assistant answering harmful questions, so in-context learning nudges the model to continue the pattern — and it gets stronger the more examples (shots) you add.
Why did bigger context windows make this worse?
Longer context windows are a feature, but they also let an attacker pack in far more demonstration turns. The attack's success scales with the number of shots, so the capability that helps users also widens this hole.
How do you defend?
Prompt-based refusals alone aren't enough. Anthropic explored classifying/filtering prompts before they reach the model and limiting how in-context examples steer behaviour; defence-in-depth outside the model is the durable answer.
Sources
- Many-shot Jailbreaking — Anthropic