ConfusedPilot: Poisoning What an Enterprise Copilot Retrieves
One document dropped into the shared corpus, and the trusted copilot starts working for the attacker.
| When | August 2024 |
|---|---|
| Target | RAG-based enterprise copilots (demoed vs M365 Copilot) |
| Researchers | UT Austin (Spark Research Lab) + Symmetry Systems |
| Technique | RAG data poisoning / confused deputy |
| Impact | Suppressed/altered/misattributed answers; some effects persist after deletion |
- 1An attacker adds a crafted document to a corpus the enterprise copilot indexes
- 2The copilot retrieves it and follows its embedded instructions
- 3It suppresses real sources, returns attacker content, and misattributes it as trusted
What happened
In 2024, researchers demonstrated “ConfusedPilot”: any user who can add a document to a corpus an enterprise RAG copilot indexes can plant crafted strings that manipulate its answers — suppressing legitimate sources, returning attacker-controlled content, and misattributing that content to trusted documents. Some effects persist after the malicious document is deleted, due to caching. Demonstrated against Microsoft 365 Copilot and presented at DEF CON AI Village 2024.
How it works
It's a confused-deputy attack on the retrieval layer: the copilot has legitimate access and user trust, and a single poisoned document in its corpus redirects that trust to the attacker's content.
Root cause
Weak control over who can add to the indexed corpus, plus retrieved content treated as trusted — an integrity attack on RAG.
Why it matters
Enterprise copilots index shared drives and wikis many people can write to — so the “add one document” precondition is often trivially met.
How to prevent it
- Govern corpus contributions; review and attribute sources.
- Enforce per-document access control and provenance.
- Don't let retrieved content act as instructions; flush caches on deletion.
Feel it yourselfThe replay lab drops a poisoned document into a corpus a copilot trusts — the RAG confused-deputy class.
FAQ
What is the confused deputy here?
The copilot has legitimate access to the corpus and is trusted by users — an attacker who can drop one crafted document into that corpus abuses the copilot's authority to shape everyone's answers.
What can it do?
Suppress legitimate sources, return attacker-controlled content, and falsely attribute that content to trusted documents — with some effects persisting after the malicious document is deleted, due to caching.
How do you defend?
Control and review who can add to indexed corpora, enforce per-document access control and provenance, and don't let retrieved content act as instructions.