The corpus is a write surface

The earlier lessons in this path treat the knowledge base as something you read from. The weakness in each is that retrieval hands back a document it should have withheld. This lesson turns the direction around. The attacker adds a document, and retrieval delivers it to somebody else.

Some vocabulary first. An assistant built on RAG, short for retrieval-augmented generation, answers from a store of documents called the corpus. Each document is cut into chunks, which are passages short enough to fit several into one prompt. Each chunk is converted into an embedding, a list of numbers that places text with similar meaning close together. The embeddings sit in a vector index, a database that can find the stored chunks nearest to a new piece of text. When a question arrives it is embedded the same way, the index returns the nearest few chunks, called the top-k, and the application pastes them into the model's context beside the question.

Teams usually picture the corpus as their own documentation. In practice most corpora have contributors. Customers post in an indexed community forum. Employees add notes to a shared wiki. Users upload contracts, tickets and reports that later questions draw on. Each path lets someone outside the engineering team place text in the index. That makes the corpus a write surface: untrusted input, stored now and read by the model later.

Web testers already know this shape as stored injection: the payload is saved in one request and fires in another, usually in a different person's session. Four things are specific to RAG:

• The index ranks by meaning and nothing else. A chunk enters the top-k because it sits close to the question. Who wrote it, whether anyone reviewed it and whether it is true play no part in that ranking unless the application adds them.
• A retrieved chunk is read as guidance. The model receives the system prompt, the question and the retrieved chunks as one block of text. Most RAG prompts go further and tell the model to answer from the chunks. Text that arrives under that instruction carries weight, whoever wrote it.
• The asking user delivers it. The attacker acts once, on the ingestion path, and leaves. The planted chunk does nothing until an ordinary person asks an ordinary question on its subject. Their question is what loads it into context.
• One planted document reaches everyone. A crafted chat message affects one conversation. A planted chunk reaches every user whose question retrieves it, for as long as it stays in the index.

There are two things a planted chunk can carry, and they need different controls. It can carry a false fact, written to be the best match for a question people really ask. Or it can carry an instruction, written to be acted on by the assistant that reads it. The two labs for this lesson take one each.

Marlow: a false procedure with a citation

The RAG Poisoning to Grounded Misinformation (Real Vector Store) lab (Intermediate) gives you Marlow, the customer-support assistant for Fernpay, a fictional payouts platform for online creators. Marlow answers from one knowledge base that holds two kinds of content: Fernpay's own official help documents, and tips contributed by the community. Both are embedded into the same index. A retrieved-context panel shows you which snippets each question pulled in and labels every one with its source tier.

The poisoning has already happened when you arrive. Somewhere in the community tier is one tip that an attacker wrote to be the closest match for one question customers commonly ask. It describes a procedure that Fernpay does not have and includes a made-up code as part of it. You upload nothing in this lab. You are the tester who has to show that the planted tip reaches a customer.

The weakness is one missing decision. Marlow's search runs across the whole index and grounds the answer on whatever ranks highest. Source tier is recorded, and the panel even displays it, but nothing uses it to decide what the model may answer from. The index holds everything that was ever ingested, and the nearest chunk wins.

Marlow itself behaves well. It is told to answer only from the retrieved snippets, to quote their details exactly, and to cite its source. Those sensible rules are why the planted tip is so effective. The false procedure comes back in Marlow's own voice with a citation, and readers take a citation as proof that an answer was checked.

The class of move is finding the anchor. A poisoned document is written for a particular question, so your work is reconnaissance in plain customer language. Ask ordinary support questions across different topics and read the source tier of every snippet in the panel. You are looking for the topic where a community-tier snippet enters the grounded set. No adversarial phrasing is needed. The question that triggers the poison is one an innocent customer would type.

This lab completes through Submit Evidence. The Completion Check is concerned with Marlow's answer, because the answer is what a customer would act on. A community snippet showing in the panel tells you the boundary failed. The evidence is the planted value stated in the assistant's own reply, and it is compared with the value issued for your session. The lab's own hints help if the topic is hard to find.

The research version of this is PoisonedRAG, published in February 2024. The researchers showed that injecting a small number of crafted texts into a RAG knowledge database makes a chosen target question return an attacker-chosen answer. The attacker only has to win the retrieval for one question.

Deskbird: a contributed note the assistant obeys

The Uploaded Document Poisons the Retrieval Corpus lab (Advanced) puts you on the other side of the ingestion path. Deskbird is the internal IT self-service assistant at Fernwood Labs. It answers everyday IT questions from a knowledge base, and it lets employees contribute notes. A contributed note is embedded into the same namespace as the built-in notes. A namespace is a named partition of a vector index, and a search runs inside one partition. In a shared one, a contributed note competes for the top-k on equal terms with vetted content.

The second design decision is in how the context is assembled. Every retrieved note is concatenated into the prompt with nothing to mark where it came from and nothing to separate information from instruction. A sentence inside a note that is addressed to the assistant reads the same as a sentence written by the operator.

The lab gives that weakness something to reach. The indexing pipeline stamps each contributed note with an internal contribution-verification code. It is bookkeeping, and Deskbird is told never to put it in a reply. It appears nowhere in the note's text, the receipt, the retrieved-notes panel or the page. Working out where it does travel, and under what condition the assistant ever has it in front of it, is part of the assessment. Asking Deskbird for it directly goes nowhere, and you should confirm that before anything else.

The class of move is second-order indirect prompt injection. Second-order means the text is stored first and acted on later, in a separate request. Your contribution has to do two jobs at once, and they pull against each other:

• It has to rank. Retrieval here is real semantic search. A note enters the context only if its meaning is close to the question being asked. Keywords stuffed into an unrelated note do not move it closer. The note has to be about the topic.
• It has to be acted on. Once retrieved, the note must lead the assistant to do something its own instructions tell it to keep out of answers. How a document comes to be read as procedure, where the same request typed in chat is refused, is the thing to study.

The ranking requirement is what makes this a vector weakness and not only a prompt trick. What enters the model's context is decided by where text lands in embedding space, and anyone who can add a document can put one where a common question will find it. This lesson does not sketch what such a note says. The lab's own hints take you there in stages if you stall.

Watch the retrieved-notes panel as you work. It separates the two failure modes: a note that never ranked, and a note that ranked and was ignored. They need different corrections. When the code appears in Deskbird's answer, Submit Evidence. The Completion Check compares it with the code issued for your session.

After the attack stage the lab asks you to choose the control that stops it. One candidate scans contributions for suspicious words and rejects matches. A blocklist tests spelling, and an instruction is a matter of meaning. Reworded without any listed word, the note is still indexed, retrieved and grounded on.

The demonstrated version against a shipped product is ConfusedPilot, from August 2024. Researchers showed that any user able to add a document to a corpus an enterprise RAG copilot indexes could manipulate its answers: suppressing legitimate sources, returning attacker-controlled content, and misattributing that content to trusted documents. It was demonstrated against Microsoft 365 Copilot.

Relevance promoted to trust

Marlow and Deskbird fail at the same point. Text from a party the operator does not control was embedded beside vetted content, ranked on similarity alone, and handed to the model with the same standing as everything else. Relevance was treated as a grant of trust. A top-k slot says a chunk is about the question, and nothing more.

This is also why the attack is hard to see from where teams normally look. Chat logs show a normal user asking a normal question. The hostile text arrived earlier through a contribution form or an upload endpoint, and that is the request to examine. In Marlow's case there is no instruction to find at all.

The consequences grow with what the assistant can reach. An assistant that only answers can be made to misinform. One that holds data in its context can be made to disclose it. One that can call tools or send messages can be made to act. Morris II, a March 2024 research demonstration, took the last case to its end: against experimental RAG-backed email assistants, processing one infected email caused the assistant to reproduce the planted prompt in its own outgoing mail, so the next assistant retrieved it in turn.

You can spot the conditions during an assessment before sending a single probe:

• any path by which a user, customer, partner or scraper adds content to the index
• one namespace or one unfiltered search covering both vetted and contributed content
• a prompt that tells the model to answer from, quote or follow what it retrieves
• secrets, internal metadata or tool access present in the same context as retrieved chunks

The fix that holds

Poisoning is an OWASP LLM09, Vector and Embedding Weaknesses, problem with a prompt-injection consequence, so the controls sit on the pipeline.

• Review at ingestion and quarantine what is unreviewed. Hold contributed and uploaded content in a separate area the assistant does not search until someone has approved it. The ingestion path is the attack surface, so that is where the gate belongs.
• Record provenance on every chunk, server-side. Provenance is the record of where a chunk came from and who added it. Stamp it at ingestion from the authenticated session, never from a field the contributor fills in, and carry it on the chunk as metadata.
• Enforce trust tiers before the search. Decide in code which tiers an assistant may ground on, and apply that as a hard filter or a separate namespace, for example namespace = "official", so untrusted chunks are never candidates. Re-ranking by trust or capping the untrusted share of the top-k is weaker, because the planted chunk can still get in.
• Keep secrets and internal metadata out of the context. Strip bookkeeping fields at the grounding boundary and fetch credentials in code at the moment of use. An instruction that is obeyed then has nothing to take.
• Gate actions in code. If the assistant can send, change or approve anything, the authorisation check runs in the application against the real user's identity, with confirmation for high-impact steps. Retrieved text must never be enough to trigger an action.
• Treat retrieved text as data. Pass chunks in a clearly delimited block marked as reference material, and do not tell the model to carry out what documents say. This lowers the odds that a planted line is followed. It is still a request to the model, so it is a layer and cannot be the boundary.
• Verify high-stakes claims and monitor grounding. Codes, links and procedures stated to a customer should come from an authoritative tier. Alert on any answer grounded on a chunk outside it.