Ranking and provenance manipulation
Retrieval does not just decide what is in context - it decides what dominates. When ranking is naive similarity with no provenance or trust weighting, an attacker who controls a contributed document can engineer it to outrank the authoritative source for a target query, and a single-result pipeline then answers entirely from the attacker's chunk.
Retrieval decides which source wins
The earlier lessons on this path were about what gets into the model's context. This one is about which source wins once it is there. A retriever does two jobs. It selects a handful of chunks out of thousands, and it orders them. The assistant then answers from the top of that order, often from the top few and sometimes from the single best match.
The order comes from similarity. Each chunk is scored by how close its embedding sits to the embedding of the question, and the highest scores win. An embedding is the list of numbers that represents a piece of text by its meaning. Closeness in that space measures one thing: this text is about what the question is about.
The score carries no information about who wrote the text, whether anyone reviewed it, or whether it is true. A carefully maintained policy page and a note posted by any member of staff are scored by the same arithmetic. If both are in the same index, they compete on equal terms for the same questions.
This matters wherever people outside the trusted authoring team can add content: community answers, shared wikis, support tickets, uploaded files. A contributor decides what their document says, so they have a great deal of influence over which questions it sits close to. In a pool ranked on similarity alone, the right to contribute includes a say in what the assistant tells other people.
Similarity is not authority
Search engines learned this years ago. A page that matched the query well was not necessarily a page worth trusting, so ranking came to include signals about the source. Many RAG pipelines have not made that step. They treat the nearest chunk as the best answer, because during development every chunk in the index was written by the team.
Two different failures follow, and the labs in this lesson cover both.
An untrusted document sits closer to a target question than the authoritative one does. It enters the retrieved set, or takes first place, and the answer is grounded on it. Nothing about the document's label was false.
The pipeline does filter by trust, and it reads the trust label from a field the contributor filled in. The document claims a provenance it does not have and the filter lets it through.
Neither needs the model to be persuaded of anything. In all three labs the assistant behaves exactly as designed. It answers from what retrieval gives it. The weakness is entirely in how retrieval chose.
Marlin: an untrusted note inside a grounded answer
The Embedding-Space Collision Injects an Untrusted Note into a Grounded Answer lab (Advanced) gives you Marlin, the internal knowledge assistant for Tidewater Mutual at marlin.tidewater.example. Marlin answers staff questions from a knowledge base that holds two kinds of content: curated Tidewater references, and community notes that any member of staff can post. Both are embedded into one pool and ranked on similarity with no trust weighting.
Some questions put to Marlin are sensitive. The lab centres on one asked by the Treasury Operations desk, which should be answered only from the curated treasury reference. You hold an ordinary staff account that can post community notes.
The class of move is to get a note of yours into the set of sources Marlin retrieves for that question. When two texts land close enough together in embedding space to be retrieved for the same query, they are said to collide. You are not writing an instruction for the model to obey. You are competing for a place in the top results, and the competition is judged on meaning.
Marlin uses a real embedding model. A real model is not fooled by repeating a few words many times. It responds to what a passage is about, how specific it is, and how it is phrased. Expect to make several attempts and to read the retrieved sources after each one. Working out what moves a note up the ranking for a question you did not write is the exercise, and the lab's own hints help if you stall. Completion is by Submit Evidence, using a value that appears only when your note reaches the grounded answer.
AtlasRAG: taking first place from the verified runbook
The Embedding-Space Collision Displaces the Verified Runbook lab (Advanced) raises the stakes. AtlasRAG is an operations assistant for Cobalt Harbor at atlas.internal.cobalt-harbor.example. Any member of staff can contribute notes into the index it retrieves from. AtlasRAG grounds each answer on the one source it ranks first.
A single-result pipeline is the sharpest form of this weakness. With several sources in context, a trusted document can still be present and still shape the answer. With one, whoever ranks first is the whole answer. Being in the top few is no longer enough for the verified runbook. It has to win outright, every time, against anything a contributor writes.
The class of move is displacement: for a particular kind of question, a contributed note outranks the verified source. What AtlasRAG then does is for you to discover. The lab deliberately tells you little, and part of the assessment is noticing how the assistant's behaviour depends on what it grounded on. Compare its answers to the same operational question before and after you contribute, and read which source each answer names. Completion is by Submit Evidence.
Rank is not a safe trigger for anything. If an application takes a decision based on which document came first, or on whether a trusted document appeared at all, then anyone who can write to the index has a hand in that decision. Treat retrieval order as untrusted input, like the text of the chunks themselves.
Juno: a provenance label the contributor wrote
The Provenance Metadata Forgery in RAG Grounding (Real Vector Store) lab (Intermediate) shows a team that understood the problem above and tried to fix it. Juno is the documentation and support assistant for Northpeak, a cloud application platform, at help.northpeak.example. Its knowledge base is fed by two pipelines: an official docs sync, and a community intake that anyone can submit to. Juno is configured to ground answers only on documents whose provenance is official.
The filter is real. An honestly labelled community guide is retrieved and then correctly dropped. Provenance means the record of where a document came from, and Juno does check it. The flaw is where Juno reads it. The value comes from a metadata field that travels with the submission, so the person submitting the document chooses what it says. The intake pipeline knows which route the document really arrived by. Juno never asks it.
In this lab you do not contribute anything. A forged submission is already in the index, written to be the best match for one ordinary customer question. Your job is the reviewer's job: find the question it was written to intercept. The class of move is to probe with plain customer phrasing across the topics a support assistant handles, and to read each answer and its citation with a sceptical eye. An answer that states a procedure with unusual specifics, or sends the customer somewhere outside Northpeak, deserves a second look at its source. Completion is by Submit Evidence.
The consequence is worse than a wrong answer. Juno presents the forged content as official guidance with a citation, and the customer has every reason to follow it. The ConfusedPilot research describes the same outcome against an enterprise copilot, where content from an attacker's document shaped answers that users took as coming from trusted sources. See ConfusedPilot: poisoning what an enterprise copilot retrieves.
The shared root cause
In each target, something the contributor controls was used as evidence of trust. For Marlin and AtlasRAG it is the document's text, which sets its position in embedding space and so its rank. For Juno it is a metadata field. In all three the application had a better signal available, namely the route by which the content entered the system, and did not use it.
Published research shows how little content is needed. The PoisonedRAG work found that a handful of crafted passages added to a very large knowledge base could decide the answer to a targeted question. See PoisonedRAG: five bad documents hijack the answer. Volume does not protect a corpus. For any one question only the top few results count, so a contributor needs to win one small contest.
When you assess a RAG feature for this, map who can write to the index and by what route. Then ask what separates their content from trusted content at ranking time, and where that separating signal is stored. If the answer is "nothing", or "a field in the document", you have a finding before you send a single message.
The fix that holds
- Assign provenance at ingestion, in code. The pipeline that receives a document knows where it came from. Stamp the trust tier there, store it where contributors cannot write, and ignore any provenance value that arrives inside the submission.
- Keep trust tiers in separate pools. Put curated and contributed content in separate indexes or namespaces. Decide in code which pools a given kind of question may draw on. Sensitive questions should not search contributed content at all.
- Weight ranking by trust as well as similarity. Where pools must mix, a verified source should need far less similarity to win than an unverified one, and an unverified source should never displace a verified one for a protected topic.
- Never gate a decision on rank or presence. Whether a credential, an escalation or a privileged path is offered must depend on the caller's authorization, checked in code. It must not depend on which document came first. If the expected source is missing, fail closed.
- Do not answer from a single result on high-stakes topics. Require agreement from more than one trusted source, or pin the authoritative document for known sensitive questions.
- Show the true source to the reader. Citations should display the trust tier assigned at ingestion, so a community contribution cannot be read as official guidance.
- Review and monitor contributions. Hold new contributed content for review before it becomes retrievable for sensitive topics, and alert when a new document starts ranking first for questions an established source used to answer.
What you should be able to do now. Explain why a similarity score says nothing about who wrote a document, and say for each target what stood in for trust. Marlin ranks community notes and curated references in one pool. AtlasRAG answers from whichever source ranks first. Juno filters on a provenance label that the contributor supplied. Complete the Lab for each one, then choose the control that takes the trust decision away from anything a contributor can write.
Rank is not authority - under naive similarity the loudest-matching chunk wins, not the most trustworthy one.
Rank with provenance, author reputation, and signed origin as first-class signals a stuffed chunk cannot forge.
Draw from several trusted sources, never a lone unverified top hit - one crafted chunk should not own the answer.
'Prefer official docs' in the prompt has nothing to act on - the model only sees what the ranker already picked.
What is ranking manipulation in a RAG system?
Ranking manipulation is when someone who can add content to a retrieval corpus writes a document that ranks above the authoritative source for a target question, so the assistant grounds its answer on their text. It works because retrieval ranks by similarity of meaning, and a similarity score says nothing about who wrote a document or whether it was reviewed. OWASP covers it under LLM09, Vector and Embedding Weaknesses.
What is an embedding-space collision?
An embedding-space collision is when two different texts sit close enough together in embedding space to be retrieved for the same query. In a pool that mixes trusted and contributed content, a contributor's note that collides with a sensitive question can enter the sources an assistant answers from, or displace the verified document entirely. The defence is to keep trust tiers in separate pools and weight ranking by trust, since the collision itself cannot be prevented.
Why is provenance metadata on a document not enough to trust it?
A provenance label is only as reliable as whoever wrote it. If the label travels inside the submitted document, the contributor chooses its value and can claim an official source. Provenance has to be assigned by the ingestion pipeline from the route the content actually arrived by, and stored where contributors cannot change it.
Why is answering from the single top search result risky in RAG?
With one source in context, whoever ranks first supplies the whole answer, so the trusted document has to win outright against anything a contributor writes. With several sources, a trusted document can still be present and shape the reply. High-stakes questions should draw only on verified sources, and no security decision should depend on which document ranked first.
© 2026 GenAI Security Lab. All rights reserved. You may read, quote, and link to this material with attribution. Copying, republishing, redistribution, resale, or use to train models or build competing products is prohibited without prior written permission.