Deleting a document is more than one operation

A retrieval-augmented generation system, RAG for short, answers questions from documents it looks up at the moment you ask. To make that lookup possible, each document is copied several times on the way in. Understanding stale content starts with counting those copies.

The original sits in a source store: a wiki, a document manager, a database row. At ingestion the text is cut into chunks, which are passages short enough to search and to fit in a prompt. Each chunk is turned into an embedding, a list of numbers that represents its meaning. The embeddings and the chunk text are written to a vector index, a store built to return the chunks whose embeddings are closest to the embedding of a question. The closest few are called the top-k, and they are what the model reads. Many products then add a cache, a store of work already done, so that a repeated request can be answered without doing the work again.

So one document ends up in at least three places: the source store, the vector index, and any cache holding results built from it. A delete button in the application usually acts on the first. Whether it reaches the other two depends on code somebody had to write, and that code is frequently missing. OWASP files the result under LLM09, Vector and Embedding Weaknesses.

The index does not notice the gap. A retriever, the component that runs the search, compares numbers. It has no idea that the document a chunk came from has been retracted, expired or reclassified unless that fact is stored next to the chunk and the query is told to use it. An embedding made last year matches a question today exactly as well as it did on the day it was written.

Three terms are worth having before you test anything:

• Soft delete. The record stays where it is and a flag such as deleted: true is set on it. Every reader of the data is expected to check the flag. Any reader that does not check it still sees the record.
• Orphaned chunk. A chunk whose source document has been removed while the chunk itself is still in the index. People also call these ghost chunks. Nothing in the application points to them any more, and they still rank.
• Stale cache entry. A stored result that was correct when it was computed and is still being served after the data behind it changed.

For a web tester the closest familiar bug is the object that disappears from the listing page and still loads by direct id. The difference in RAG is that you do not need an id. You reach the leftover copy by meaning, with a question.

The two labs in this lesson put the gap in two different places. In the first, the delete never reaches the index. In the second, it reaches the index and stops there.

Sable Knowledge Desk: a deleted flag the retriever never reads

The Soft-Deleted Document Still Retrieved lab (Intermediate) gives you Sable Knowledge Desk, an internal knowledge-base assistant at kb.sable.internal. An employee asks a question, the app embeds it, pulls the nearest articles from a real vector store, hands them to a model, and shows the answer together with a panel of the sources it retrieved.

The scenario is a clean-up that looked finished. A legacy billing platform was decommissioned last quarter. The knowledge-base memo about it recorded the old connection string, so the memo was retracted. It no longer appears in the visible document list, and the operators believe the content is gone. You are asked to find out whether the retraction reached the retrieval index or only the list.

Start by seeing why the obvious check tells you nothing. The document list is rendered by application code that reads the source store and skips anything flagged as deleted. The retriever is a different piece of code reading a different store. If the retriever does not apply the same condition, the two will disagree, and only one of them decides what the model reads. A list that looks clean is evidence about the list.

The class of move follows from that. You cannot ask the index for its deleted items, and the model has no such list to give you. What you can do is ask a question whose meaning sits close to what the retired document was about, and then read the retrieved sources, giving them more attention than the answer text. Routine questions about live topics will only ever bring back live articles, so they prove nothing in either direction. The question has to be aimed at the subject that was supposed to disappear. The lab briefing tells you what that subject is. Working out a question that lands near it is the exercise, and the lab's own hints narrow it down if you get stuck.

Notice what kind of search this is. The retrieval is dense, which means it ranks on meaning. Words such as "deleted" or "retracted" in your question do not pull deleted things towards you. They pull articles that are about deleting and retracting. Testers who come from keyword search lose time here.

If the weakness is present, a source that is absent from the document list turns up in the retrieved-sources panel, marked as retracted. That mismatch is the finding. Completion is by Submit Evidence: you read the decommissioned connection string out of the retracted source and submit it exactly as shown. It exists in no live article, so having it shows that retrieval served the retracted one.

Beacon: a takedown that stops at the index

Now assume the index is handled properly. A document is withdrawn, its chunks stop being returned, and a new search comes back empty. Many teams would call that a completed deletion. The second lab shows the copy they forgot.

The Cache Takedown Invalidation Gap (Real Vector Store) lab (Intermediate) puts you in front of Beacon, the internal knowledge assistant for Meridian at beacon.meridian.internal. Beacon embeds your question, searches the staff knowledge base, grounds a model on the passage it finds and returns an answer. To stay fast it caches its answers. It also has a records and index panel where a document can be withdrawn, the way a retraction or an erasure request would be actioned. One document in the knowledge base carries an advisory reference. Your job is to determine whether that reference is still reachable after its document has been withdrawn.

A response cache stores finished answers. Each entry is filed under a key, a value derived from the incoming request, most often a tidied-up form of the question text. When a request arrives, the application computes the key and looks it up. On a hit it returns the stored answer and skips retrieval and the model entirely. On a miss it runs the full pipeline and stores the result. Some products cache retrieval results instead of answers, and some use a semantic cache that matches questions by embedding similarity. The security property is the same in each design: a hit replays text that was produced from whatever the index held at the time.

That is the weakness to look for. A takedown that removes a document from the index changes what a miss will produce. It changes nothing about entries already stored unless the takedown also evicts them. The stored answer is a derived copy of the document, and it will keep being served until it expires or something deletes it.

The class of move is a before-and-after test, and it differs from the Sable lab in an important way. There you could show up at any time and probe. Here the result depends on history: on what the system computed before the withdrawal. So you have to design the test around the takedown itself. Establish what Beacon serves while the document is live. Carry out the withdrawal. Then work out which of your requests are recomputed against the current index and which are replayed from stored work, and compare what each returns. The tell is disagreement between the two paths about a document that is supposed to be gone. Which requests fall on which side is for you to find. The lab's own hints will point you there.

No unusual phrasing or injected instruction is involved. Every document in this knowledge base is one you are allowed to read, so this is a lifecycle failure with no access-control element. The defect is that content whose removal was the whole point kept being served.

Completion is by Submit Evidence again. You submit the full advisory reference, exactly as written, recovered after the withdrawal.

Published research shows the same shape. The ConfusedPilot work on enterprise RAG copilots, demonstrated against Microsoft 365 Copilot in 2024, found that some effects of a malicious document persisted after that document was deleted, due to caching. See ConfusedPilot: poisoning what an enterprise copilot retrieves.

The shared root cause

Both targets fail because the delete has a shorter reach than the data. Sable's retraction reached the application's view and stopped before the index. Beacon's takedown reached the index and stopped before the cache. In each case the team verified the delete at the store they had changed, saw the right result, and never checked the stores they had not.

The pattern is general. Every transformation of a document creates a new copy with its own lifetime: chunks, embeddings, summaries, cached retrievals, cached answers, replicas, exported snapshots. Deleting the original deletes none of them. The same applies to copies held by someone else. In July 2025, ChatGPT conversations that users had shared with a "discoverable" option were found in Google results. OpenAI removed the feature, and the content also had to be de-indexed, because a search engine's index is a separate copy. That case involves no vector store, and the lesson carries across. See "Discoverable" ChatGPT chats got indexed by Google.

Reclassification fails the same way as deletion. A chunk indexed while a document was public keeps its old reach after the document is marked internal, unless the access label stored with the chunk is updated too. So does a revoked permission that the retriever never rechecks.

When you assess a RAG system for this, treat "deleted" as a claim to verify at each layer:

• List every place text derived from a document can live. Ask the team, and read the delete path if you have the code. A handler that removes the database row and returns success without a call into the vector store is the classic shape.
• After a delete, probe by meaning. Ask about the removed document's subject and inspect the retrieved sources or citations, where the product shows them.
• Test across time. Exercise the content before the delete, perform the delete, then compare requests the system must recompute with requests it may replay.
• Do the same for a reclassification and a permission change. They are deletes for one audience.
• Report which store still held the content. "Still retrievable from the index" and "still served from the answer cache" lead to different fixes.

The fix that holds

The weak fixes in both labs change what is displayed or how long the problem lasts. The durable ones make the delete reach every copy, and make the retriever refuse what should not be served.

• Purge the index in the same operation as the source delete. When a document is removed, remove its chunks and embeddings as part of that transaction, so the document's state and the index's state cannot drift apart. Key chunks by document id so the purge can find them all.
• If you soft delete, enforce the flag inside retrieval. Apply it as a metadata filter, a condition on stored chunk attributes that runs before ranking, on every query. Something like deleted = false belongs in the search call itself. A flag that only the user interface reads protects the user interface.
• Invalidate caches on takedown. Record which documents each cached entry was built from and evict those entries when a document is withdrawn. The alternative is to include an index version in the cache key and bump it on every takedown, so every older entry misses.
• Put caches in scope for retention. A response cache holds real content. It belongs under the same retention, erasure and takedown rules as the primary store.
• Update chunk metadata on reclassification, and recheck access at query time. Re-embedding unchanged text fixes nothing here, because the vector did not go out of date. The label did.
• Reconcile on a schedule. Compare the index against the source of truth, and alert on chunks that map to no live, authorised document.
• Test the delete end to end. After a takedown, assert that the content is absent from every store, including requests that were answered before the takedown. A fresh search returning nothing only covers the index.