RAG Poisoning: How Untrusted Documents Can Influence an AI Assistant

A RAG assistant answers from your documents. That's its strength, and it's also its weak point: it can only be as right as the documents it retrieves. If a wrong document gets into the index, the assistant can repeat it confidently, with a citation that makes it look trustworthy.

That's what RAG poisoning means: documents in the index that push answers away from the truth. Sometimes it's deliberate. Much more often it's ordinary: an archived policy nobody removed, a draft that was never approved, an old FAQ, or a team wiki page written years ago. The cover image shows the idea. The pipeline is identical; only the source documents differ, and the response comes out broken.

This post measures it with a controlled test dataset, then tests three defences:

1. Source validation: deciding what is allowed into the index.
2. Access control: deciding who can write to sources, and who can retrieve what.
3. Evaluation: catching a changed answer whenever the index changes.

The Problem: Retrieval Doesn't Know What's True

Retrieval finds documents that are similar to the question. It has no idea which document is current, approved or correct. An old version of a policy is, by design, almost identical to the new one, so it scores almost the same.

We indexed all 11 documents and checked which document ranked first for each question. A disagreeing document came first for 5 of the 12 questions:

The first three won by 0.003 or less. The FAQ and the wiki page did better than the policy because they're written the way people ask ("How many vacation days do I get?"), while the policy is written formally. That's a general lesson: informal, unofficial content often matches questions better than the official document.

What the Assistant Said

With everything indexed, the two wrong answers were exactly what you'd fear:

• Vacation days: "[2] and [3] provide different numbers, but [1] clearly states 25 vacation days." Source [1] was the 2022 FAQ. The current policy says 26.
• Home office budget: the assistant read both the draft (€600) and the approved policy (€400), then concluded "[1] is more recent and provides a specific budget size of EUR 600." The draft was more recent; it just wasn't approved.

The three "no clear answer" responses listed both figures without choosing, such as "30 days and 60 days respectively, depending on the policy year". An employee would have to go and check, which defeats the point of the assistant.

Defence 1: Source Validation

The most effective fix is to keep bad documents out of the index in the first place. Every document carries metadata (source, status, owner, review date), and ingestion checks it:

TRUSTED_SOURCES = {"policy-portal"}          # sources with a small, named set of writers and a review step
 
 
def validate(doc):
    """Return None if the document may be indexed, otherwise the reason it is rejected."""
    if doc["source"] not in TRUSTED_SOURCES:
        return f"source '{doc['source']}' is not an approved source"
    if doc["status"] != "approved":
        return f"status is '{doc['status']}', not approved"
    if not doc.get("owner"):
        return "no owner"
    if doc.get("superseded_by"):
        return f"superseded by {doc['superseded_by']}"
    if (TODAY - date.fromisoformat(doc["reviewed"])).days > 365:
        return f"not reviewed since {doc['reviewed']}"
    return None

It rejected all five disagreeing documents, each with a reason you can log and review:

ARC-TRV-2024    status is 'archived', not approved
ARC-EXP-2025    status is 'archived', not approved
DRAFT-REM-2026  source 'hr-sharepoint' is not an approved source
FAQ-LEAVE-2022  source 'intranet-faq' is not an approved source
WIKI-IT-TIPS    source 'team-wiki' is not an approved source

Answers went back to 12 of 12, with no outdated figures in the text.

Why not just tell the model which sources to trust?

The tempting alternative is to index everything, show the model each source's status and date, and add a rule: use only approved sources; if they disagree, use the most recently reviewed approved one.

It helped: no answer was wrong. But 8 of the 12 answers still quoted the outdated figure alongside the right one, which leaves employees reading two numbers. And it depends on the model reasoning correctly about the labels. In one answer it wrote "[1] is more recently reviewed (2026-01-15) than [2] (2026-05-20)", which is backwards. It reached the right answer only because of the "approved" label.

Notice also that the rule had to say approved, not just newest. The HR draft was the most recent document in the set. Drafts usually are. "Prefer the newest document" on its own would have picked the draft.

Labels are a useful second layer. They're no substitute for keeping the document out of the index.

Defence 2: Access Control

Access control matters in two places.

Who can write to an indexed source. Indexing a source means trusting everyone who can write to it. Our four sources look very different from that angle:

This is also the main defence against deliberate poisoning. A planted document exploits exactly what our test documents did: it's similar to the question, recent and confident. You can't reliably spot that by reading the text. You can control whose text gets in. If 1,400 people can edit a source, anything in it should be treated as unverified.

Who can retrieve what. We gave the HR draft a reader list of HR only, and filtered retrieval by the user's groups:

def retrieve(index, question, user_groups=("all",), k=3):
    q = embed([question], "search_query: ")[0]
    scores = index["vecs"] @ q
    order = np.argsort(-scores)
    hits = []
    for i in order:
        d = index["docs"][i]
        if "all" not in d["readers"] and not set(d["readers"]) & set(user_groups):
            continue                                   # this reader isn't allowed to see this document
        hits.append((d, float(scores[i])))
        if len(hits) == k:
            break
    return hits

For an ordinary employee, the two remote-work questions came back clean: "employees may work from home up to 3 days per week" and "The home office equipment budget is EUR 400." For an HR user, who is allowed to see the draft, they didn't: the budget answer again concluded the draft's €600 was "more relevant".

That's the limit of access control: it decides who sees a document, not whether it's true. Filter by permission before retrieval, never after, so a document a user can't read can't shape their answer. But keep drafts out of an assistant that answers "what is the policy?", even for the people allowed to read them. Build an AI assistant that answers from your own documents shows why filtering in the prompt instead isn't enough: there, prompt-only permissions leaked in 3 of 3 tests.

Defence 3: Evaluation on Every Index Change

Validation checks a document's metadata, not its content. So what happens when an approved policy is edited badly?

We kept the 12 questions and their expected answers as a fixed test set. We then changed the index twice and re-ran the set after each change, flagging any answer that didn't match its expected value:

    flagged = [(r["q"], r["grade"], r["answer"][:90]) for r in rows if r["grade"] != "correct"]   # any answer off its expected value

• Change A: an approved policy re-published with an old figure pasted back in (the leave policy saying 25 days instead of 26). It passed source validation, since it's approved, owned and current. Evaluation flagged it: Q08 answered "Full-time employees receive 25 vacation days per year."
• Change B: a genuine policy update (the Germany per diem raised from €55 to €58). It also passed validation, and evaluation flagged Q01 too, because the answer no longer matched the expected €55.

Both flags are useful. A flag doesn't mean "bad document"; it means an answer changed, so a person should look. For change A, the person fixes the policy. For change B, they confirm the change and update the expected answer. Without the test set, both changes would have reached employees without anyone noticing.

A Note on Deliberate Poisoning

Everything above used honest mistakes. A deliberately planted document attacks the same weak points: it's written to match likely questions, looks current, and sits in a source the assistant indexes. Some planted documents also contain text aimed at the model rather than the reader; prompt injection and agent hijacking covers that side.

The defences don't change:

• Index only sources with a small, known set of writers and a review step.
• Validate metadata from the system of record.
• Filter by permission before retrieval.
• Re-run the evaluation set on every index change.
• Show citations so users can check the source themselves.

If your assistant can also act, not just answer, a poisoned document can do more than mislead. Treat retrieved text as untrusted input, as the threat model for an AI agent does.

Checklist

• Every indexed source is listed with who can write to it; broad-write sources aren't used for authoritative answers.
• Ingestion checks source, status, owner, superseded flag and review date, using metadata from the system of record.
• Archived and superseded versions are removed from the index, not just labelled.
• Drafts stay out of assistants that answer "what is the policy?"
• Retrieval is filtered by the user's permissions before anything reaches the model.
• Answers cite their sources, and source status and date are visible to the user.
• A fixed question set with expected answers runs on every index change, and every flag gets a human decision.
• Rejections and flags are logged, so you can see what was kept out and why.

The Short Version

An AI assistant doesn't know which of your documents are true. It believes whatever it retrieves. Decide carefully what goes into the index and who can put it there, and test the answers every time the documents change.