Prompt Engineering for RAG: How Should You Prompt an AI That Has Retrieved Documents?

RAG From Scratch builds the retrieval half of this system by hand, chunking, embeddings, a vector store, a similarity threshold. This post picks up exactly where that one hands off: documents have already been retrieved, correctly, and the question is what to actually do with them in the prompt, because a perfect retrieval and a bad prompt still produces a wrong answer.

The Full Path

Context Boundaries

Retrieved documents are exactly the External Content layer, information to reason from, never instructions to follow, and the same delimiter discipline covered for any untrusted content applies here directly: wrap retrieved chunks in a clear, consistent boundary, and label each one with its source.

Context:
[Source: deployment-runbook.md]
{chunk text}
 
[Source: incident-2026-03.md]
{chunk text}
 
Question: {question}

A model working from an unlabeled wall of retrieved text has no way to tell where one document ends and another begins, which is exactly what makes citation and conflict detection unreliable later in the same prompt.

Source Grounding: "Answer Only From Supplied Context"

Missing Information

Conflicting Documents

A question a retrieval-based system will eventually hit that a single-document prompt never has to: two retrieved chunks that disagree, an old policy document and its replacement, two versions of a runbook, a draft and a final. Silently picking one is worse than answering "I don't know," because it looks confident while being arbitrary.

Citations

Citing "the documents" isn't a citation. Citing which specific source, and ideally which specific claim came from which source, is what makes an answer checkable rather than just plausible.

Output format:
Answer the question directly, then list each claim with its
source in brackets, like this: "Rollback requires a manual
approval step [deployment-runbook.md]."

The same discipline covered for any output format applies directly here: naming the exact citation format, inline brackets, a footnote list, a structured field, gets a consistent, parseable result. Asking generically for "sources" gets a different format every time.

Retrieval Quality vs. Prompt Quality

Before changing anything, check which document actually made it into the context for that specific query. That one check almost always tells you immediately which half of this table you're actually debugging.

A Complete Worked Template

You are answering questions using only the context provided below.
Answer only from this context, if it doesn't contain enough
information to answer, say so explicitly rather than guessing. If
sources disagree, state the conflict explicitly and cite both. Cite
the specific source for every claim using the format [filename].
 
Context:
[Source: {name}]
{chunk}
 
Question: {question}

Every sentence in that template maps to a real failure mode covered above: the first line sets the boundary, the second forces an honest gap instead of a guess, the third surfaces disagreement instead of hiding it, and the fourth makes every claim checkable.

The Bottom Line

RAG's reputation for still hallucinating despite "having the right documents" is usually a prompt problem wearing a retrieval costume. Retrieval did its job, found the right chunk, but the prompt around it never told the model what to do when the context was incomplete, contradictory, or simply present, and the model filled that silence with its own confident guess. The fix isn't better retrieval. It's a prompt that draws the same boundaries this post walks through, every single time context is retrieved and handed off.

Has a RAG system you've built ever answered confidently from the wrong document, or blended in an answer that wasn't actually in what got retrieved? Drop a comment with what the prompt was missing.