Skip to content

Technique

Advanced

Retrieval-Augmented Generation (RAG)

Ground AI answers in relevant source documents retrieved at question time, instead of relying only on what the model memorized.

What it is

Retrieval-Augmented Generation (RAG) is an application pattern, not a single prompt. When a question comes in, the system first retrieves relevant passages from a knowledge source (company docs, a help center, a database, the web), then inserts them into the prompt and asks the model to answer using that material.

RAG helps with two core limits of language models: they don't know your private information, and their built-in knowledge stops at a training cutoff. Grounding answers in retrieved text makes them more current and checkable, though it doesn't eliminate errors.

When to use it

  • Answering questions about private or specialized documents (policies, manuals, contracts)
  • Information changes often and the model's built-in knowledge may be outdated
  • Answers must be traceable to a source, with citations or quotes
  • Building a support bot, internal search assistant, or document Q&A tool

How it works

The pattern

A typical RAG system has three stages:

  1. Index: documents are split into chunks and stored so they can be searched, often using embeddings (numeric representations of meaning) in a vector database, sometimes combined with keyword search.
  2. Retrieve: the user's question is used to find the most relevant chunks.
  3. Generate: those chunks are placed in the prompt along with instructions to answer from them.

The prompt is the part that turns retrieved text into a grounded answer:

Prompt
Answer the question using ONLY the sources below.
- Quote the exact sentence(s) that support your answer and cite the source ID.
- If the sources don't contain the answer, say "I couldn't find this in
  the provided documents" instead of guessing.

<sources>
[1] Employee Handbook, section 4.2: "..."
[2] HR FAQ, updated March: "..."
</sources>

Question: How many days of parental leave do new parents get?

The instructions to use only the sources, cite them, and admit when the answer isn't there are what make the output trustworthy.

The non-developer version

You don't need a vector database to use the spirit of RAG. Paste the relevant document into your chat and ask for a grounded answer:

Before

What's the cancellation policy for my gym?

After

Below is my gym membership contract. Using only this text, tell me how to cancel and whether there's a fee. Quote the exact clauses you rely on. If the contract doesn't say, tell me that rather than guessing. Contract: """ [paste contract] """

The first prompt invites a generic or invented answer. The second grounds it in your actual contract, and the quotes let you verify every claim.

Limits

Watch out: RAG reduces but doesn't eliminate hallucination. The model can misread a passage or blend sources. If retrieval finds the wrong chunks, the answer will be confidently wrong. Always spot-check the quotes against the original.

Answer quality depends heavily on retrieval quality: chunk size, search method, and how many passages are included all matter. Very long pasted documents can also make it harder for the model to find details, so include the most relevant sections when you can.

Step by step

  1. 1

    Collect the source material the answer should come from.

  2. 2

    Select the most relevant passages (by search in an app, or by choosing sections yourself).

  3. 3

    Insert them into the prompt with clear labels or IDs.

  4. 4

    Instruct the model to answer only from the sources and to cite or quote them.

  5. 5

    Tell it exactly what to say when the sources don't contain the answer.

  6. 6

    Verify quoted passages against the originals before relying on the answer.

Weak vs strong

Weak prompt
What does our company's remote work policy say about working from abroad?
Why it falls short: The model has never seen your company's policy, so it will either refuse or, worse, produce a plausible-sounding generic policy that may be completely wrong for you.
Strong prompt
Using only the remote work policy pasted below, answer: Can I work from another country for 3 weeks? Quote the relevant section(s) word for word, then give a one-sentence answer. If the policy doesn't address this, say "Not covered in the policy" and suggest who I might ask.

Policy:
"""
[paste policy]
"""
Why it works: It supplies the source, restricts the answer to it, requires verbatim quotes you can check, and defines a safe fallback when the answer isn't there.

Practical use cases

  • Document Q&A

    Ask questions about a lease, insurance policy, or user manual by pasting it and requesting quoted answers.

  • Customer support assistants

    Apps retrieve the relevant help-center articles for each question so replies match current policies.

  • Internal knowledge search

    Teams let employees ask natural-language questions across wikis, handbooks, and past project notes.

  • Research with citations

    Summarize several papers or reports with every claim tied to a specific source and passage.

Practice exercise

Grounded answers by hand. Find a real document you rely on: a product manual, a lease, a course syllabus, or a policy.

  1. Paste it and ask three questions, requiring quoted evidence and a "not in the document" fallback.
  2. Include one question the document doesn't answer.
  3. Check every quote against the original and note whether the AI correctly admitted the missing answer.
Practice in the PlaygroundAll techniques