What is RAG?

What is RAG?

RAG stands for Retrieval-Augmented Generation. It's a technique that gives language models access to information they weren't trained on, by fetching relevant content and injecting it into the context before generating a response.

It solves the two biggest limitations of pure parametric knowledge: the knowledge cutoff problem and hallucination on specific factual questions.

How RAG Works

The basic flow:

1. User asks a question: "What were our Q3 sales figures?"
2. Retrieve relevant documents: search a knowledge base, database, or document store for content related to the question
3. Augment the prompt: inject the retrieved documents into the context along with the original question
4. Generate a grounded response: the model now answers based on the retrieved content, not just its weights

Because the actual source material is in the context window, the model can quote from it, refer to it, and be constrained by it. This dramatically reduces hallucination on factual questions — the model doesn't need to invent an answer, it just needs to summarise or extract from what it's been given.

Why RAG is Widely Used

RAG has become the dominant pattern for building AI applications that need to work with specific, up-to-date, or proprietary information:

Customer support bots: retrieve from help documentation Enterprise search: query internal wikis, reports, emails Research assistants: retrieve academic papers Legal AI: retrieve case law and contracts Financial analysis: retrieve earnings reports and filings

The alternative — fine-tuning the model on private data — is more expensive, harder to update, and doesn't provide the same kind of "show your work" transparency. With RAG, you can see which documents were retrieved and cited.

Limitations of RAG: retrieval quality depends on the search system. If the wrong documents are fetched, the answer will be wrong or incomplete. "Garbage in, garbage out" still applies.

Chunking and Retrieval

In practice, documents are too long to inject wholesale into a context window. RAG systems typically:

1. Chunk documents into smaller segments (paragraphs, pages, or semantic units)
2. Embed each chunk as a vector (a numerical representation of its meaning)
3. Index those vectors in a vector database
4. At query time, embed the user's question and find the most similar chunks (nearest-neighbour search)
5. Inject the top-k chunks into the context

This is why vector databases (Pinecone, Weaviate, pgvector, Chroma) have become important infrastructure for AI applications — they're the backbone of most RAG systems.