What Is RAG? Retrieval-Augmented Generation Explained Simply (with Python)

RAG (retrieval-augmented generation) means searching your own data for the passages that answer a question, then giving those passages to a large language model along with the question. The model answers from that context instead of from memory, so it can use private, recent information and cite its sources, with no retraining.

Why do LLMs need RAG?

A language model only knows what was in its training data. That causes three problems in real applications:

• It doesn't know your data. Your company wiki, product docs, contracts or support tickets were never in the training set.
• Its knowledge is frozen. Anything that changed after the training cutoff is missing or out of date.
• It answers confidently anyway. Ask about something it doesn't know and it may produce a fluent, wrong answer, often called a hallucination.

You could paste every document into the prompt, but context windows are limited, long prompts cost more and get slower, and models attend less reliably to details buried in very long inputs. RAG solves this by sending only the few passages that matter for this question.

Think of it as an open-book exam: the model is still the student writing the answer, but retrieval hands it the right pages first.

How does a RAG pipeline work?

A RAG system has two halves. Ingestion prepares your documents once (and again whenever they change). Querying runs on every question.

The quality of the final answer is capped by retrieval. If the right chunk is not in the prompt, no model can answer correctly from it, which is why most RAG work is really search work.

How should you chunk documents?

Chunking decides what a "passage" is, and it has more effect on answer quality than most people expect.

Practical tips from building these systems:

• Prepend context to each chunk. A chunk that says "It expires after 30 days" is useless alone. Store it as "Refund policy → Store credit: It expires after 30 days".
• Clean the text first. Headers, footers, navigation menus and broken tables from PDF extraction add noise that ends up in the prompt.
• Keep tables and code blocks whole where you can; splitting them mid-row produces chunks nobody can use.

There is no universally best chunk size. Pick a sensible default, build a small test set of real questions, and adjust based on what retrieval actually returns.

What are embeddings and vector databases?

An embedding is a list of numbers (a vector) that represents the meaning of a piece of text. Texts with similar meaning get vectors that point in similar directions, so "How do I reset my password?" lands close to "Steps to change your login credentials" even though they share few words.

Similarity is usually measured with cosine similarity: the cosine of the angle between two vectors, where 1 means same direction and values near 0 mean unrelated.

A vector database stores those vectors and finds the nearest ones quickly. Exact search compares the question with every vector, which is fine for thousands of chunks. For millions, databases use approximate nearest neighbour (ANN) indexes such as HNSW, trading a tiny amount of accuracy for a large speed-up.

Common options, with no single right answer:

• In-memory or a local library (for example Chroma or FAISS): prototypes and small datasets.
• pgvector: a PostgreSQL extension, good when your app already uses Postgres and you want vectors next to your relational data and permissions.
• Managed vector databases (for example Pinecone, Weaviate, Qdrant): large collections, high query volume, or teams that do not want to operate the infrastructure.

Many production systems also use hybrid search: combine vector similarity with keyword search (BM25) so exact terms such as product codes, error messages and names are not missed.

How do you build a simple RAG pipeline in Python?

Here is the whole idea in a few lines. embed() and llm() stand for whichever embedding model and LLM API you use; the retrieval logic is plain NumPy.

import numpy as np

def embed(texts: list[str]) -> np.ndarray:
    """Call your embedding model; return one vector per text."""
    ...

def llm(prompt: str) -> str:
    """Call your chat model; return its reply."""
    ...

# --- Ingest (run once) ---
chunks = [
    "Refunds: items can be returned within 30 days of delivery.",
    "Refunds: store credit expires 12 months after it is issued.",
    "Shipping: orders over 999 rupees ship free within India.",
]
vectors = embed(chunks)
vectors = vectors / np.linalg.norm(vectors, axis=1, keepdims=True)

# --- Query (run per question) ---
def answer(question: str, k: int = 2) -> str:
    q = embed([question])[0]
    q = q / np.linalg.norm(q)
    scores = vectors @ q                      # cosine similarity
    top = np.argsort(scores)[::-1][:k]        # best k chunks

    context = "\n".join(f"[{i + 1}] {chunks[j]}" for i, j in enumerate(top))
    prompt = (
        "Answer using only the context below. Cite sources like [1]. "
        "If the answer is not in the context, say you don't know.\n\n"
        f"Context:\n{context}\n\nQuestion: {question}"
    )
    return llm(prompt)

print(answer("How long do I have to return something?"))

Normalising every vector to length 1 makes the dot product equal to cosine similarity, so one matrix multiplication scores every chunk at once. Swap the NumPy array for a vector database when the collection grows; the shape of the code stays the same.

RAG vs fine-tuning vs prompting: which should you use?

Start with prompting. Add RAG when the model needs knowledge it doesn't have. Consider fine-tuning when prompting and RAG work but the model still does not follow your format or style reliably. They also combine: a fine-tuned model can still sit behind a RAG pipeline.

How do you evaluate a RAG system?

"It looked right in the demo" is not evaluation. Build a test set of 30 to 100 real questions with the answer and the source passage for each, and measure the two halves separately:

Generation metrics are often scored by another LLM acting as a judge, with a human spot-checking a sample. Re-run the test set whenever you change chunking, the embedding model, k, the prompt or the LLM, so you know whether a change helped.

What are common RAG failures and how do you fix them?

What is agentic RAG?

Basic RAG retrieves once and answers. Agentic RAG lets the model decide how to search: it can rewrite the question, split it into sub-questions, call retrieval as a tool several times, query different sources (a vector store, a SQL database, a web search), and check whether it has enough information before answering.

It handles multi-step questions such as "Compare our refund policy with last year's" much better, at the cost of more LLM calls, higher latency and more ways to go wrong. Get simple retrieval working and measured first, then add agent behaviour where single-shot retrieval clearly falls short.

Key takeaway

RAG is search plus a language model: retrieve the passages that answer the question, put them in the prompt, and make the model answer only from them with citations. Spend your effort on clean text, sensible chunks, the right embedding model and a test set, because retrieval quality sets the ceiling for every answer. For a one-page visual of the pipeline and the surrounding ML and GenAI concepts, see the free AI/ML and GenAI handwritten notes (the RAG pipeline is on page 20), and brush up on the language with the Python handwritten notes. If you want a RAG system or AI assistant built for your own data, see AI engineering services.