RAG Pipeline (Retrieval-Augmented Generation)

What it is

Retrieval-Augmented Generation is an architecture where the model is handed the relevant facts at question time instead of being expected to remember them. The pipeline retrieves the most relevant pieces of your own data, puts them into the prompt, and only then asks the LLM to answer.

The LLM never learns your data. It reads it, the same way you would read a reference page before answering a question in an open-book exam. Change the data and the next answer reflects the change immediately, with no retraining.

This site is a RAG product. The knowledge base is a plain text file about my career. A FastAPI backend embeds it at startup, retrieves the five most relevant paragraphs for each visitor question, and asks gpt-4o-mini to answer from them, returning JSON that also tells the React frontend which card to display.

Why it's needed

LLMs have a training cutoff, know nothing about private data and will confidently invent details when asked about things they have not seen. Fine-tuning does not fix that: it changes style and behaviour, not reliable recall of facts, and it has to be repeated every time the data changes.

RAG is the standard answer to 'make the model answer from our documents'. It is cheaper than fine-tuning, updates instantly, lets you show sources, and keeps the data on your side of the API boundary. Almost every production LLM feature that touches company data is some form of RAG.

How it works

• Index time: split the source into chunks, embed each chunk, store vectors alongside the text. Do this once at startup or whenever the data changes.
• Query time, step 1: embed the user's question with the same model.
• Step 2 (Retrieval): score the question against every chunk and keep the top k.
• Step 3 (Augmentation): build a prompt that contains the retrieved chunks, clear instructions and the question.
• Step 4 (Generation): send the prompt to the LLM and return its answer, ideally with the chunks it used as sources.
• Wrap it in a fallback: if the LLM call fails, return a keyword-based answer or a graceful message rather than a 500.

Code example

import numpy as np
from fastapi import FastAPI
from pydantic import BaseModel
from sentence_transformers import SentenceTransformer
from openai import OpenAI

app = FastAPI()
client = OpenAI()
model = SentenceTransformer("all-MiniLM-L6-v2")

# --- index time: once at startup ---
text = open("knowledge_base.txt", encoding="utf-8").read()
chunks = [p.strip() for p in text.split("\n\n") if p.strip()]
index = model.encode(chunks, normalize_embeddings=True)

class Ask(BaseModel):
    question: str

@app.post("/ask")
def ask(req: Ask):
    # --- query time ---
    q = model.encode([req.question], normalize_embeddings=True)[0]
    top = np.argsort(index @ q)[::-1][:5]                 # retrieval
    context = "\n\n".join(chunks[i] for i in top)
    prompt = ("Answer only from the context below. If the answer is not "
              f"there, say so.\n\nContext:\n{context}\n\nQuestion: {req.question}")
    reply = client.chat.completions.create(                 # generation
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return {"answer": reply.choices[0].message.content,
            "sources": [chunks[i] for i in top]}

Where I used this

Use cases

• Customer support assistants grounded in help-centre articles
• Internal knowledge search over wikis, tickets and documents
• Portfolio and resume assistants like this one
• Contract and policy Q&A where every answer must cite its source
• Product assistants that read live catalogue data

Common mistakes

• No instruction to stay inside the context, so the model blends retrieved facts with its own guesses and you cannot tell which is which.
• Embedding the knowledge base inside the request handler instead of at startup, making every call seconds slower.
• Returning answers without sources. Sources are how you debug retrieval and how users learn to trust the system.
• Testing with three questions you wrote yourself. Real users phrase things differently; collect their questions and build a test set.
• No fallback path. When the LLM provider has an outage your whole product goes down with it.

Interview questions