A RAG pipeline that works has five parts, and only one of them is a language model. You split documents into chunks that each carry their own context, embed them, store them next to filterable metadata, retrieve with both keyword and vector search, rerank the survivors with a cross-encoder, and then hand the top five to the model with an instruction to cite or refuse. Skip any of those and the system produces answers that sound right and are wrong.
Key Takeaways
- Chunking beats model choice. Start near 400 tokens with 15% overlap, split on structure, and measure two or three configurations against real questions.
- Postgres with pgvector is enough for most applications. Move to a dedicated vector database when you hit a limit you can name.
- Hybrid plus reranking is the standard shape. Keyword search catches exact identifiers, vectors catch paraphrase, a cross-encoder sorts what survives.
- Without an eval suite you are guessing. Fifty real questions with known answer chunks turns tuning into engineering.
This guide is the build, not the concept tour. If you want the conceptual version first, read RAG explained. Everything below assumes Python 3.11 or newer and a Postgres 16 database, and every snippet is runnable rather than illustrative.
The Whole Pipeline in One Page
A production RAG pipeline has two loops. The ingest loop runs offline: parse, chunk, embed, and index with metadata. The query loop runs per request: embed the question, retrieve candidates by keyword and by vector, fuse the two rankings, rerank the top candidates with a cross-encoder, assemble a context budget, and generate with a citation requirement. Everything else is variation on that shape.
Here is the dependency set and the file layout. Small on purpose, because a RAG project accumulates abstraction faster than any other kind of application.
# Python 3.11+, Postgres 16 with the pgvector extension
pip install openai psycopg[binary] pgvector tiktoken sentence-transformers
# rag/
# ingest.py parse + chunk + embed + insert
# retrieve.py hybrid search + RRF + rerank
# generate.py context assembly + prompt + citations
# eval.py golden set, recall@k, nDCG@k
# schema.sql tables and indexes
The schema is where most of the durability lives. Store the text, the vector, and the metadata you will need to filter on, in one row.
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE chunks (
id bigserial PRIMARY KEY,
doc_id text NOT NULL,
doc_title text NOT NULL,
section text,
source_url text,
updated_at timestamptz NOT NULL DEFAULT now(),
tenant_id text NOT NULL,
content text NOT NULL,
embedding vector(1536) NOT NULL,
tsv tsvector GENERATED ALWAYS AS
(to_tsvector('english', content)) STORED
);
-- build the ANN index after the bulk load, not before
CREATE INDEX chunks_emb_idx ON chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
CREATE INDEX chunks_tsv_idx ON chunks USING gin (tsv);
CREATE INDEX chunks_tenant_idx ON chunks (tenant_id, doc_id);
Two details in that schema save you later. The generated tsvector column means lexical search stays in sync with the text automatically. The tenant_id column means a permission filter is a WHERE clause instead of a rewrite.
Chunking: The Decision That Matters More Than the Model
Chunk size and chunk boundaries decide what is retrievable at all. A chunk that splits a definition from its subject, or a table from its header row, cannot be retrieved correctly by any embedding model or any reranker. Start near 400 tokens with roughly 15% overlap, split on document structure first and token counts second, and prepend the document title and section heading to the text you embed.
Character-count splitters are the most common cause of quiet RAG failure. They cut mid-sentence, orphan pronouns from their antecedents, and split tables from their headers. A structure-aware splitter cuts at headings and paragraph boundaries, then packs paragraphs up to a token budget.
import re
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
TARGET_TOKENS = 400
OVERLAP_TOKENS = 60
def token_len(text: str) -> int:
return len(enc.encode(text))
def split_sections(markdown: str):
"""Yield (heading_path, body) using ATX headings as boundaries."""
path, buf = [], []
for line in markdown.splitlines():
m = re.match(r"^(#{1,6})\s+(.*)$", line)
if m:
if buf:
yield " / ".join(path), "\n".join(buf).strip()
buf = []
level = len(m.group(1))
path = path[: level - 1] + [m.group(2).strip()]
else:
buf.append(line)
if buf:
yield " / ".join(path), "\n".join(buf).strip()
def tail_of(paragraphs, overlap):
tail, total = [], 0
for p in reversed(paragraphs):
tail.insert(0, p)
total += token_len(p)
if total >= overlap:
break
return tail, total
def pack(paragraphs, target=TARGET_TOKENS, overlap=OVERLAP_TOKENS):
chunks, cur, cur_tokens = [], [], 0
for p in paragraphs:
t = token_len(p)
if cur and cur_tokens + t > target:
chunks.append("\n\n".join(cur))
cur, cur_tokens = tail_of(cur, overlap)
cur.append(p)
cur_tokens += t
if cur:
chunks.append("\n\n".join(cur))
return chunks
def chunk_document(markdown: str, doc_id: str, doc_title: str):
out = []
for heading, body in split_sections(markdown):
if not body:
continue
paras = [p.strip() for p in re.split(r"\n\s*\n", body) if p.strip()]
for i, text in enumerate(pack(paras)):
# the embedded string carries its own context
header = f"{doc_title} / {heading}" if heading else doc_title
out.append({
"doc_id": doc_id,
"doc_title": doc_title,
"section": heading or doc_title,
"ordinal": i,
"content": f"{header}\n\n{text}",
})
return out
Two caveats before you run it. A paragraph longer than the target ships as an oversized chunk, so wide tables and long code listings need a handler that keeps the header row attached. And the header prefix is doing real work: it is the cheap version of contextual retrieval, the technique Anthropic published in which a short model-written note about each chunk's place in its document is prepended before embedding.
How to actually pick a chunk size
Do not argue about it. Index the same corpus three times at 200, 400, and 800 tokens, run the eval suite from section 07 against each, and compare recall@20. Dense reference material and policy documents usually favor the small end because one chunk holds one fact. Tutorials and narrative prose usually favor the large end because the reasoning needs to stay together. The experiment takes an afternoon and settles the question for your data instead of someone else's.
Embeddings: Choosing a Model Without Guessing
Pick an embedding model by three properties: retrieval quality on data like yours, dimension count (which drives storage and index size), and whether it must run inside your own network. Hosted options such as OpenAI's text-embedding-3-small and -large or Cohere's embed v3 family are the fastest path; open models like the BGE and E5 families run locally through sentence-transformers when data cannot leave your environment.
Dimension count is a cost, not a quality score. A 3072-dimension vector takes twice the storage and roughly twice the index memory of a 1536-dimension one, for gains that are often small on domain data. Some models support Matryoshka truncation, where you request fewer dimensions and keep most of the quality. Check the MTEB leaderboard on Hugging Face for current scores rather than trusting any ranking printed in an article, including this one.
import psycopg
from pgvector.psycopg import register_vector
from openai import OpenAI
client = OpenAI()
MODEL = "text-embedding-3-small" # 1536 dims
def embed(texts: list[str]) -> list[list[float]]:
resp = client.embeddings.create(model=MODEL, input=texts)
return [d.embedding for d in resp.data]
def index_chunks(dsn: str, records, tenant_id: str, batch=128):
with psycopg.connect(dsn) as conn:
register_vector(conn)
for i in range(0, len(records), batch):
window = records[i : i + batch]
vectors = embed([r["content"] for r in window])
with conn.cursor() as cur:
cur.executemany(
"""INSERT INTO chunks
(doc_id, doc_title, section, tenant_id, content, embedding)
VALUES (%s, %s, %s, %s, %s, %s)""",
[(r["doc_id"], r["doc_title"], r["section"],
tenant_id, r["content"], v)
for r, v in zip(window, vectors)],
)
conn.commit()
One rule with no exceptions: the query and the documents must be embedded by the same model and the same version. Swapping the embedding model means re-embedding the entire corpus. Some open models also expect asymmetric prefixes, so a question is embedded as query: ... and a passage as passage: .... Getting that backwards degrades retrieval silently and is a common bug in code copied between projects. Our explainer on embeddings covers the underlying geometry.
Vector Store: pgvector vs Qdrant vs Chroma vs Pinecone
Postgres with pgvector is the right default for most applications because it puts vectors, metadata, and transactions in one system you already back up. Qdrant and Milvus earn their operational cost at very large scale or when you need native sparse-dense hybrid scoring. Chroma is for local prototypes. Pinecone is for teams who would rather pay than operate anything.
| Option | Where it runs | Metadata filtering | Built-in hybrid | Best for |
|---|---|---|---|---|
| Postgres + pgvector | Your existing database | Full SQL | Assemble it yourself | Almost every app under ~10M vectors |
| Qdrant | Separate service or cloud | Rich payload filters | Sparse + dense native | Large corpora, filter-heavy search |
| Milvus | Heavier cluster to operate | Yes | Yes | Hundreds of millions of vectors |
| Chroma | In-process, local | Basic | No | Prototypes and notebooks |
| Pinecone | Managed only | Yes | Yes | Teams with no ops capacity |
| OpenSearch / Elasticsearch | Cluster you run | Yes | True BM25 + kNN | Shops already running the cluster |
The honest comparison is about operations, not benchmarks. Every option here returns good neighbors; what differs is whether you now have a second stateful system to back up, migrate, and keep in sync with your source of truth. The detailed vector database comparison goes deeper on index types and pricing; the Postgres guide covers pgvector tuning alongside the rest of the database.
Two pgvector settings that quietly cost you recall
- Build the HNSW index after the bulk load. Building first makes ingest slow and the graph worse.
- Raise
hnsw.ef_searchat query time. The default favors speed;SET hnsw.ef_search = 100;before a search costs a few milliseconds and recovers neighbors the default misses. Measure both.
Hybrid Search: Lexical Plus Vector, Fused With RRF
Vector search alone misses exact tokens: part numbers, error codes, statute citations, rare proper nouns. Lexical search alone misses paraphrase. Run both, then combine the two ranked lists with reciprocal rank fusion, which scores each document as the sum of 1/(k + rank) across the lists it appears in, with k=60 from the original 2009 paper by Cormack, Clarke, and Buettcher.
RRF has a property that makes it the pragmatic default: it uses ranks, not scores, so you never have to normalize a cosine similarity against a lexical relevance score. The two systems do not need to agree on units.
def vector_ids(conn, qvec, tenant_id, limit=50):
with conn.cursor() as cur:
cur.execute("SET LOCAL hnsw.ef_search = 100")
cur.execute(
"""SELECT id FROM chunks
WHERE tenant_id = %s
ORDER BY embedding <=> %s
LIMIT %s""",
(tenant_id, qvec, limit),
)
return [r[0] for r in cur.fetchall()]
def lexical_ids(conn, query, tenant_id, limit=50):
with conn.cursor() as cur:
cur.execute(
"""SELECT id
FROM chunks
WHERE tenant_id = %s
AND tsv @@ websearch_to_tsquery('english', %s)
ORDER BY ts_rank_cd(tsv,
websearch_to_tsquery('english', %s)) DESC
LIMIT %s""",
(tenant_id, query, query, limit),
)
return [r[0] for r in cur.fetchall()]
def rrf(*rankings, k=60, top_n=20):
"""Reciprocal rank fusion over any number of ranked ID lists."""
scores = {}
for ranking in rankings:
for rank, doc_id in enumerate(ranking, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)[:top_n]
def hybrid_search(conn, query, tenant_id, top_n=20):
qvec = embed([query])[0]
return rrf(
vector_ids(conn, qvec, tenant_id),
lexical_ids(conn, query, tenant_id),
top_n=top_n,
)
One accuracy note so you are not surprised in production: Postgres full-text search is not BM25. ts_rank_cd is a serviceable lexical ranker but it does not do BM25's document-length normalization. If lexical precision is central to your product, use OpenSearch, or add a BM25 extension to Postgres, or compute BM25 in the application with the rank_bm25 package over a candidate set.
Reranking: The Best Quality per Line of Code
An embedding model encodes the query and the document separately, so relevance is only ever approximated by vector distance. A cross-encoder reads the query and the passage together in a single forward pass and scores the pair directly. Retrieve 30 to 50 candidates cheaply, rerank them, keep the top 5. This is usually the largest single quality improvement available in a RAG pipeline.
from sentence_transformers import CrossEncoder
# small and fast; swap for a larger reranker if latency allows
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2", max_length=512)
def rerank(query: str, candidates: list[dict], top_n=5):
if not candidates:
return []
pairs = [(query, c["content"]) for c in candidates]
scores = reranker.predict(pairs)
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
return [dict(c, rerank_score=float(s)) for c, s in ranked[:top_n]]
The cost is latency and it is real. A cross-encoder scores every pair, so reranking 50 candidates is 50 forward passes. On CPU with a small model that is typically tens to a few hundred milliseconds; on GPU it is negligible; with a hosted reranker it is a network round trip. Measure it against your latency budget before deciding how wide to retrieve. If the budget is tight, retrieve 20 instead of 50 rather than dropping the reranker.
Measuring Retrieval Quality With Real Metrics
Build a golden set of at least 50 real questions paired with the chunk IDs that should answer them, then measure recall@k, MRR, and nDCG@k on every change. Recall@k tells you whether the right chunk is in the candidate pool at all. MRR tells you how high it lands. nDCG@k weights position. Run the suite in CI so a silent recall regression fails the build.
Take questions from real user logs or support tickets. A model can draft candidates from your chunks, but a human confirms which chunk is the answer, otherwise the eval set inherits the pipeline's blind spots.
import json, math, statistics
def recall_at_k(retrieved, relevant, k):
if not relevant:
return None
hits = len(set(retrieved[:k]) & set(relevant))
return hits / len(relevant)
def mrr(retrieved, relevant):
rel = set(relevant)
for i, doc_id in enumerate(retrieved, start=1):
if doc_id in rel:
return 1.0 / i
return 0.0
def ndcg_at_k(retrieved, relevant, k):
rel = set(relevant)
dcg = sum(1.0 / math.log2(i + 2)
for i, d in enumerate(retrieved[:k]) if d in rel)
ideal = sum(1.0 / math.log2(i + 2)
for i in range(min(len(rel), k)))
return dcg / ideal if ideal else 0.0
def run_eval(conn, path="golden.jsonl", k=20):
# each line: {"q": "...", "relevant_ids": [12, 87], "tenant": "acme"}
rows = [json.loads(l) for l in open(path)]
r, m, n = [], [], []
for row in rows:
ids = hybrid_search(conn, row["q"], row["tenant"], top_n=50)
r.append(recall_at_k(ids, row["relevant_ids"], k))
m.append(mrr(ids, row["relevant_ids"]))
n.append(ndcg_at_k(ids, row["relevant_ids"], k))
return {
f"recall@{k}": round(statistics.mean(r), 3),
"mrr": round(statistics.mean(m), 3),
f"ndcg@{k}": round(statistics.mean(n), 3),
"n": len(rows),
}
Keep retrieval metrics separate from answer metrics. If recall@50 is low, no amount of prompt work will fix the answer. If recall is high and answers are still wrong, the bug is in context assembly or the generation prompt. Layer answer-level checks such as faithfulness and citation accuracy on top afterward; the LLM evaluation guide covers that half, and structured outputs make grading far easier to automate.
Assembling Context and Forcing Citations
Number the retrieved passages, spend a fixed token budget on them, and instruct the model to cite the number after every claim and to emit a specific refusal string when the sources do not contain the answer. A refusal you can detect programmatically is worth more than a fluent guess.
SYSTEM = (
"Answer only from the numbered sources below. "
"Cite the source number in square brackets after every claim. "
"If the sources do not answer the question, reply exactly: NOT_IN_SOURCES"
)
def build_context(chunks, budget_tokens=3000):
parts, used = [], 0
for i, c in enumerate(chunks, start=1):
block = f"[{i}] {c['doc_title']} / {c['section']}\n{c['content']}"
t = token_len(block)
if used + t > budget_tokens:
break
parts.append(block)
used += t
return "\n\n".join(parts)
def answer(conn, question, tenant_id):
ids = hybrid_search(conn, question, tenant_id, top_n=30)
candidates = load_chunks(conn, ids) # SELECT ... WHERE id = ANY(%s)
top = rerank(question, candidates, top_n=5)
context = build_context(top)
resp = client.chat.completions.create(
model="gpt-4.1-mini",
messages=[
{"role": "system", "content": SYSTEM},
{"role": "user",
"content": f"Sources:\n{context}\n\nQuestion: {question}"},
],
temperature=0,
)
return resp.choices[0].message.content, top
Returning the top chunks alongside the text matters. It lets the interface render real source links, and it lets you log which chunks produced which answer, which is how you find the bad chunk when a user reports a wrong answer three weeks later.
The Failure Modes Everyone Hits
Chunks with no context of their own
A chunk that says "the limit is 90 days" is unretrievable for "what is the appeal window" because nothing in the text names the subject. The fix is a heading path or a model-written context line prepended before embedding.
Missing metadata
Without tenant, date, and source columns you cannot filter by permission, prefer the current revision, or show a citation. Superseded documents then win retrieval and the system confidently quotes last year's policy.
No eval suite
Without a golden set, every change is judged by spot-checking three questions someone remembers. Improvements and regressions look identical, and tuning becomes superstition.
Near-duplicate chunks
Boilerplate headers and repeated policy language fill the top 5 with five copies of the same passage, so the real answer never makes the context window even though retrieval "worked."
Three more that show up in code review. Changing the embedding model without re-embedding the corpus, which produces vectors from two incompatible spaces in one table. Retrieving only 5 candidates and then reranking them, which cannot recover anything the first stage missed. And measuring nothing end to end, so a retrieval fix that helps recall but hurts answers ships unnoticed.
When Not to Build RAG at All
RAG is the wrong tool when the corpus fits in the context window, when the question is an aggregate rather than a lookup, or when what you actually need is a change in the model's style rather than its knowledge.
Small corpus, aggregate questions, style changes
A 40-page handbook belongs in the prompt with caching enabled. "How many contracts closed last quarter" is a SQL query, not a similarity search; retrieval over rows returns some rows, never a correct count. And teaching a model your house format is fine-tuning territory, not retrieval.
Large, changing, permissioned, citable
The corpus is far larger than the window, documents change weekly, different users may see different subsets, and answers must cite a specific source record. Those four conditions are what the pipeline above is for, and no context window size removes them.
There is a middle path worth knowing: retrieve a generous candidate set and let a large-context model read all of it, skipping the reranker. That trades token cost for pipeline simplicity, and for low-volume internal tools it is often the right trade.
Build the Smallest Version First
The fastest way to a working system is to build it in the order the failures appear. Ingest 20 real documents with the structure-aware splitter. Write 50 questions from real users and label the answer chunks. Get vector-only retrieval running and record recall@20. Add lexical search and RRF, measure again. Add the reranker, measure again. Only then write the generation prompt. Every step after the eval suite is measurable, which is the whole point.
Frameworks help once the shape is clear. LlamaIndex gives you ingest and retrieval abstractions with less code, and LangChain gives you composition across steps. Both are easier to use well after you have built the raw version once, because you will know what their defaults are hiding. If you want a narrower end-to-end walkthrough in Python, the RAG tutorial builds a smaller version of this same pipeline, and the vector databases and embeddings guide covers the storage layer in more depth.
Frequently Asked Questions
What chunk size should I use for RAG?
Start at roughly 300 to 500 tokens with about 15% overlap, split on document structure rather than raw character counts. That range is a starting point, not an answer. Dense reference material and policy text usually retrieve better at the small end because a single chunk then holds a single fact, while tutorial and narrative prose usually needs larger chunks so the reasoning stays intact. Index the corpus at two or three sizes and compare recall@20 on your own questions. Changing chunk size is almost always cheaper and more effective than changing the language model.
Do I need a dedicated vector database, or is Postgres enough?
For most applications, Postgres with pgvector is enough. You get HNSW indexing, cosine and L2 distance, metadata filtering with ordinary SQL, transactional writes, and one system to back up instead of two. Dedicated vector databases such as Qdrant, Weaviate, and Milvus become worth the operational cost past roughly ten million vectors, or when you need native sparse-dense scoring and per-collection quantization controls that pgvector does not expose. Start on Postgres and migrate when you hit a limit you can name.
Does a reranker actually improve RAG quality?
Yes, and it is usually the best quality gain per line of code in the whole pipeline. Embedding search compares vectors that were computed independently, so it can only approximate relevance. A cross-encoder reads the query and the candidate passage together in one forward pass and scores the pair directly, which catches passages that were close in vector space but do not answer the question. Retrieve 30 to 50 candidates, rerank, pass the top 5. The cost is latency, from tens of milliseconds to a few hundred depending on model size and hardware.
How do I know whether my RAG pipeline is getting better?
Build a golden set of 50 or more real questions with the chunk IDs that should be retrieved, then measure recall@k, MRR, and nDCG@k on every change and run it in CI. Keep retrieval metrics separate from answer metrics: low recall means no prompt will save the answer, and high recall with wrong answers means the bug is in context assembly or the generation prompt. Without that split, tuning is guessing.
Is RAG still necessary now that models have very large context windows?
For a corpus that fits comfortably in the window, no. A few dozen pages belong in the prompt with caching enabled, which is simpler and often more accurate than any retrieval pipeline. RAG earns its complexity when the corpus is far larger than the window, when documents change often enough that re-sending everything is wasteful, when different users may see different subsets of the data, or when answers must cite a specific source record. Large context windows shrank the set of problems that need RAG without removing the ones that do.
Further reading: pgvector documentation, MTEB retrieval leaderboard, Sentence-Transformers retrieve-and-rerank