RAG Pipeline

Overview

RAG grounds an LLM in your data by retrieving relevant context at query time. Most RAG quality problems are retrieval problems, not generation problems — if the right chunk isn't retrieved, no prompt can save the answer. Optimize retrieval first.

When to use

• The model must answer over private/large/changing documents.
• You need citations and reduced hallucination.
• Fine-tuning is overkill or data changes too often.

Pipeline stages

1. Ingest & chunk documents.
2. Embed chunks → vector store.
3. Retrieve (dense + sparse) for a query.
4. Rerank top candidates.
5. Assemble prompt with context + citations.
6. Generate and evaluate.

Chunking

• Start at ~500-1000 tokens with ~10-15% overlap.
• Prefer semantic/structural boundaries (headings, paragraphs) over fixed char counts.
• Keep metadata (source, title, section, URL) on every chunk for citations and filtering.

Embeddings & store

• Choose an embedding model by your domain + the MTEB leaderboard; match it at query and index time.
• Normalize vectors; use cosine similarity.
• Vector stores: pgvector (already have Postgres), Qdrant/Weaviate/Milvos (scale), FAISS (local/offline).

Hybrid retrieval + reranking (the biggest quality lever)

# 1. Dense (semantic) + 2. Sparse (BM25 keyword) -> union
dense_hits  = vstore.search(embed(query), k=20)
sparse_hits = bm25.search(query, k=20)
candidates  = dedupe(dense_hits + sparse_hits)

# 3. Cross-encoder rerank for precision
from sentence_transformers import CrossEncoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
ranked = sorted(candidates,
                key=lambda c: reranker.predict([(query, c.text)]),
                reverse=True)[:5]

Prompt assembly

• Insert only the top-k reranked chunks; respect the context window.
• Instruct: "Answer only from the context; if it's not there, say you don't know."
• Require inline citations to chunk metadata so answers are auditable.

Evaluation (measure both halves)

• Retrieval: recall@k, MRR, hit-rate on a labeled query→chunk set.
• Answer: faithfulness/groundedness, answer-relevance, context-precision (e.g., RAGAS).
• Keep a fixed eval set; track scores via experiment-tracking as you tune chunking/k.

Pitfalls

• Dense-only retrieval misses exact keywords/IDs — add BM25.
• Chunks too large dilute relevance; too small lose context.
• Mismatched embedding models at index vs query time.
• No reranker → low precision context → confident wrong answers.
• No retrieval eval → you tune blind.

Hand-off

A retriever + generator with a measured retrieval/answer score that model-serving exposes behind an API.