Retrieval-Augmented Generation done right — chunking strategies, embedding models, and evaluation frameworks for when errors are costly.
When getting it wrong matters
Retrieval-Augmented Generation has become the default architecture for grounding LLM outputs in real data. In many applications, a retrieval miss is a minor inconvenience — the user rephrases, the system returns something useful, life goes on. In high-stakes applications — healthcare, legal, financial, compliance — a retrieval failure that causes the model to hallucinate a confident answer from incomplete context can be genuinely costly.
The architecture decisions that matter in these contexts are different from the ones that matter in a general-purpose assistant. Precision matters more than recall. Failure modes need to be explicit, not silent. Uncertainty needs to be surfaced to users, not papered over with confident-sounding prose.
Chunking strategies that actually work
How you split documents has enormous impact on retrieval quality, and yet most teams treat chunking as an afterthought. Naive chunking — fixed character counts with sliding windows — ignores document structure entirely. A 500-token chunk that starts mid-sentence and ends mid-paragraph is unlikely to contain the coherent unit of information retrieval is looking for.
Better approaches: semantic chunking based on paragraph or section boundaries; hierarchical chunking that stores both fine-grained chunks and their parent sections; and document-specific chunking that respects the logical structure of the content type. For legal documents, clause boundaries. For medical records, note boundaries. For code, function boundaries.
- Preserve semantic boundaries — never split sentences or meaningful conceptual units
- Include structural metadata in the chunk: document title, section heading, page number
- Store overlapping chunks for passages that span natural boundaries
- Index both fine-grained and coarse-grained versions for multi-level retrieval
Embedding model selection
The embedding model determines how semantic similarity is computed — and general-purpose embedding models trained on web text may perform poorly on domain-specific content. A model optimized for web documents may encode legal language, medical terminology, or financial jargon in ways that don't reflect the semantic relationships domain experts care about.
Evaluation matters here. Before committing to an embedding model, build a domain-specific test set: pairs of queries and known-relevant documents, along with hard negatives (documents that look relevant but aren't). Measure recall@k — for your test queries, what fraction of truly relevant documents appear in the top k results? This tells you far more than any benchmark score.
Hybrid retrieval and reranking
Dense retrieval (embedding-based) and sparse retrieval (BM25-style keyword matching) have complementary failure modes. Dense retrieval handles semantic paraphrasing well but can miss exact terminology. Sparse retrieval excels at precise term matching but misses synonymy. Hybrid retrieval that combines both tends to outperform either alone — especially in technical and professional domains where precise terminology matters.
Reranking is the second-pass step that should separate the candidates your retriever returns from the ones that actually go into the model context. A cross-encoder reranker — which takes the query and each candidate document as a pair and scores them jointly — dramatically outperforms the cosine similarity scores from your initial retrieval. The cost is latency; the gain is precision.
Evaluating your RAG pipeline
The most underinvested area in RAG deployments is evaluation. Teams measure end-to-end answer quality but rarely decompose it into retrieval quality and generation quality. This makes debugging hard: when the output is wrong, you don't know if the retriever didn't find the right document, or the generator hallucinated despite having the right document.
Build separate evaluation for each component. For retrieval: given a question, was the relevant document in the retrieved set? For generation: given the retrieved context, did the model produce an accurate answer? This decomposition points clearly to where to invest. Most of the time, retrieval is the bottleneck — not the model.
Handling retrieval failure gracefully
In high-stakes applications, what the system does when retrieval fails is as important as what it does when retrieval succeeds. A confident hallucination is worse than an honest 'I couldn't find the relevant information.' Design explicit handling for the case where retrieved documents don't contain the answer — and make sure the model communicates this to the user rather than confabulating.
Retrieval confidence signals — the similarity scores of your top results, the presence or absence of high-similarity matches — can inform whether to answer at all. If the best result has a similarity score below your threshold, declining to answer or flagging uncertainty is often the right choice. Train your users to expect and trust this behavior.
