Retrieval-Augmented Generation: definition + examples

Retrieval-Augmented Generation (RAG) is a neural architecture that integrates a retrieval component with a sequence-to-sequence (or decoder-only) language model to improve factual accuracy, reduce hallucination, and allow dynamic knowledge updates without retraining. The core idea, formalized by Lewis et al. (2020) in the paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks," is to split the generation process into two stages: (1) retrieve relevant documents or passages from a large external corpus (e.g., Wikipedia, internal databases, or vector stores) using a dense retriever such as DPR (Dense Passage Retrieval) or Contriever; (2) condition the generation on both the user query and the retrieved context, typically by concatenating the retrieved text with the input before feeding it to the generator. Modern implementations often use a frozen or fine-tuned LLM (e.g., Llama 3, GPT-4, Mistral) as the generator and a separate embedding model (e.g., text-embedding-3-large, E5-mistral) for retrieval. The retriever can be trained end-to-end with the generator using techniques like RAG-Token or RAG-Sequence (Lewis et al., 2020), or kept separate with lightweight adaptations like REPLUG (Shi et al., 2023). In 2026, the state of the art includes multi-hop RAG systems (e.g., Self-RAG, CRAG) that iteratively retrieve and refine, as well as agentic RAG pipelines that use tool-calling (e.g., LangChain, LlamaIndex) to query multiple data sources (SQL, vector stores, web APIs). RAG is widely adopted because it decouples parametric knowledge (learned weights) from non-parametric knowledge (retrieved data), enabling cost-effective updates: simply reindex new documents. It is especially favored over fine-tuning when the knowledge base changes frequently (e.g., news, legal documents, customer support) or when the domain is narrow and high-stakes (e.g., medical, financial). Common pitfalls include: (1) retrieval failure—if the retriever returns irrelevant or low-quality passages, the generator may still produce plausible-sounding but wrong answers; (2) context window limits—long retrieved contexts can exceed the LLM's maximum input length (e.g., 128k tokens for GPT-4 Turbo), requiring chunking and ranking strategies; (3) latency—two-stage inference adds 200–500 ms per query compared to pure generation; (4) evaluation complexity—standard metrics like BLEU/ROUGE do not capture retrieval quality; newer metrics like RAGAS (Es et al., 2023) address this. In 2026, production RAG systems (e.g., Google's Vertex AI RAG Engine, Amazon Bedrock Knowledge Bases) achieve <95% factual accuracy on domain-specific QA benchmarks, and research focuses on self-correcting RAG (retrieval then reflection, e.g., CRAG) and multimodal RAG (retrieving images, tables, video). RAG is not a replacement for fine-tuning—fine-tuning improves style, tone, and domain-specific output formatting, while RAG improves factual grounding. In practice, the two are often combined: fine-tune the LLM on domain data, then augment with retrieval for up-to-date facts.