RAG Explained 2026: What Is Retrieval-Augmented Generation?

What Is RAG? A Plain-English Definition

Retrieval-augmented generation is not a new model — it's an architecture that sits around an existing LLM. IBM defines it as a pattern that combines retrieval from external sources with generation, so a model can answer using current or proprietary information rather than relying solely on what it learned during training. Crucially, RAG doesn't touch the model's weights.

IBM's own reference architecture splits the pattern into two distinct phases. The first happens at build time: source data is converted into vectors by an embedding model and stored as chunks, specifically to make retrieval easier later. The second happens at runtime, every time a user submits a prompt: the system searches that stored data, pulls back the most relevant chunks, and hands them to the model alongside the question.

That two-phase split matters for how you scope a project. The build-time phase is a data-engineering job — cleaning, chunking, embedding, indexing. The runtime phase is a search-and-generate job. Neither one, on its own, is a large-model-training exercise, which is why RAG pilots tend to look very different from the GPU-cluster narrative that dominates most AI vendor pitches.

Why RAG: The Core Problem It Solves for LLMs

LLMs are trained on a fixed snapshot of data. Ask one about a policy update from last month, an internal pricing sheet, or a contract clause specific to your organisation, and it will either say it doesn't know or, more dangerously, guess. Equinix frames RAG's job plainly: it's used to optimise AI inference by helping models generate more accurate results through retrieval of relevant information, rather than through retraining.

For a UK business this reframes a familiar problem. Company knowledge — HR policies, product specs, compliance procedures, contract terms — usually lives in SharePoint, wikis, PDFs and ticketing systems, updated constantly. Baking that into a model's weights via fine-tuning means retraining every time something changes. RAG instead keeps the knowledge in an external, updatable store and retrieves the current version at query time. Update the source document, and the next query reflects it — no retraining, no redeployment of the model itself.

How RAG Works: Step-by-Step Pipeline

The RAG pipeline has six practical steps, and each one carries a concrete infrastructure cost.

1. Chunking. Source documents are cleaned and split into chunks — Big Data Boutique's 2026 explainer puts typical chunk sizes at 256 to 1024 tokens. Too small and you lose context; too large and irrelevant text dilutes the match.

2. Embedding. Each chunk is passed through an embedding model, which converts it into a vector — a list of numbers representing its meaning — as IBM describes.

3. Storage and indexing. Databricks' RAG workflow description has each chunk's embedding written to a vector store along with the original text and metadata, ready to be searched. NVIDIA's ingestion guidance gives a concrete sizing rule: roughly 1 GB of storage per 100,000 embeddings at 2048 dimensions in FP32 precision.

4. Retrieval. When a user asks a question, Microsoft Learn's description of the Azure Databricks approach has the system query a vector store, keyword search index, or SQL database to find supporting data.

5. Augmentation. That supporting data is combined with the user's original request, often via a template with extra formatting and instructions, per Microsoft Learn — this becomes the actual prompt sent to the model.

6. Generation. The LLM produces an answer grounded in the retrieved text, rather than purely from its training data.

Worked example: imagine a UK facilities-management firm with 500,000 document chunks covering maintenance manuals and compliance records. Using NVIDIA's ingestion rule of 1 GB per 100,000 embeddings, that collection needs roughly 5 GB of vector-index storage before any retrieval infrastructure is added. At enterprise scale — NVIDIA's example uses 10 million vectors — the retrieval layer alone needs around 100 GB of RAM, which is the point at which storage and memory planning, not model choice, becomes the real infrastructure decision.

RAG Architecture: Choosing Your Retrieval Backend

Retrieval doesn't have to mean a dedicated vector database. Google Cloud's RAG reference architecture documents two distinct designs: a fully managed, serverless vector search option built for large-scale applications, and an alternative that stores vector embeddings alongside operational data inside AlloyDB for PostgreSQL — a database-native approach. Choosing between them is a genuine architectural decision: a dedicated vector search service scales retrieval independently of your operational systems, while embedding vectors inside an existing database keeps data, access controls and backups in one place.

Beyond Google's two patterns, Big Data Boutique's 2026 architecture explainer lists the vector databases most commonly used in production RAG systems: Pinecone, Weaviate, Qdrant, OpenSearch, Elasticsearch, and Milvus. Each sits somewhere on a spectrum between pure vector search and a search engine with vector support added.

The practical UK question is less 'which product' and more 'which retrieval backend': pure vector similarity, keyword/full-text search, or SQL-based lookup against structured data — all three are legitimate options depending on whether your source material is unstructured documents, structured records, or a mix, per Microsoft Learn's description of the retrieval step.

The Infrastructure Reality: Storage and CPU, Not Giant GPUs

This is where RAG diverges hardest from the assumption that AI equals GPU spend. NVIDIA's enterprise RAG deployment guide states that storing vectors in CPU memory instead of GPU memory requires query-node scaling to somewhere between 100 GB and 500 GB of RAM, and that a collection of around 10 million vectors will need roughly 100 GB of RAM on its own.

NVIDIA's retrieval sizing guide gets more specific still: a query node built to serve 4 million vectors is sized at 16 vCPU and 40 GiB of memory, alongside a data node that uses just 0.25 of a GPU. The GPU footprint here is a fraction of one card, not a fleet. Across all of NVIDIA's sizing guidance, retrieval nodes are specified in vCPU and RAM, not GPU count — a direct, sourced counterpoint to the 'RAG needs big GPUs' assumption behind a lot of vendor pitches.

For a UK IT lead scoping a pilot, that reframes the budget conversation. The line items to plan for are storage capacity for the vector index, RAM for the query nodes that will hold and search those vectors, and enough vCPU to keep retrieval latency acceptable — see our detailed RAG infrastructure requirements breakdown. It's still worth checking generation-side needs separately, since the model itself has its own memory profile — an VRAM for LLMs guide covers that side of the equation.

RAG vs Fine-Tuning vs Prompt Engineering

RAG isn't the only way to make an LLM answer with organisation-specific knowledge — it competes with prompt engineering and fine-tuning, and each fits a different problem.

Prompt engineering means writing better instructions or pasting relevant context directly into each prompt. It needs no new infrastructure, but it's static: whatever you paste in is frozen at that moment, and it can't scale past whatever fits in the model's context window — worth understanding via our LLM context windows explainer, since chunk size and prompt assembly both live inside that limit.

Fine-tuning retrains the model itself on your data, changing its underlying behaviour, tone or domain reasoning. It's the heaviest option to build and maintain, and every data update means retraining again.

RAG sits between the two: it needs a genuine data-and-search infrastructure layer, but it keeps knowledge current without retraining, and it lets you show which document an answer came from — useful for both accuracy and audit.

Implementation Pitfalls, Evaluation and UK Compliance

The most common implementation mistakes are unglamorous. Chunking too coarsely or too finely against the 256–1024 token range that Big Data Boutique documents as typical is one: get it wrong and retrieval either misses the right passage or returns too much noise. Skipping the cleaning step that Databricks' workflow describes — deduplication, formatting fixes, removing boilerplate — before chunking is another; poor input produces irrelevant retrieval output. Sizing retrieval infrastructure after deployment rather than before is a third: NVIDIA's guidance shows RAM requirements scale directly with vector-collection size, so a pilot built on a small test set can hit a wall in RAM and query latency once rolled out across a full document estate.

Evaluating a RAG system in practice means checking two separate things: whether retrieval surfaces the right chunks for a given question, and whether the generated answer is actually grounded in what was retrieved rather than drifting back into the model's own assumptions. Test both before scaling vCPU and RAM budgets.

For UK organisations, RAG's external-knowledge-store design is also a genuine compliance advantage. Because source documents stay in a database or vector store you control, rather than inside model weights, you can apply UK GDPR-aligned access controls and retention policies at the retrieval layer, choose a UK- or EU-region deployment for the vector store, and produce an audit trail of exactly which document an answer was drawn from — something a fine-tuned model's opaque weights can't offer.

Decision Framework and What Comes Next

Use RAG when your knowledge changes often, when you need to cite sources for an answer, or when GPU budget is genuinely constrained and the sizing evidence above shows that's a viable trade. Reach for fine-tuning instead when the problem is the model's behaviour, tone or reasoning style rather than its factual knowledge. Stick with prompt engineering for small, one-off tasks where pasting context by hand is still faster than building a pipeline.

A sensible UK pilot path starts small: pick one internal knowledge base — an HR policy set or a product-support wiki — chunk and embed it, choose a retrieval backend that matches whether the source is unstructured documents or structured records, and measure retrieval latency against a realistic vCPU/RAM budget before any GPU procurement decision. An AI GPU calculator can help separate the generation-side hardware question from the retrieval-side one.

Looking past 2026, the architectural pattern documented by Google Cloud — retrieval as either a dedicated serverless search layer or embedded inside an operational database — points toward RAG becoming less of a bolt-on pipeline and more of a native database and search-engine capability, with retrieval increasingly triggered by autonomous agents deciding when a lookup is actually needed rather than firing on every query.

Sources

Every figure in this article traces to the sources below.

• •IBM — RAG definition and architectural pattern
• •IBM — two-phase build/runtime architecture and embedding step
• •Microsoft Learn — retrieval and augmentation step description
• •Databricks — RAG ingestion and vector store workflow
• •Google Cloud — serverless vector search and AlloyDB reference architectures
• •NVIDIA — enterprise RAG deployment guide, CPU-memory query-node sizing
• •NVIDIA — retrieval scaling and sizing summary (4M-vector example)
• •NVIDIA — ingestion scaling guide, storage-per-embedding rule
• •Big Data Boutique — RAG architecture and chunk-size guidance (2026)
• •Equinix — RAG's role in optimising AI inference accuracy