What is RAG? Retrieval-Augmented Generation, Defined

RAG (Retrieval-Augmented Generation) is a pattern where an LLM retrieves relevant context from an external knowledge store at inference time and grounds its response on that context, reducing hallucination and enabling fresh knowledge.

RAG (Retrieval-Augmented Generation) is a pattern where an LLM retrieves relevant context from an external knowledge store at inference time and grounds its response on that retrieved context. It is the dominant 2026 technique for giving LLMs access to fresh, domain-specific, or enterprise-private knowledge without retraining the model.

What RAG actually is

RAG has two phases:

• Ingestion (offline): split source documents into chunks, generate embeddings for each chunk using an embedding model (OpenAI Ada, Cohere Embed, Snowflake Arctic Embed, or open-source alternatives), store the chunks and embeddings in a vector database. This is the indexing step.
• Inference (online): embed the user query using the same embedding model, find the top-k most similar chunks via cosine similarity (or hybrid lexical + semantic search), inject the retrieved chunks into the LLM prompt as context, generate the response.

The shared idea: the model does not need to know the answer. It needs to retrieve the right context and ground the response on it.

What people confuse it with

• "RAG is the same as fine-tuning." No. Fine-tuning changes the model's weights; RAG keeps the model frozen and injects context at inference. They solve different problems.
• "RAG just means having a vector database." Mechanically partly true, but production RAG is a system with retrieval logic, chunking strategy, metadata filtering, reranking, and query rewriting. The vector database is one piece.
• "RAG eliminates hallucination." Reduces, does not eliminate. The model can still hallucinate facts not in the retrieved context, or misinterpret retrieved content.

When RAG matters

RAG matters when:

• The use case requires knowledge that is fresh, domain-specific, or private to the organization.
• The LLM's training-data knowledge is insufficient or stale.
• Source citation is a requirement (the user needs to know which document the answer came from).
• The corpus is large enough that injecting it all into the prompt is infeasible.

When RAG does not help

RAG does not help when:

• The use case is general-knowledge Q&A and the LLM's built-in knowledge is sufficient.
• The corpus is small enough to inject directly into the prompt without retrieval.
• The required reasoning is multi-hop across many documents and the retrieval step cannot surface all the needed chunks.

How Thinklytics works on RAG

We scope RAG engagements as data-architecture-first, not vector-database-first. Retrieval quality follows from data hygiene, chunking strategy, and metadata design. See LLM grounding data architecture.

Frequently asked questions

What is RAG in one sentence?

RAG (Retrieval-Augmented Generation) is a pattern where an LLM retrieves relevant context from an external knowledge store (typically a vector database) at inference time and grounds its response on that retrieved context, reducing hallucination and enabling the model to use knowledge that was not in its training data.

How does RAG work?

Three steps. (1) At ingestion time: split documents into chunks, generate embeddings for each chunk (using a model like OpenAI Ada or Cohere Embed), store the chunks and embeddings in a vector database. (2) At query time: embed the user query, find the top-k most similar chunks via cosine similarity. (3) Inject the retrieved chunks into the LLM prompt as context, generate the response.

What problem does RAG solve?

Three problems. LLM hallucination on facts the model does not know reliably. LLM knowledge staleness (the model's training data has a cutoff). Inability to cite specific source documents in the response. RAG addresses all three by grounding the response on retrieved source content.

Is RAG the same as fine-tuning?

No. Fine-tuning adjusts the model's weights to internalize new knowledge or behavior. RAG keeps the model frozen and injects knowledge at inference time. Fine-tuning is better for stylistic or behavioral changes; RAG is better for factual knowledge that changes frequently. Many production systems use both.

What is a vector database?

A database optimized for k-nearest-neighbor search over high-dimensional vectors (embeddings). Examples include Pinecone, Weaviate, Qdrant, Chroma, pgvector (Postgres extension), and the vector features inside Snowflake Cortex Search, Databricks Vector Search, OpenSearch, and Elastic.

What is the difference between RAG and semantic search?

Semantic search is the underlying retrieval mechanism (embedding-based similarity). RAG is the broader pattern that combines semantic search with LLM generation. You can have semantic search without RAG (just retrieval), but RAG without semantic search (or some equivalent retrieval) is just an LLM.

What are RAG's limitations?

Retrieval quality is the gating constraint. If the relevant chunk is not retrieved, the LLM cannot use it. Chunk boundaries can break context. Long documents may not fit in the model's context window even after retrieval. Multi-hop reasoning (combining facts from multiple chunks) is harder than single-document RAG. Production systems address these with hybrid search, reranking, query rewriting, and structured retrieval.

How does Thinklytics work on RAG?

We scope RAG engagements as data-architecture-first, not vector-database-first. The retrieval quality follows from data hygiene, chunking strategy, and metadata design. See LLM grounding data architecture.