RAG & Vector Search
This domain tests your ability to design, build, and evaluate retrieval-augmented generation (RAG) pipelines that ground large language model (LLM) responses in external data. It covers the full pipeline from raw documents through chunking, embedding, vector storage, retrieval, reranking, augmented generation, and evaluation. This domain has an 18% practice allocation in the DataCertPrep assessment: 45 questions in a 250-question format. This is our own course assessment, not a vendor certification exam.
RAG Overview: Why LLMs Hallucinate and How Retrieval Grounds Responses
Hallucination occurs when an LLM generates text that is fluent but factually incorrect or fabricated. Root causes include:
- Knowledge cutoff: training data has a fixed date; models cannot know recent events.
- Out-of-scope data: proprietary, internal, or domain-specific documents were never in training.
- Parametric memory limits: facts are encoded imprecisely in weights; low-frequency facts are recalled poorly.
RAG (Retrieval-Augmented Generation) solves this by injecting relevant, retrieved text into the prompt at inference time. The model is grounded in non-parametric (retrieved) knowledge rather than relying solely on parametric (weight-encoded) memory.
High-level RAG pipeline:
The complete RAG pipeline from document ingestion to answer generation.
Key distinction the exam tests: RAG does not retrain or fine-tune the model. It changes what the model sees at inference time, not the model weights.
| Approach | Changes Weights? | Handles New Data After Training? | Cost |
|---|---|---|---|
| Fine-tuning | Yes | No (static after fine-tune) | High |
| RAG | No | Yes (update vector DB) | Low per update |
| Prompt engineering | No | Only if fits in context | None |
Common traps:
- Confusing RAG with fine-tuning — RAG is retrieval at inference, fine-tuning is training.
- Assuming RAG eliminates hallucination entirely — it reduces it but does not guarantee accuracy.
- Thinking RAG requires a small context window — modern RAG works with any context size but retrieval precision matters more as context grows.
Embeddings: What They Are and How Similarity Search Works
An embedding is a dense, fixed-length numerical vector that encodes the semantic meaning of text. Semantically similar texts produce vectors that are close together in high-dimensional space.
Embedding dimensions vary by model (e.g., 1536 for text-embedding-ada-002, 1536 or 3072 for text-embedding-3-large). Higher dimensions generally capture more nuance but cost more to store and search.
Cosine similarity is the standard metric for comparing text embeddings:
- Range: −1 to +1 (in practice, text embeddings stay between 0 and 1).
- A score of 1.0 means identical direction (semantically identical).
- A score of 0.0 means orthogonal (unrelated).
- Formula:
cos(θ) = (A · B) / (‖A‖ × ‖B‖)
Dot product and Euclidean distance (L2) are alternatives:
| Metric | When Used | Notes |
|---|---|---|
| Cosine similarity | Text embeddings (default) | Magnitude-invariant; most common |
| Dot product | When vectors are normalized | Equivalent to cosine on unit vectors |
| Euclidean (L2) | Image embeddings, some retrieval | Sensitive to vector magnitude |
How similarity search works: At query time, the user's query is embedded using the same model used to embed documents. The vector DB then finds the k nearest neighbors to the query vector — this is k-NN search (or approximate k-NN / ANN for scalability).
Common traps:
- Using a different embedding model for queries than for documents — this breaks the search entirely.
- Assuming higher cosine similarity always means the answer is more factually correct — it means more semantically similar, not necessarily more accurate.
- Confusing cosine similarity range: text embeddings produce non-negative values (0–1), not the full −1 to +1 range.
Sign in and enroll free to read the full guide
11 more sections (about 2,614 words) continue below. This course is free: enroll from the course page to unlock every guide.