LLM Evaluation & Fine-tuning
This domain tests your ability to assess LLM output quality using systematic evaluation methods and to select and apply the right fine-tuning strategy for a given use case. It accounts for 17% of the exam, making it one of the largest single domains. Expect questions that require you to distinguish evaluation metrics by their strengths and failure modes, choose between prompting and fine-tuning given a scenario, and identify the correct platform, dataset format, or hyperparameter for a specific fine-tuning approach.
Evaluation Dimensions
The four core dimensions used to judge LLM outputs are distinct and frequently tested as distractors for one another.
| Dimension | Definition | Typical Failure It Catches |
|---|---|---|
| Faithfulness | Output claims are supported by the source/context provided | Hallucinated facts not in the retrieved context |
| Relevance | Output addresses the question actually asked | Correct but off-topic responses |
| Coherence | Output is logically structured and internally consistent | Contradictory or rambling text |
| Groundedness | Every factual claim can be traced to a specific source passage | Mixing real context with invented detail |
Faithfulness vs. Groundedness — the exam conflates these. Faithfulness is binary (supported/not supported by context). Groundedness requires attribution — each claim is traceable to a specific source passage. Groundedness is a stricter superset of faithfulness.
Common traps
- Treating "relevance" as equivalent to "correctness." A response can be highly relevant yet factually wrong.
- Confusing "coherence" with "fluency." Fluency is about grammar; coherence is about logical structure and flow.
- Assuming faithfulness implies accuracy — the source itself may be wrong; faithful means consistent with the given context, not with ground truth.
Automated Evals: ROUGE, BLEU, BERTScore
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures n-gram overlap between a generated summary and a reference summary. ROUGE-1 counts unigrams, ROUGE-2 counts bigrams, ROUGE-L uses longest common subsequence.
BLEU (Bilingual Evaluation Understudy) measures precision of n-gram overlap of a generated translation against one or more references, with a brevity penalty to discourage short outputs. Originally designed for machine translation.
BERTScore uses contextual embeddings from BERT to compute cosine similarity between generated and reference tokens, capturing semantic similarity that exact-match metrics miss.
| Metric | Orientation | Best Use Case | Key Limitation |
|---|---|---|---|
| ROUGE | Recall | Summarization | Sensitive to phrasing; penalizes valid paraphrases |
| BLEU | Precision | Translation | Requires multiple references for reliability |
| BERTScore | Semantic | Any generation task | Computationally heavier; scores not interpretable on absolute scale |
Shared limitation for generation tasks: All three require a reference answer. In open-ended generation (creative writing, conversational QA), there is no single correct reference, so all three metrics underperform. They also cannot measure faithfulness, factuality, or instruction-following.
Common traps
- Choosing BERTScore as a faithfulness metric — it measures semantic similarity to a reference, not whether a claim is supported by source context.
- Using BLEU for summarization — BLEU is a precision metric; ROUGE-recall is standard for summarization.
- Assuming high ROUGE/BLEU implies high quality — paraphrased correct answers still score low.
Unlock with a paid plan
11 more sections (about 2,574 words) continue below. Every paid plan unlocks all AI Intermediate study guides, the full question bank and mock exams, from $4.97/month.