AI Intermediate · 17% of the exam

LLM Evaluation & Fine-tuning: free practice questions

5 sample questions from our 281-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real DataCertPrep exam.

1. A team is setting up LLM-as-judge evaluations to score the quality of chatbot responses on a scale of 1–5. A senior researcher raises the concern of 'position bias.' What does position bias refer to in this context?

A. The judge LLM tends to give higher scores to shorter responses regardless of their quality
B. The judge LLM's score is influenced by whether the evaluated response appears first or second in a pairwise comparison, rather than purely on quality✓ Correct
C. The judge LLM systematically rates responses on topics outside its training distribution lower than warranted
D. The judge LLM inflates scores for responses that include confident, assertive language

Explanation

Position bias in LLM-as-judge refers to the tendency of a judge model to favor responses based on their position in a prompt (e.g., preferring the first option in a pairwise comparison) rather than their actual quality. This is a well-documented failure mode. Option A describes verbosity bias (or the inverse, brevity bias), which is a different known bias. Option C describes a domain coverage issue, not a position-related bias. Option D describes a confidence or style bias, not position bias.

2. A startup is building a legal clause extraction system. They have 800 labeled examples and a well-engineered few-shot prompt that achieves 78% F1. A senior engineer proposes fine-tuning a smaller open-source model instead. Which TWO factors most strongly support choosing fine-tuning over continuing to optimize the prompt in this scenario?

A. Fine-tuning will eliminate the need for any evaluation after deployment.
B. The task requires consistently structured outputs (e.g., JSON with specific fields), which fine-tuning enforces more reliably than prompting.✓ Correct
C. Fine-tuning a smaller model can reduce per-request inference cost significantly compared to running a large model with a lengthy few-shot prompt.✓ Correct
D. Fine-tuning guarantees higher accuracy than prompt engineering for any task with more than 500 examples.
E. Fine-tuning removes the need to maintain retrieval infrastructure.
F. Prompt engineering cannot work with fewer than 1,000 examples.

Explanation

Fine-tuning excels at enforcing consistent output formats (e.g., strict JSON schemas) because the model learns the exact structure from labeled examples, reducing parse errors common with prompting. Fine-tuning a smaller, cheaper model also lowers inference cost per call versus sending long few-shot prompts to a large API model — a concrete operational advantage. Fine-tuning does not eliminate evaluation needs; rigorous post-deployment monitoring is still required. Fine-tuning does not guarantee higher accuracy at any fixed dataset size — prompt engineering can sometimes match or exceed fine-tuning at low data volumes. Fine-tuning is unrelated to retrieval infrastructure. Prompt engineering has no hard minimum example count; zero-shot prompting works with no examples at all.

3. A team is building a golden evaluation dataset for a RAG-based HR policy assistant. They want each example to include a user question, the correct answer, and the specific document chunks that should be retrieved. Which of the following practices BEST improves the reliability of this dataset? (Select TWO)

A. Use a single expert annotator to ensure consistency across all examples
B. Have at least two independent annotators label each example and measure inter-annotator agreement✓ Correct
C. Include edge cases and adversarial questions that the system is likely to encounter in production✓ Correct
D. Generate all questions synthetically using the same LLM that will be evaluated, to ensure domain coverage
E. Restrict examples exclusively to questions with a single unambiguous answer found verbatim in one document chunk
F. Document clear annotation guidelines specifying what constitutes a complete and correct answer✓ Correct

Explanation

Having multiple independent annotators and measuring inter-annotator agreement catches labeling errors and ensures label quality. Including edge cases and adversarial questions makes the evaluation dataset more representative of real-world difficulty. Documenting clear annotation guidelines reduces subjectivity and ensures consistency across annotators. A single annotator introduces idiosyncratic bias and provides no quality check. Generating questions using the same LLM being evaluated introduces circular bias and may not cover gaps or failure modes. Restricting only to verbatim, single-chunk answers creates an unrealistically easy benchmark that misses multi-hop and ambiguous real-world questions.

4. A team has used the OpenAI fine-tuning API to fine-tune GPT-4o-mini for an internal legal document review task. After deployment, they notice the model occasionally ignores explicit instructions in the system prompt. Which of the following is the MOST likely cause and appropriate remediation?

A. The model was quantized during fine-tuning by the API, reducing its instruction-following capacity; the fix is to request non-quantized fine-tuning.
B. The fine-tuning examples may have included training completions that deviate from the system prompt instructions, teaching the model to ignore them; the fix is to audit and correct the training data to ensure completions consistently follow the system prompt.✓ Correct
C. OpenAI's fine-tuning API does not support system prompts; the model was trained without seeing them, causing inference-time confusion.
D. The model requires RLHF to learn instruction following; SFT via the fine-tuning API is inherently incapable of producing instruction-compliant behavior.

Explanation

When fine-tuning with the OpenAI API using chat-format data, the model learns from the (system, user, assistant) turn patterns in the training set. If training completions consistently deviate from or contradict the system prompt, the model learns to do so. Auditing training data to ensure all assistant completions faithfully follow the system prompt instructions is the correct fix. The OpenAI fine-tuning API does not expose quantization settings as a user-facing parameter, and quantization is not the described mechanism behind instruction-following degradation. The OpenAI fine-tuning API fully supports system prompts in the chat format — this is well-documented and commonly used. SFT is entirely capable of producing strong instruction-following behavior; it is the foundation of instruction-tuned models like InstructGPT, so claiming SFT is inherently incapable is incorrect.

5. An ML team is building annotation guidelines for a golden evaluation dataset for a medical Q&A system. Which three practices most improve the reliability and validity of the resulting dataset? (Select THREE)

A. Having a single domain expert annotate all examples to ensure consistency
B. Defining explicit, example-anchored rubrics for each quality dimension✓ Correct
C. Including inter-annotator agreement measurement (e.g., Cohen's kappa) as a quality gate✓ Correct
D. Maximizing dataset size by scraping unannotated web data without review
E. Seeding the dataset with adversarial and edge-case examples✓ Correct
F. Using the model being evaluated to pre-label examples before human review

Explanation

Explicit, example-anchored rubrics reduce ambiguity and help annotators apply criteria consistently. Measuring inter-annotator agreement (e.g., Cohen's kappa) quantifies label reliability and identifies where guidelines need clarification. Including adversarial and edge-case examples ensures the dataset probes failure modes rather than only easy cases. Using a single annotator eliminates the ability to measure agreement and introduces unchecked individual bias. Scraping unannotated web data without review introduces noise and unlabeled examples that do not constitute a golden dataset. Using the evaluated model to pre-label creates a circular dependency and contaminates the evaluation with the model's own errors.

276 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free