AI Intermediate: LLM Evaluation & Fine-tuning Practice Questions

Malformed JSON in outputs most commonly traces back to inconsistent or malformed training examples — the model learns the noisy format it was trained on. Cleaning the dataset is the highest-leverage fix. Increasing epochs could worsen overfitting to bad patterns if the root cause is data quality. The OpenAI fine-tuning API's chat format fully supports structured outputs including JSON, so switching formats is unnecessary. Adding a system prompt at inference time may help marginally but does not address the root cause in the training data and is less reliable than fixing the data.

Context recall measures how much of the ground-truth answer's information is covered by the retrieved context passages, directly quantifying retrieval completeness. Faithfulness measures whether the generated answer is supported by the retrieved context — a generation-side metric, not a retrieval coverage metric. Answer relevance scores how well the generated answer addresses the user question, also a generation-side metric. Answer correctness compares the generated answer to the ground truth holistically and is not specific to retrieval coverage.

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures n-gram overlap between generated text and reference texts, making it a standard metric for summarization tasks. BERTScore uses contextual embeddings to compare semantic similarity, not n-gram overlap. RAGAS faithfulness measures whether RAG-generated answers are grounded in retrieved context, not reference overlap. Perplexity measures how well a language model predicts a sequence, not output quality against references.

Augmenting training data with more diverse contract types directly addresses the distribution mismatch — the model has not seen enough variety to generalize. Using a validation split and early stopping prevents the model from overfitting to the narrow training set, which is the likely cause of poor performance on novel structures. Increasing epochs beyond defaults encourages memorization of the training distribution, worsening generalization — the opposite of what is needed. The choice of fine-tuning platform (OpenAI vs. Vertex AI) does not determine generalization; neither platform has a proprietary generalization algorithm that compensates for insufficient data diversity. Adding a system prompt to a fine-tuned model does not teach it patterns it never learned during training; it may marginally help but does not address the root cause.

Concrete anchor examples for each score level reduce ambiguity by showing annotators exactly what a '2' vs. a '3' looks like in practice, directly improving agreement. Calibration sessions allow annotators to surface and resolve differing interpretations before independent work begins, aligning mental models. Explicit written criteria distinguishing score levels ensure annotators apply the same conceptual standards. A PhD requirement does not improve agreement; what matters is shared understanding of the rubric, not academic credentials. Assigning each item to only one annotator prevents computing inter-annotator agreement at all and removes redundancy for quality control — best practice is multiple annotators per item. Spell-checking model outputs may slightly reduce distraction but has negligible impact on inter-annotator agreement for relevance judgments.