Databricks Generative AI Engineer Associate: Evaluation and monitoring Practice Questions

groundedness evaluates whether the generated answer is supported by the retrieved context — it requires only the answer and the context, not a reference answer. answer_relevance evaluates whether the answer addresses the user's question — it requires only the question and the answer, not a reference answer. answer_correctness compares the generated answer against a known ground-truth reference, so it requires reference answers. exact_match is a string comparison metric that requires a reference answer. ROUGE-L is an n-gram overlap metric that also requires a reference answer to compute overlap against.

Enabling inference tables captures every request/response along with metadata like token counts (which proxy cost) and latency timestamps—the raw data needed for both cost and quality tracking. Lakehouse Monitoring on the inference table then provides automated statistical analysis, dashboards, and alerts over those logged records, covering latency percentiles and enabling cost trend analysis. MLflow autologging is designed for training experiments, not production serving cost/quality monitoring. Storing model weights in a Unity Catalog volume is about artifact management, not cost tracking for inference. Delta Live Tables to an external warehouse adds unnecessary complexity when Lakehouse Monitoring is the native Databricks solution for this purpose.

Inference Tables automatically log every request payload and response from a Model Serving endpoint to a Delta table, providing a persistent record of production traffic for downstream analysis. MLflow Autologging captures training metrics and model artifacts during model development, not production endpoint traffic. Lakehouse Monitoring analyzes data stored in Delta tables (such as those populated by Inference Tables) for drift and quality, but it does not itself capture the request/response logs. Unity Catalog Lineage tracks data lineage across tables and assets but does not log endpoint traffic.

Enabling inference tables captures the full context (query, retrieved chunks, generated response) needed to evaluate groundedness at scale. Running Mosaic AI Agent Evaluation with the groundedness metric then uses an LLM judge to score whether each answer is supported by the retrieved context, directly quantifying the hallucination problem. Increasing top-k retrieval may improve recall but does not detect or quantify existing hallucination. Adding a prompt instruction may reduce hallucination going forward but does not measure the current extent of the problem. Switching to a larger embedding model addresses retrieval quality, not the detection/quantification of hallucination in existing responses.

groundedness measures whether the model's answer is supported by the retrieved context passages, making it the correct metric to detect fabrications or hallucinations relative to the source documents. answer_correctness compares the answer against a ground-truth reference, which requires labeled data and measures factual accuracy against a known answer rather than context support. chunk_relevance evaluates whether the retrieved chunks are relevant to the query, not whether the final answer is supported by those chunks. answer_relevance measures whether the answer addresses the user's question, not whether it is grounded in the retrieved evidence.