Evaluation and monitoring Study Guide — Generative AI Engineer Associate

Evaluation and Monitoring

This domain tests your ability to design, implement, and interpret evaluation and monitoring pipelines for GenAI applications built on Databricks. It covers how to measure quality before deployment (offline evaluation), how to track quality and cost after deployment (online monitoring), and how to close the loop between production signals and model improvement. The domain represents 12% of the exam, making it one of the mid-weight areas. Expect 6–9 questions spanning tooling specifics (Mosaic AI Agent Evaluation, Inference Tables, Lakehouse Monitoring), metric definitions (LLM-as-judge, retrieval quality), and architectural decisions about when to use which approach.

Mosaic AI Agent Evaluation and LLM-as-Judge Metrics

Core Concepts

Mosaic AI Agent Evaluation is the Databricks-native framework for evaluating RAG chains, agents, and LLM endpoints. It is invoked via mlflow.evaluate() with model_type="databricks-agent". The framework is built on top of MLflow and stores results as MLflow runs, meaning evaluation results are versioned, comparable, and linked to the model registry.

LLM-as-judge is the evaluation paradigm where a powerful LLM (the "judge") scores model outputs on defined quality dimensions. Databricks uses GPT-4-class models (specifically, Databricks-hosted or Azure OpenAI endpoints) as judges by default. The judge reads the question, the ground-truth answer (if provided), the retrieved context, and the model's response, then returns a numeric score and a written rationale.

Built-in LLM-as-Judge Metrics

Scoring scale: All built-in metrics return scores on a 1–5 scale by default, plus a pass/fail binary when a threshold is configured.

Evaluation Input Formats

mlflow.evaluate() for agents accepts an evaluation set as a Pandas DataFrame or Delta table with these columns:

• request (required): the user question
• response (optional): pre-computed model output; if absent, MLflow calls the model
• expected_response (optional): ground truth answer; required for answer_correctness
• retrieved_context (optional): list of retrieved chunks; required for answer_groundedness and context_relevance
• trace (optional): MLflow trace object for agent step-level evaluation

Custom LLM-as-Judge Metrics

You can define custom judges using mlflow.metrics.genai.make_genai_metric(). Key parameters:

• name: metric name
• definition: natural language description of what the metric measures
• grading_prompt: the prompt template the judge uses
• examples: few-shot examples for the judge
• model: the judge model URI (e.g., "endpoints:/databricks-dbrx-instruct")
• aggregations: how to aggregate per-row scores (e.g., ["mean", "variance"])

Common Traps

• Confusing answer_groundedness with answer_correctness: groundedness checks if the answer follows from the retrieved context (not ground truth). Correctness checks against ground truth. A grounded answer can still be wrong if the context was wrong.
• Assuming ground truth is always required: answer_relevance, answer_groundedness, context_relevance, and safety do NOT need ground truth labels.
• Forgetting that mlflow.evaluate() can call the model: if you pass a model URI instead of pre-computed responses, Databricks will invoke the model during evaluation.
• Judge model cost: the judge LLM is called once per metric per row. Large eval sets can be expensive; the exam may test whether you understand this cost implication.

Worked Scenario

Requirement: You have a RAG chatbot and want to detect when it fabricates information not in the retrieved documents. You have no ground truth answers.

Reasoning: Fabrication detection maps to answer_groundedness. It compares the response to the retrieved context, not to ground truth. You do NOT need expected_response in your eval set, but you DO need retrieved_context. The correct metric is answer_groundedness, not answer_correctness.