Generative AI Engineer Associate · 12% of the exam

Evaluation and monitoring: free practice questions

5 sample questions from our 30-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real Databricks exam.

1. An ML team has set up Lakehouse Monitoring on the inference table of their production chatbot. They configured a `time_series` monitor profile. After two weeks, the dashboard shows a sudden spike in `p95_latency_ms` starting on a specific date. What is the MOST likely first investigative step?

  • A. Immediately roll back the LLM to the previous MLflow model version.
  • B. Correlate the spike date with any deployment changes, traffic volume increases, or external LLM API changes that occurred around that time.✓ Correct
  • C. Disable the inference table to reduce endpoint overhead.
  • D. Switch the monitor profile from `time_series` to `snapshot` to get a clearer picture.
Explanation

A sudden latency spike tied to a specific date is most effectively investigated by correlating the timing with known events such as model or config deployments, traffic surges, or external API provider changes — standard root cause analysis practice. Rolling back immediately without investigation is premature; the cause could be external (e.g., the LLM API provider), making a rollback ineffective. Disabling the inference table would eliminate the observability data needed for investigation and would not fix the underlying issue. Switching monitor profiles changes how data is aggregated but does not provide additional diagnostic information about the cause of the spike.

2. A team is using Mosaic AI Agent Evaluation and receives a low `chunk_relevance` score for many queries. What is the MOST likely root cause indicated by this metric?

  • A. The LLM is generating answers that contradict the retrieved documents
  • B. The retrieval step is returning documents that are not pertinent to the user's question✓ Correct
  • C. The prompt template is too long and exceeds the context window
  • D. The ground-truth answers in the evaluation set are outdated
Explanation

**chunk_relevance** measures whether the retrieved chunks are actually relevant to the question asked. A consistently low score means the **retrieval step** is failing to surface pertinent documents—pointing to issues in the embedding model, vector index, or search configuration. **Contradicting retrieved documents** would surface as a low `groundedness` score, not chunk_relevance. **Exceeding the context window** would typically cause truncation errors or degraded generation quality, not a low retrieval relevance score. **Outdated ground-truth answers** would affect `answer_correctness` scores, not the retrieval relevance metric.

3. A data science team has deployed a RAG-based customer support chatbot on Databricks. They want to systematically evaluate whether the chatbot's answers are grounded in the retrieved documents and not fabricated. Which Mosaic AI Agent Evaluation metric is MOST directly designed to detect this issue?

  • A. answer_correctness
  • B. groundedness✓ Correct
  • C. chunk_relevance
  • D. answer_similarity
Explanation

**groundedness** is the LLM-as-judge metric specifically designed to assess whether the generated answer is supported by the retrieved context, directly targeting hallucination or fabrication. **answer_correctness** measures whether the answer matches a ground-truth reference, which requires labeled data and doesn't focus on retrieval grounding. **chunk_relevance** evaluates whether the retrieved chunks are relevant to the question, not whether the answer stays within those chunks. **answer_similarity** is a semantic similarity score against a reference answer, not a grounding check.

4. A machine learning engineer is explaining the difference between offline evaluation and online monitoring to a new team member working on a Databricks-hosted RAG application. Which of the following statements BEST describes the key distinction?

  • A. Offline evaluation uses real production traffic, while online monitoring uses a fixed curated dataset.
  • B. Offline evaluation runs before or outside of production using a static eval set, while online monitoring continuously tracks quality on live traffic.✓ Correct
  • C. Offline evaluation can only measure latency metrics, while online monitoring can measure quality metrics.
  • D. Offline evaluation requires LLM-as-judge, while online monitoring relies solely on human feedback.
Explanation

Offline evaluation is performed using a curated, static dataset before or independently of production deployment, enabling controlled quality measurement. Online monitoring tracks metrics continuously on live production traffic over time. Option A reverses the definitions. Option C is incorrect because offline evaluation can measure quality metrics (e.g., groundedness, correctness) in addition to latency. Option D is incorrect because both offline and online evaluation can leverage LLM-as-judge, and online monitoring does not exclusively rely on human feedback.

5. A team's RAG chatbot is performing well on their offline eval set but receiving poor user ratings in production. After investigation, they find that the offline eval set questions were written by the data science team and don't reflect real user query patterns. Which THREE steps would most effectively close this evaluation gap going forward? (Select THREE)

  • A. Use inference tables to mine a representative sample of real production queries for inclusion in the eval set✓ Correct
  • B. Increase the size of the existing eval set by paraphrasing the existing data-science-authored questions
  • C. Incorporate user thumbs-down feedback signals to identify low-quality responses and add them to the eval set✓ Correct
  • D. Use Mosaic AI Agent Evaluation to score the newly curated production-traffic-based eval set and establish a new quality baseline✓ Correct
  • E. Disable Lakehouse Monitoring to avoid alert fatigue from the misaligned metrics
  • F. Switch to a larger LLM without updating the eval set
Explanation

Mining real production queries from inference tables (A) ensures the eval set reflects actual user intent distribution. Incorporating thumbs-down feedback (C) surfaces known failure cases, enriching the eval set with hard examples the model currently struggles with. Re-running Mosaic AI Agent Evaluation on the new eval set (D) establishes a new, valid quality baseline aligned with real usage. Paraphrasing existing team-authored questions (B) does not solve the distribution mismatch since they still originate from the same non-representative source. Disabling Lakehouse Monitoring (E) would eliminate the observability needed to detect ongoing issues. Switching to a larger LLM (F) without updating the eval set does not address the eval-production distribution gap and may not improve real user satisfaction.

25 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free