Machine Learning Operations Engineer Associate · 14% of the exam

Implement generative AI quality assurance and observability: free practice questions

5 sample questions from our 33-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real Microsoft exam.

1. A cloud architect at Contoso is reviewing the production observability strategy for an Azure AI Foundry-hosted agent. They notice that the throughput metric (requests per minute) is well within the model's provisioned capacity, yet users are reporting long wait times. Which metric should the architect examine NEXT to diagnose the root cause?

  • A. Token consumption per request, to check if individual responses are consuming unusually large token counts and hitting per-request processing limits✓ Correct
  • B. The groundedness score trend, to determine whether low-quality responses are being regenerated
  • C. The number of active Foundry projects in the subscription
  • D. The fluency metric average, to correlate poor language quality with latency
Explanation

When throughput (requests/minute) is within capacity but latency is high, the next logical metric to examine is token consumption per request. Responses with very large token counts take significantly longer to generate, even when the total request rate is low—this is a classic 'low RPS but high time-to-complete' scenario. Option B (groundedness score) measures factual grounding, not speed; retries due to low quality would show up as increased throughput, not as silent latency. Option C (number of Foundry projects) is an administrative concern unrelated to per-request latency. Option D (fluency metric) measures grammatical quality and has no correlation with response time.

2. A data scientist at Contoso is preparing a test dataset to evaluate a retrieval-augmented generation (RAG) chatbot built in Azure AI Foundry. The dataset must include the fields required by the built-in groundedness evaluator. Which combination of fields is REQUIRED in the test dataset for the groundedness evaluator to function correctly?

  • A. query, response, context✓ Correct
  • B. query, response, ground_truth
  • C. response, ground_truth, context
  • D. query, context, ground_truth
Explanation

The built-in groundedness evaluator in Azure AI Foundry measures whether the model's response is supported by the retrieved context. It requires the 'query' (user input), 'response' (model output), and 'context' (retrieved documents or passages) fields. 'ground_truth' is used by other evaluators such as the similarity or F1 evaluators, not groundedness. Option B is incorrect because groundedness does not compare against a ground-truth answer—it checks whether the response is grounded in the provided context. Option C omits the query, which is needed to frame what is being asked. Option D omits the response, leaving nothing to evaluate.

3. Contoso's AI team has built a custom evaluator in Azure AI Foundry to score domain-specific accuracy of a legal document summarization model. After running a batch evaluation, they notice the custom evaluator scores are consistently higher than the built-in coherence scores for the same responses. What is the most likely explanation for this discrepancy?

  • A. The custom evaluator's prompt or scoring rubric is not calibrated to the same 1–5 scale as the built-in coherence evaluator, causing inflated results✓ Correct
  • B. The built-in coherence evaluator automatically penalizes domain-specific terminology, lowering its scores
  • C. Custom evaluators always produce higher scores than built-in evaluators because they use a different underlying model
  • D. The batch evaluation run used different test dataset rows for the custom and built-in evaluators
Explanation

The most likely cause is that the custom evaluator's scoring prompt or rubric is not properly calibrated—if the rubric is lenient or its scale differs from the standard 1–5 range used by built-in evaluators, scores will appear inflated. Option B is false: built-in coherence does not specifically penalize domain terminology; it evaluates logical structure. Option C is a misconception—custom evaluators do not inherently score higher; their output depends entirely on how they are designed. Option D is possible but unlikely in a properly configured Foundry batch run, where both evaluators process the same dataset; it also does not explain a consistent directional difference.

4. A principal engineer at Adatum is implementing production debugging for a complex Foundry-based agent that intermittently returns incorrect tool-call results. They have enabled OpenTelemetry tracing and are exporting spans to Azure Monitor. After reviewing traces, they see that the tool-call span completes successfully (HTTP 200), but the agent's final response is still wrong. Which TWO debugging approaches should the engineer pursue NEXT? (Select TWO)

  • A. Inspect the span attributes and events within the tool-call span to verify the actual payload returned by the tool, not just the HTTP status code✓ Correct
  • B. Increase the groundedness evaluator's threshold to 4.5 to force the agent to retry until it produces a grounded response
  • C. Enable verbose logging of the agent's prompt construction step to check whether the tool output is being correctly parsed and incorporated into the next prompt turn✓ Correct
  • D. Switch the model deployment from GPT-4o to GPT-3.5-turbo to reduce hallucination frequency
  • E. Disable the online evaluation flow temporarily to reduce overhead and see if latency improves
Explanation

When a tool-call span shows HTTP 200 but the final answer is wrong, the issue lies in the content of the tool response or how it is processed—not in connectivity. Option A (inspecting span attributes and events for the actual payload) reveals whether the tool returned semantically correct data despite a successful HTTP status—common issues include malformed JSON, empty results, or unexpected schema. Option C (enabling verbose logging of prompt construction) determines whether the tool output is being correctly parsed and injected into the subsequent prompt; incorrect parsing would cause the agent to reason from a garbled or missing context. Option B is incorrect—raising a groundedness threshold does not fix incorrect tool output processing; it only changes when evaluation flags a response. Option D (switching models) is a drastic change that doesn't address a tool-integration bug and may introduce regressions elsewhere. Option E (disabling the evaluation flow) removes observability and does not help diagnose the root cause of incorrect results.

5. A team at Northwind Health is configuring an automated evaluation pipeline in Azure AI Foundry. They want to assess whether their medical-information chatbot produces responses that read naturally and are grammatically correct, so that patients do not perceive output as machine-like. Which built-in evaluator should they prioritize for this goal?

  • A. Groundedness evaluator
  • B. Relevance evaluator
  • C. Fluency evaluator✓ Correct
  • D. Coherence evaluator
Explanation

Fluency evaluates the grammatical correctness, natural phrasing, and readability of the generated text—exactly what the team needs to ensure responses do not feel machine-like. Groundedness (option A) checks factual support from context documents, not readability. Relevance (option B) measures whether the response answers the question, not how naturally it is phrased. Coherence (option D) measures logical flow and consistency across sentences, which overlaps somewhat but is distinct from surface-level grammatical fluency. For patient-facing naturalness, Fluency is the primary metric.

28 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free