Operationalizing ML and Generative AI Solutions · 13% of the exam

Implement GenAI quality assurance and observability: free practice questions

5 sample questions from our 33-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real Microsoft exam.

1. A FinOps analyst at Lamna Healthcare wants to implement cost tracking for multiple Azure OpenAI model deployments consumed by different business units. Each business unit should see its own token consumption and estimated cost in Azure AI Foundry. Which two mechanisms should be configured to support this requirement? (Select TWO)

A. Use separate Azure AI Foundry projects per business unit and associate each project with its own Azure OpenAI deployment✓ Correct
B. Tag Azure OpenAI deployments with business-unit cost center tags and use Azure Cost Management to filter spend by tag✓ Correct
C. Configure a single shared deployment and use custom request headers to identify the business unit in each API call
D. Enable token consumption metrics in Azure Monitor and create workbook views filtered by deployment name
E. Set up a single Foundry hub with all deployments in one project and use role-based access control to restrict visibility

Explanation

Using separate Foundry projects per business unit provides natural isolation of token usage and cost data within the Foundry portal. Tagging deployments with cost center tags and filtering in Azure Cost Management is the standard FinOps practice for chargeback/showback across business units. Shared deployments with custom headers do not provide native per-unit cost breakdowns in Foundry or Cost Management. Azure Monitor workbooks can show metrics by deployment name but do not map to business-unit cost centers without tagging. Using a single project with RBAC restricts visibility but does not separate cost reporting.

2. A team at Northwind Health is configuring an automated evaluation pipeline in Azure AI Foundry. They want to assess whether their medical-information chatbot produces responses that read naturally and are grammatically correct, so that patients do not perceive output as machine-like. Which built-in evaluator should they prioritize for this goal?

A. Groundedness evaluator
B. Relevance evaluator
C. Fluency evaluator✓ Correct
D. Coherence evaluator

Explanation

Fluency evaluates the grammatical correctness, natural phrasing, and readability of the generated text—exactly what the team needs to ensure responses do not feel machine-like. Groundedness (option A) checks factual support from context documents, not readability. Relevance (option B) measures whether the response answers the question, not how naturally it is phrased. Coherence (option D) measures logical flow and consistency across sentences, which overlaps somewhat but is distinct from surface-level grammatical fluency. For patient-facing naturalness, Fluency is the primary metric.

3. An AI safety engineer at Fourth Coffee is configuring risk and safety evaluations in Azure AI Foundry for a public-facing beverage recommendation chatbot. During testing, the evaluator reports a high 'Indirect Attack' (prompt injection) risk score on several test cases. The engineer wants to understand what this score specifically indicates about the chatbot's behavior. What does a high Indirect Attack score indicate?

A. The chatbot's responses contain profane or sexually explicit language that violates the content policy
B. The model is susceptible to malicious instructions embedded in retrieved external content, such as documents or web pages, that cause it to deviate from its intended behavior✓ Correct
C. The model produces responses that directly contradict the user's query, indicating a grounding failure in the retrieval pipeline
D. The chatbot's system prompt can be extracted by users who craft adversarial queries, exposing confidential configuration

Explanation

An Indirect Attack (also called prompt injection via retrieved content) specifically measures whether the model can be manipulated by malicious instructions hidden within external data it retrieves and processes—such as a poisoned document, web page, or tool output—causing it to override its original instructions and perform unintended actions. This is distinct from a direct jailbreak attempt made by the user. Profane or explicit language (option A) is flagged by the Hate/Unfairness or Sexual content safety evaluators, not the Indirect Attack evaluator. Contradicting the user's query (option C) describes a groundedness or relevance failure, not a safety attack. System prompt extraction (option D) is associated with the direct jailbreak or prompt extraction attack surface, which is a separate evaluation category from Indirect Attack.

4. An AI engineer at Bellows College is setting up continuous monitoring for a deployed RAG application in Azure AI Foundry. He wants automated alerts when the average groundedness score for sampled production traffic drops below 3.0 on a 5-point scale. Which approach is correct within Azure AI Foundry's monitoring framework?

A. Configure an online evaluation monitor in Azure AI Foundry that samples production inference data, runs the groundedness evaluator, and triggers an Azure Monitor alert when the metric threshold is breached.✓ Correct
B. Set an Azure Cost Management budget alert with a custom metric filter for groundedness score.
C. Create an Azure Logic App that polls the Foundry evaluation API every hour and sends an email if groundedness drops.
D. Enable Azure Defender for AI and configure a custom detection rule for low groundedness scores.

Explanation

Azure AI Foundry's continuous monitoring feature supports online evaluation monitors that sample production traffic, run evaluators (including groundedness), compute metrics, and integrate with Azure Monitor to fire alerts when thresholds are breached. Azure Cost Management handles budget/cost alerts, not AI quality metrics. A Logic App polling approach is a manual workaround, not the native monitoring solution. Azure Defender for AI focuses on security threats, not AI quality metric thresholds.

5. An AI engineer at Lamna Healthcare is interpreting evaluation results from Azure AI Foundry for a clinical summarization model. The coherence score returned is consistently 2 out of 5, while fluency averages 4.2 out of 5. What does this combination of scores most likely indicate?

A. The model produces grammatically correct text but the summaries lack logical structure and flow between ideas.✓ Correct
B. The model is retrieving irrelevant context from the knowledge base, causing inaccurate summaries.
C. The model's outputs are grounded in source documents but do not address the user's original query.
D. The model is generating hallucinations that are syntactically correct but factually wrong.

Explanation

Fluency measures grammatical correctness and natural language quality (high at 4.2), while coherence measures the logical organization and flow of the response (low at 2). This combination specifically indicates well-formed sentences that lack structural coherence — common in models that generate each sentence independently. The other options describe groundedness/relevance issues or hallucination, which are distinct metrics not reflected by this score combination.

28 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free