Microsoft AI-300: Implement generative AI quality assurance and observability Practice Questions

Azure AI Foundry supports OpenTelemetry-based tracing that emits spans for each step of a multi-agent pipeline. These traces are sent to a connected Azure Monitor / Log Analytics workspace, where they can be queried by correlation ID to pinpoint exactly which step introduced latency for a specific request. The token usage dashboard shows aggregate token metrics, not per-step trace data. Application Insights real-time metrics are aggregate, not per-request traces. Azure Policy is for governance, not tracing.

Using separate Foundry projects per business unit provides natural isolation of token usage and cost data within the Foundry portal. Tagging deployments with cost center tags and filtering in Azure Cost Management is the standard FinOps practice for chargeback/showback across business units. Shared deployments with custom headers do not provide native per-unit cost breakdowns in Foundry or Cost Management. Azure Monitor workbooks can show metrics by deployment name but do not map to business-unit cost centers without tagging. Using a single project with RBAC restricts visibility but does not separate cost reporting.

Azure AI Foundry supports scheduled evaluation runs through its Evaluation SDK and portal, enabling recurring automated workflows that score datasets with built-in evaluators and persist results to the project for trend analysis. Azure Logic Apps and Azure Monitor alert rules are not designed for orchestrating AI evaluation pipelines. Azure ML pipelines can orchestrate ML workflows but are not the native mechanism for Foundry evaluation scheduling.

Groundedness measures whether the response is supported by the context, and relevance measures whether the response addresses the query — both require 'query', 'response', and 'context'. 'ground_truth' is needed for similarity-based metrics (like F1 or exact match) but not for groundedness or relevance evaluations. 'embedding' and 'temperature' are runtime parameters, not test dataset fields required by these evaluators.

Groundedness measures whether the model's generated response is factually anchored in the retrieved context. To compute this metric, the evaluator needs (1) the user query so it understands what was asked, and (2) the retrieved context passages, because groundedness scoring compares the model output against those passages to detect unsupported claims. Token usage and latency (option C) are operational metrics unrelated to groundedness scoring. A ground-truth answer (option D) is needed for relevance or similarity metrics but is not required for groundedness, which focuses solely on whether the response is supported by the retrieved context. The system prompt (option E) is part of the pipeline configuration, not a per-record evaluation field. Embedding vectors (option F) are internal retrieval artifacts and are not used as inputs to the groundedness evaluator.