Microsoft AI-103: Plan and manage an Azure AI solution Practice Questions

Relevance measures how well the model's response addresses the user's actual question or intent. A low relevance score means the model is answering the wrong question, providing tangential information, or missing the point of the query. Hallucinated facts not found in a knowledge base describe a groundedness failure. Grammatically poor responses describe a fluency failure. Logical inconsistency across sentences describes a coherence failure.

Relevance measures how well the response addresses the user's actual query or need. High fluency means the text is grammatically well-formed, but a summary could be fluent yet miss the key points users care about—relevance would capture this gap. Coherence measures logical structure and flow, which could also be high for a fluent but unhelpful summary. Groundedness checks factual support from context, which would not directly explain why users find summaries unhelpful. Token usage per request is a cost metric, not a quality metric.

Indirect prompt injection occurs when malicious instructions are embedded within external content (such as uploaded documents) that the model processes, causing it to deviate from its system prompt. Direct prompt injection involves a user crafting the user-turn message itself to override instructions. Jailbreaking via role-play uses conversational manipulation (e.g., 'pretend you are an unrestricted AI') rather than external documents. Model inversion attack is a privacy attack aimed at reconstructing training data, unrelated to prompt manipulation.

The AI Red Teaming Agent in Azure AI Foundry automates adversarial simulation by generating and sending jailbreak attempts, prompt injections, and other attack patterns at scale, then reporting which inputs bypassed safety controls. This is the purpose-built tool for pre-production safety testing. The Evaluation SDK with groundedness metrics measures factual accuracy against source documents, not resilience to attacks. Azure Monitor alert rules on token consumption track usage and cost, not security vulnerabilities. Azure Policy assesses resource configuration compliance, not conversational attack resilience.

Scheduled Azure AI Foundry evaluation runs track quality metrics (groundedness, relevance, coherence, fluency) over time and can alert when scores degrade below baseline—addressing requirement 1. Azure Monitor metric alerts on token consumption metrics allow setting per-request or aggregate cost thresholds and triggering alerts—addressing requirement 2. Azure Application Insights with content filter logging captures which requests were flagged or blocked by content safety filters, allowing targeted investigation of policy violations—addressing requirement 3. Azure Advisor provides general optimization recommendations but does not monitor response quality or content policy violations. Azure Policy enforces resource governance (like tagging) and does not monitor AI output quality or costs at the request level. The fine-tuning dashboard tracks model training metrics, unrelated to production monitoring.