Implement generative AI quality assurance and observability Study Guide — AI-300

Implement GenAI Quality Assurance and Observability

This domain covers how you evaluate, monitor, and debug generative AI applications and agents built on Azure AI Foundry. It accounts for 14% of the AI-300 exam, meaning roughly 7 questions. Questions test your ability to select the correct evaluator type, interpret metric scores, configure monitoring pipelines, and diagnose production issues using tracing and logging tools. You must know specific service names, metric definitions, score ranges, and when to use built-in versus custom evaluators.

Test Datasets and Data Mapping for Evaluation

Core Concepts

Evaluation in Azure AI Foundry requires a test dataset — a structured file (typically JSONL) containing input/output pairs that the evaluator uses to score model responses. The dataset must be mapped to the schema the evaluator expects.

Dataset columns must be mapped to evaluator inputs:

Datasets can be uploaded to Azure AI Foundry directly or referenced from Azure Blob Storage. The evaluation job reads each row, runs the evaluator, and writes scores back to a results dataset.

Data mapping is the explicit binding of a dataset column name to an evaluator's expected parameter name. If column names match exactly, mapping is automatic. Otherwise, you configure it manually in the Foundry UI or via the SDK parameter column_mapping.

Built-in Dataset Formats

• JSONL (JSON Lines): Each line is a self-contained JSON object. This is the preferred format.
• CSV: Supported but requires header row matching or manual mapping.
• Minimum recommended size: at least 10–20 rows for statistically meaningful evaluation; larger datasets (100+) for production benchmarking.

Common Traps

• Forgetting to include context when evaluating groundedness. Groundedness requires the retrieved context to be in the dataset — it cannot be inferred from the query alone.
• Assuming ground_truth is always required. It is only required for metrics like similarity, F1, and exact match — not for coherence or fluency.
• Uploading CSV with mismatched headers. The evaluator will fail or produce null scores if columns are not mapped.
• Confusing the evaluation dataset with training data. Test datasets must not overlap with fine-tuning data.

Worked Scenario

Requirement: You are evaluating a RAG application. Your JSONL dataset has columns user_question, model_answer, and source_passages. Which column mapping is required?

Reasoning: Map user_question → query, model_answer → response, source_passages → context. The evaluator for groundedness requires all three. Without context, groundedness cannot be computed.

AI Quality Metrics: Groundedness, Relevance, Coherence, Fluency

Bold Caption: AI quality metric evaluation flow in Azure AI Foundry

Metric Definitions and Score Ranges

All four primary AI quality metrics in Azure AI Foundry use a 1–5 integer scale, where 5 is best and 1 is worst. They are computed by an LLM-as-judge pattern — a separate AI model (typically GPT-4) evaluates the response.

Key Distinctions

• Groundedness vs. Relevance: Groundedness checks whether facts in the response are traceable to the provided context. Relevance checks whether the response answers the question. A response can be relevant but ungrounded (answers correctly but cites facts not in context), or grounded but less relevant (stays within context but misses the point).
• Coherence vs. Fluency: Coherence is about logical structure and flow of ideas. Fluency is purely about grammar and language quality. A fluent response can be incoherent.
• All four are reference-free except when paired with similarity metrics (no human-written ground truth needed for these four).

Common Traps

• Assuming these metrics return 0–1 scores. They use 1–5. Similarity (e.g., cosine similarity) uses 0–1, which is different.
• Thinking fluency requires context. Fluency only needs the response text.
• Confusing groundedness with accuracy. Groundedness measures support from provided context, not real-world factual correctness.
• Expecting integer output. The judge model returns a score that is parsed as an integer, but evaluation results may show floating-point averages across a dataset.