Perform evaluation, error analysis, and tuning Study Guide — GH-600

Perform Evaluation, Error Analysis, and Tuning

This domain tests your ability to define what success looks like for agentic AI tasks, measure it using signals and tooling, diagnose failures from artifacts and traces, and act on those findings to improve agent behavior. It accounts for 18% of the exam, making it one of the largest weighted domains. Questions will ask you to select the right evaluation signal for a scenario, classify a failure type from a trace excerpt, or choose the correct tuning action given a described symptom.

Specifying Expected Outcomes and Operational Constraints for Agent Tasks

Expected outcomes are the explicit, measurable results an agent task must produce. Operational constraints bound how the agent may pursue those results — covering latency, cost, tool access, iteration limits, and safety thresholds.

Outcome Specification

Every agent task specification must include three elements:

Operational Constraints

Hard constraints halt execution if violated (e.g., maximum tool call budget, forbidden file paths). Soft constraints generate warnings but allow continuation (e.g., preferred response format). GitHub Copilot agent mode and GitHub Models both expose constraint surface through system prompts, agent instructions files (.github/copilot-instructions.md), and workflow-level environment guards.

Key constraint categories the exam tests:

• Iteration/step limits: Maximum number of agentic loop turns before forced termination. Default in GitHub Copilot agent mode is implementation-defined but questions assume you know that unbounded loops are a misconfiguration.
• Tool access scope: Which MCP servers or built-in tools an agent may invoke.
• Output schema: Structured output requirements that define correctness of form independent of content.
• Confidentiality constraints: Which context the agent may include in responses (e.g., no secrets, no internal URLs in public-facing output).

Common Traps

• Confusing goal conditions (what to achieve) with acceptance criteria (how to verify achievement). Both are required.
• Treating soft constraints as equivalent to hard constraints in failure analysis — the exam distinguishes them.
• Omitting boundary conditions from the spec and then being unable to classify an out-of-scope action as a failure.
• Assuming the .github/copilot-instructions.md file controls all constraints — it controls instruction-level behavior, not tool-level access scoping (that is MCP server configuration).

Identifying Qualitative and Quantitative Evaluation Signals

Evaluation signals are the evidence sources used to judge whether an agent task met its expected outcomes. The exam distinguishes between signals that produce numeric scores and those that require human or model judgment.

Key Evaluators in GitHub/Azure AI Tooling

The Azure AI Evaluation SDK (used alongside GitHub Models and GitHub Copilot pipelines) provides built-in evaluators:

• RelevanceEvaluator — model-as-judge; scores how relevant the response is to the query (1–5 scale).
• CoherenceEvaluator — model-as-judge; scores logical flow and consistency.
• GroundednessEvaluator — model-as-judge; scores whether claims are supported by retrieved context. Critical for RAG agents.
• FluencyEvaluator — model-as-judge; scores language quality.
• SimilarityEvaluator — requires a ground-truth answer; computes semantic similarity.
• F1ScoreEvaluator — token-overlap F1 against ground truth; fully deterministic.
• ViolenceEvaluator, HateEvaluator, SexualEvaluator, SelfHarmEvaluator — Azure AI Content Safety evaluators for safety signal generation.

Exam rule: If a question asks which evaluator requires a ground-truth answer, the answer is SimilarityEvaluator or F1ScoreEvaluator. Model-as-judge evaluators do not require ground truth.

Common Traps

• Selecting GroundednessEvaluator for a task that has no retrieved context — it is only meaningful when a context/document is supplied.
• Treating CI test results (binary pass/fail) as qualitative signals — they are quantitative/deterministic.
• Assuming model-as-judge scores are reproducible at exact values — they are stochastic and should be averaged over multiple runs.
• Confusing SimilarityEvaluator (semantic, model-based, needs ground truth) with F1ScoreEvaluator (token overlap, deterministic, needs ground truth).