GitHub Agentic AI Developer · 18% of the exam

Perform evaluation, error analysis, and tuning: free practice questions

5 sample questions from our 11-question bank for this domain — answers and explanations included. These are the same scenario-based style as the real GitHub exam.

1. Your team is evaluating a coding agent in GitHub Copilot Workspace. You want to align evaluation criteria with the original development intent. The agent was designed to autonomously fix failing CI tests without altering public API signatures. Which TWO of the following evaluation criteria are MOST aligned with that intent? (Select TWO.)

  • A. The agent resolves all failing test cases identified in the CI run✓ Correct
  • B. The agent generates detailed commit messages explaining each change
  • C. The agent does not modify any method signatures in files marked as public API✓ Correct
  • D. The agent produces fixes in under 30 seconds per test case
  • E. The agent adds new unit tests to improve overall code coverage
Explanation

The development intent has two explicit goals: fix failing CI tests and preserve public API signatures. Option A directly measures whether the agent achieves the primary objective (resolving failing tests). Option C directly measures adherence to the key constraint (no public API signature changes). Option B (commit message quality) is a communication quality concern not stated in the intent. Option D (speed per test case) is a performance metric not mentioned as a design goal. Option E (adding new tests) goes beyond the stated scope and may even violate the constraint of minimal change.

2. A team uses an automated scanning tool integrated into their GitHub Actions workflow to generate evaluation signals for their Copilot agent. After a run, the scanner flags that the agent's output contains a hardcoded connection string. In the context of evaluation, how should this signal PRIMARILY be used?

  • A. To retrain the underlying language model with the flagged output as a negative example
  • B. To inform a revision of the agent's instructions or constraints to explicitly prohibit credential hardcoding✓ Correct
  • C. To immediately revoke the agent's GitHub token and halt all future runs
  • D. To add the flagged output to the agent's long-term memory so it avoids the pattern in future sessions
Explanation

Evaluation signals from automated scanners are used to identify gaps between agent behavior and desired behavior, and the appropriate response is to tune the agent — most directly by revising instructions or adding constraints (e.g., 'never include credentials in output files'). Option A (retraining the model) is outside the scope of agent-level tuning and is not a standard GitHub Copilot workflow. Option C (revoking the token) is a security incident response, not an evaluation-driven tuning action. Option D (storing the bad output in memory) would propagate the undesired behavior rather than prevent it.

3. After reviewing evaluation results, your team finds that a GitHub Copilot agent frequently retrieves stale context from its memory store when handling long-running tasks that span multiple sessions. The retrieved context causes the agent to repeat work already completed in prior sessions. What is the MOST appropriate tuning action?

  • A. Increase the agent's maximum token context window to retain more information per session
  • B. Refine the memory retrieval strategy to prioritize recency and include session completion markers✓ Correct
  • C. Disable memory entirely so the agent relies only on the current session's context
  • D. Add a post-execution instruction telling the agent to summarize its work at the end of every response
Explanation

The root cause is that the memory store returns stale, outdated entries rather than current state. The targeted fix is to refine memory retrieval — for example, by weighting recent entries, filtering by recency, or ensuring session completion markers are written and used during retrieval. Option A (larger context window) addresses token capacity, not stale retrieval logic. Option C (disabling memory) is a drastic measure that would prevent the agent from maintaining any cross-session continuity. Option D (post-execution summaries) may help write better memory entries going forward but does not fix the retrieval logic that currently selects stale content.

4. You are analyzing a trace from a GitHub Copilot agent that was asked to refactor a Python module. The trace shows the agent called the `read_file` tool successfully, then called `write_file` with a payload that was 0 bytes, and then marked the task complete. The original file is now empty. Which root cause classification BEST fits this failure?

  • A. Context/environment issue — the file system was read-only and silently discarded the write
  • B. Tool misuse — the agent invoked `write_file` with an empty payload instead of the refactored content✓ Correct
  • C. Reasoning error — the agent incorrectly determined the task was complete without validating output
  • D. Memory issue — the agent lost the refactored content between planning and execution steps
Explanation

The trace clearly shows the agent called `write_file` with a 0-byte payload — it invoked the correct tool but provided incorrect (empty) content. This is tool misuse: the tool was used incorrectly (wrong argument), resulting in data loss. Option A is wrong because the write succeeded (the file is now empty, not unchanged). Option C (reasoning error) could be a secondary issue but does not describe the proximate cause visible in the trace. Option D (memory issue) is speculative and not supported by the trace evidence, which shows the agent did execute a write call.

5. Which of the following BEST describes an 'operational constraint' when specifying expected outcomes for an agentic task in GitHub Copilot?

  • A. A numeric threshold that defines when the agent is considered to have succeeded
  • B. A boundary or rule the agent must respect during execution, regardless of task outcome✓ Correct
  • C. A log entry that records every tool call made by the agent
  • D. A prompt template used to initialize the agent's system instructions
Explanation

An operational constraint is a boundary or rule the agent must observe during task execution — for example, 'never push directly to the main branch' or 'only read files within the repository scope.' It governs how the agent behaves, independent of whether it achieves the goal. Option A describes a success metric or threshold, not a constraint. Option C describes a logging artifact. Option D describes a system prompt component, which may contain constraints but is not itself the definition of an operational constraint.

6 more questions in this domain

Practice the full bank with instant grading, flashcards, and a timed mock exam.

Start practicing free