GitHub GH-600: Perform evaluation, error analysis, and tuning Practice Questions

The root cause is that the memory store returns stale, outdated entries rather than current state. The targeted fix is to refine memory retrieval — for example, by weighting recent entries, filtering by recency, or ensuring session completion markers are written and used during retrieval. Option C (larger context window) addresses token capacity, not stale retrieval logic. Option B (disabling memory) is a drastic measure that would prevent the agent from maintaining any cross-session continuity. Option D (post-execution summaries) may help write better memory entries going forward but does not fix the retrieval logic that currently selects stale content.

Option D (policy-as-code compliance) measures safety by ensuring generated infrastructure passes organizational guardrails. Option E (changes match the intended state) measures correctness against the task specification. Option A (no production database deletion without human approval) is a critical safety operational constraint aligned with responsible agentic execution. Option F (API round-trips) is an efficiency metric, not a correctness or safety signal. Option B (commit message format) is a style/quality concern unrelated to correctness or safety. Option C (surfacing a diff) is a good UX/workflow practice but is a process step, not itself an evaluation signal for correctness or safety outcomes.

The development intent has two explicit goals: fix failing CI tests and preserve public API signatures. Option D directly measures whether the agent achieves the primary objective (resolving failing tests). Option B directly measures adherence to the key constraint (no public API signature changes). Option E (commit message quality) is a communication quality concern not stated in the intent. Option C (speed per test case) is a performance metric not mentioned as a design goal. Option A (adding new tests) goes beyond the stated scope and may even violate the constraint of minimal change.

Because the logs show the agent's plan never includes a step to inspect the /security directory, the failure originates in the agent's reasoning/planning phase — it does not recognize or prioritize security-relevant paths when constructing its plan. This is a reasoning error. Option B (tool misuse) would apply if the tool were called but with wrong parameters; the plan never reaches that step. Option D (context/environment issue) would apply if access were denied at runtime, but the plan omission occurs before any tool is invoked. Option C (memory corruption) is not a standard failure classification and is not supported by the evidence provided.

The diff artifact shows the exact file path and content that the write_file tool modified. If the diff shows the correct fix content applied to the wrong file path, this confirms the tool was invoked with the correct content but an incorrect file path — a tool invocation error. Option C (the agent's summary) is generated by the agent itself and may not accurately report which file was changed (especially if the agent also made an error in self-reporting). Option A (the system prompt) documents initial instructions and does not show what happened during execution. Option B (the evaluation scorecard) gives aggregate results but not the granular tool call details needed to distinguish error types.