- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
-
The paper introduces AgentMonBench, including SpecGAP (omitted requirements) and SilentSwap (implementation changes that violate requirements while passing tests) from DeNovoSWE, and FeedbackTrace (agent decisions preceding user pushback) from SWE-chat, to evaluate whether monitors identify consequential decisions requiring user verification. Human review achieves 92–94% annotation agreement, supporting dataset reliability
-
The authors propose EBG, an evidence graph that organizes code and execution histories around agent behaviors. Experiments show improved decision identification and evidence localization compared with direct evidence access and a RepoGraph baseline
-
The authors conduct failure analysis of EBG-equipped monitors and find that most failures occur even when relevant evidence is visible, suggesting that interpreting evidence remains a major challenge
Critical strengths
-
The paper studies a practically relevant problem of identifying consequential agent decisions that require user verification. The three task construction methods (SpecGAP, SilentSwap, FeedbackTrace) convenient ways to construct evaluation tasks from existing repositories and interaction histories. Independent human review supporting dataset reliability
-
EBG provides a useful representation that organizes code and execution histories around task-relevant behaviors. Experiments across eight models show improved decision identification compared to the baselines
Critical weaknesses
-
The practical benefit of adding EBG to existing agent harnesses remains insufficiently established. RepoGraph is a reasonable component-level baseline, but the five Codex case studies do not replace a systematic comparison with and without EBG in commonly used harnesses such as Codex or Claude Code, under matched review instructions and budgets
-
The human review does not fully establish whether the target decisions can be identified without privileged information. SpecGAP reviewers see the omitted requirements, while FeedbackTrace reviewers see the later user feedback, which may make the target issues easier to identify. A human evaluation using only the inputs available to the monitor would help clarify whether low scores reflect monitoring failures or ambiguity in the tasks
Questions for authors
As noted in the weaknesses, I’m curious about the results of comparing commonly used agent harnesses with and without EBG, and whether any human audit verifies that the target decisions can be identified using only monitor-visible inputs, without to omitted privileged information
4 comments
Response to Reviewer KWz7 (Part 2)
Q2. Can humans still identify these targets without access to the omitted requirements or subsequent feedback?
Corresponds to: Critical Weakness concerning privileged information, and the second Question for Authors.
A2: We completed a blinded human audit using only Monitor-visible inputs. For each benchmark, 50 instances were randomly sampled without replacement from the 100 instances, and three reviewers independently examined the same set of samples. The original construction-stage review in Appendix B.5 focused on annotation reliability, whereas the new blinded audit directly tests whether the targets can be identified from Monitor-visible materials.
Audit setup. The audit lasted seven days. None of the three reviewers participated in benchmark construction. AI assistance and discussion among reviewers were prohibited throughout, and each reviewer independently submitted judgments and supporting evidence. They received the same inputs as the Raw Monitor:
- SpecGAP: The task document with requirements removed and the Monitor-visible code repository.
- SilentSwap: The original task document and the code repository containing semantic substitutions.
- FeedbackTrace: The interaction trajectory before the target user feedback.
Construction-time information, including omitted requirements, subsequent target feedback, and gold annotations, was withheld from the reviewers.
Scoring protocol. We use the original scoring criteria in Appendices C.3–C.4. Semantic metrics are independently scored by GLM-5.2 and Qwen3.7-Max, and their scores are averaged. LF1, LS, and EHR use the original rule-based evaluators. Each reviewer's scores are averaged across the 50 instances, and Human average is then calculated by averaging across the three reviewers. For SpecGAP, precision, recall, and F1 are each calculated at the instance level before averaging.
Table B6. Human audit on SpecGAP (50 tasks per reviewer).
| Evaluator | Semantic precision | Semantic recall | Semantic F1 | Localization F1 |
|---|---|---|---|---|
| Human reviewer 1 | 0.86 | 0.74 | 0.796 | 0.75 |
| Human reviewer 2 | 0.84 | 0.7 | 0.764 | 0.73 |
| Human reviewer 3 | 0.82 | 0.72 | 0.767 | 0.69 |
| Human average | 0.84 | 0.72 | 0.776 | 0.723 |
Table B7. Human audit on SilentSwap (50 tasks per reviewer).
| Evaluator | Localization Correctness (LC) | Change Correctness (CC) | Localization Score (LS) |
|---|---|---|---|
| Human reviewer 1 | 0.78 | 0.7 | 0.8 |
| Human reviewer 2 | 0.74 | 0.72 | 0.78 |
| Human reviewer 3 | 0.7 | 0.66 | 0.76 |
| Human average | 0.74 | 0.693 | 0.78 |
Table B8. Human audit on FeedbackTrace (50 sessions per reviewer).
| Evaluator | Verification Point Alignment | Evidence Support Score | Evidence Hit Rate | Strict target recovery |
|---|---|---|---|---|
| Human reviewer 1 | 0.78 | 0.82 | 0.84 | 60% |
| Human reviewer 2 | 0.82 | 0.84 | 0.88 | 66% |
| Human reviewer 3 | 0.8 | 0.8 | 0.86 | 64% |
| Human average | 0.8 | 0.82 | 0.86 | 63.33% |
Strict target recovery requires VPA, ESS, and EHR to all equal 1 for the same instance. The recovery rate is the proportion of the 50 sessions satisfying all three conditions.
In all three subsets, the three independent reviewers identified reference targets and supporting evidence from Monitor-visible inputs, showing that these targets have a basis in the visible evidence. The audit tests whether targets can be discovered without the deleted requirements or later feedback, supporting the use of these annotations to evaluate monitoring.
Manuscript changes: We have summarized the blinded audit results in the Monitor-visible human audit paragraph of §3.4 (p. 5, Lines 242–245) and added Appendix B.5.1 Human Audit with Monitor-Visible Inputs (pp. 22–23, Lines 1146–1206). This section specifies random sampling, reviewer inputs, scoring, and the strict recovery criterion. Tables B6 (p. 22), B7 and B8 (p. 23) report the complete results for the three subsets.
Thanks for the thorough response from the authors. The response have largely addressed my concerns and I will maintain my positive scores towards this work.
Thank you for your careful review and constructive feedback. We are glad that our response has largely addressed your concerns, and we appreciate your positive assessment of our work. Your suggestions have helped us strengthen the evaluation and improve the clarity of the manuscript.
Response to Reviewer KWz7 (Part 1)
Thank you for suggesting a systematic comparison within an existing agent harness and a human audit restricted to monitor-visible inputs. The first tests EBG's gains in an existing execution framework; The second tests whether human reviewers can identify the reference targets using the same information available to the monitor.
Q1. Is there a systematic comparison with and without EBG in an existing agent harness?
Corresponds to: Critical Weakness concerning the insufficiency of five case studies as a substitute for a systematic comparison, and the first Question for Authors.
A1: We added a systematic comparison in the Codex harness with and without EBG, comprising 90 paired comparisons. Both configurations use GPT-5.6 Luna, the same 30 samples from each of SpecGAP, SilentSwap, and FeedbackTrace, and identical review instructions. Neither configuration imposes a fixed cumulative token budget per example. We report both task metrics and actual average token usage. Semantic metrics are scored using Qwen3.7-Max, while rule-based metrics retain their original definitions.
Section 5.2.4 of the revised manuscript summarizes the experimental setup, and Appendix G.3 and Table G1 report the complete results.
With the model, samples, and review instructions held fixed, EBG improves Codex's oversight quality across all three subsets, with gains on all nine task metrics. On SilentSwap and FeedbackTrace, EBG improves task scores while reducing token usage by 24.3% and 3.2%, respectively, improving both oversight quality and token efficiency. On SpecGAP, all three metrics improve with 10.4% more tokens. These 90 paired comparisons support using EBG to improve oversight in Codex and provide systematic evidence of the quality gains and token costs of this integration.
Manuscript changes: Section 5.2.4 Application in Codex Harness (p. 10, Lines 516–517) summarizes the systematic comparison. The model, identical review instructions, and absence of a fixed cumulative token budget are specified in Appendix G.3 Systematic Comparison with and without EBG (p. 55, Lines 2934–2941), with complete results in Table G1 (p. 55). The original five case studies are presented in Appendix G.5 (pp. 57–62).