- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- Defines instrumental monitor evasion and frames it as a control problem (“Instrumental AI Control”).
- Shows that many frontier models will attempt to evade monitoring under low/regular levels of task completion pressure.
- Contributes EvasionBench, a benchmark of regular, non-harmful tasks paired with task-specific policy monitors that prevent LLMs from completing the task. The native Claude Code and Codex hooks are used to improve the realism of the monitor being applied.
- Provides detailed analysis of interesting behaviours across the 10 evaluated models, e.g. GPT-6 Astra not evading the monitor but overrefusing under fake monitor-denial injections, and verbalised eval awareness in Claude Sonnet 5 resulting in it not evading the monitor.
Critical strengths
- This paper shows that under regular task completion pressure, many frontier models will take steps to work around monitors despite having been initially stopped. This is important for understanding the efficacy of monitoring systems applied to frontier models. The separate reporting of success and attempts is also very valuable as it shows the potential for what could have happened given different configurations of strength of the monitor policy.
- The paper contributes a benchmark of paired tasks and monitor policies, allowing a range of evaluations to be run that vary the way the monitor is applied (e.g. history, monitor model, feedback, situational prompting). Using the same hooks as Claude Code/Codex is good for realism, though unfortunately not sufficient to prevent Claude Sonnet from verbalising eval awareness.
Critical weaknesses
- Confidence intervals are not presented with the key results. These would help in understanding how much weight we should place on the results and if the relative ordering of the models’ propensity to work around the monitor is robust. Some of the results (e.g. those in Figure 4, which use one run per task) do not have many repeats and given the lack of CI included it is not clear to the reader if the conclusions drawn from the results are overclaimed. It would be helpful to see visually what degree of confidence we can place in these results, and have the claims updated accordingly.
- There are no ablations across varying levels of task pressure, or with the model explicitly told not to evade the monitors. These ablations would better support the claim of “monitor evasion that arises from ordinary task pressure”. In particular given that the tasks are benign the models may not understand that they should not be circumventing the the monitor. This is raised in limitations but it could have been strengthened with an ablation.
Questions for authors
Clarifications:
- Is there publicly available information stating that GPT-6 Astra was trained not to evade monitors? As the paper goes on, it becomes less clear that this is speculation.
- Can you position your work against PropensityBench (Sehwag et al., 2025)? From my understanding, it is similar in that it has tasks that cannot be completed through “safe” tool use.
3 comments
Thank you for your response and answering my questions, I appreciate the effort to address my concerns. Regarding (W1) would it be possible for you to report confidence intervals on the results you already have without needing to run any additional experiments? Overall, I am satisfied with the rebuttal and will keep my score as 3.
Thanks for the quick response and yes that is possible! We will report 95% bootstrap confidence intervals.
We thank the reviewer for recognizing the importance of studying evasion under task pressure, reporting attempts separately from successes, and the use of native agent hooks.