- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Popular coding agents now have autonomous modes where another model decides if the main model’s actions are safe or unsafe where the goal is to reduce the number of permissions requests that the human user needs to process and avoid the pitfalls of dangerously skip permissions mode. This paper aims to red-team these classifier models. To this end, the authors study various attack strategies and identify under what conditions they succeed. Finally, the authors discuss some possible mitigation that can improve the autonomous mode.
Critical strengths
- The topic is very timely. Assessing the safety of autonomous mode is critical as this mode is essential to make agents effective and reduce cognitive burden on humans.
- The authors attack coverage is broad including prompt injection into the monitoring classifiers, multi-agent attacks, coverage gaps and unintended mistakes. The coding agents craft these attacks adaptively and are much more effective than static attacks.
- Evals are also well thought out, the authors investigate loop holes in the auto mode setup to begin with and then pivot to on-policy and off-policy adaptive attacks. The findings seem important for example that compaction summaries are generally not monitored and are an attack surface.
- The enhancements to autonomous mode explored seem sensible such as monitoring all edits and agentic classifiers that can interact with the environment.
Critical weaknesses
- The paper touches upon fixes the issues found from red-teaming but does not give deeper treatment to it. The discussed solutions are sensible but monitoring all edits or using agentic monitors can make the whole system slow and expensive. A discussion on how to better operationalize the S1->S2->agent cascade will be useful to make them practical.
Questions for authors
Could we see similar effectiveness from automode ++ and automode strict if we use much smaller model but fine-tuned specifically for guardian tasks?
More broadly, how does model choices and harness choice affect the lessons from this work?