- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies omission attacks where an agent causes harm by silently skipping a required action. It argues that monitors reading only diffs or transcripts cannot detect what those inputs do not contain. It introduces OmissionBench, with two parts: 94 synthetic, incident-anchored PR triples and 285 LinuxArena sysadmin trajectories. Diff-only LLM monitors score roughly 0.06–0.13 AUROC lower on omissions than on commissions. Transcript-monitor AUROC falls as attacker capability rises. A state-audit monitor, which receives simulated shell-check responses from a hidden oracle, reaches 0.952 AUROC. The paper concludes that monitors should audit post-execution state.
Critical strengths
The most interesting finding is in §3: transcript-monitor AUROC on the live-agent track drops from 0.955 to 0.758 as the attacker moves from Haiku to Opus. The authors read this as monitors flagging execution clumsiness. If it holds up, this matters for AI control, because it suggests action monitors may degrade exactly as agents become more capable. It is the one result that goes beyond a restatement of the paper’s premise, though it rests on few tiers and one task distribution.
Critical weaknesses
- Limited novelty and trivial core claim. The central thesis is that a monitor cannot detect information absent from its input. That is a direct consequence of the data-processing inequality and needs no benchmark to establish. The proposed method of checking post-conditions against the post-execution state, is standard practice in runtime verification, compliance auditing, and missing-check detection, all of which the paper itself cites (CRIX, RoleCast). Relabeling this as an agentic monitor contributes no new mechanism. Without a technically new method or a surprising empirical result on the headline claim, the contribution is mostly reframing.
- A large share of the paper disclaims its own results. The state-audit numbers come from simulated responses from a hidden oracle. State-oracle rows score AUROC 1.00 even for irrelevant predicates, which indicates label leakage. The human panel was simulated, the Cody result is a proxy, and the real-PR split does not reconcile. Table 1 and the scope paragraph exist mainly to narrow claims after the fact. The paper is also internally inconsistent: Limitations calls cross-family attackers untested, which section 3 contradicts.
Questions for authors
State monitors score AUROC 1.00 even with irrelevant predicates, which suggests the oracle’s output leaks the label. How is the 0.952 state-audit result not affected by the same leakage?