- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors study whether an executed agent action was approved before it ran, turning this into a per-action ground-truth label computed from the platform's process event stream by a state machine implementing the frozen Approval-Coverage Semantics v1. ControlBench has 1,800 simulated human-agent sessions with about 8,300 executed actions, generated with LLM operators by a Qwen2.5-7B primary simulator and a harder Mistral-7B replication. Open-weight LLM judges from 1.5B to 72B, plus two hosted API judges on small subsets, see the transcript, the process event stream, or both. The authors report an "observation gap", where judges given transcripts alone err in both directions, while the process event stream cuts mid/large format-competent judges' conditional false-approval rate (FAR) to 0.2-1.3% on the primary corpus. They also report a "semantic-execution gap": on same-transcript counterfactual pairs, judges given the event stream stay at chance on approval order and stale versions, but a deterministic replay reader always gets them right. The benchmark's hard split, a retrospective analysis of gating on event-grounded verdicts, tau-bench logs, and public SWE-agent/OpenHands runs round out the evaluation.
Critical strengths
- Executable ground truth, plus a counterfactual battery isolating rule application. Each per-action label is computed by a frozen specification, tested for conformance. Each same-transcript pair varies only in event order, binding, or version. This separates reading the record from applying the rule: binding is mostly readable for two judges, but order and version stay at chance, and in the stage-wise probe, Qwen3-4B finds the relevant events but misjudges precedence, while Llama-3.1-8B fails earlier at extraction. This result is the most informative in the paper, and the main reason ControlBench could serve to certify LLM readers.
- Broad controls around the main comparison. Taken together, cross-simulator replication, a family-disjoint judge, trained probes, learned sequence-model baselines, verdict-event deletion, plus natural-prevalence accuracy make it unlikely that the gain from events is an artifact of one simulator or judge.
Critical weaknesses
- The observation gap mostly measures if the answer-bearing event is in the prompt. Each label is defined over recorded approve and execute events, and the replay reader implements the semantics used to generate the labels; the paper itself calls decision-event sufficiency "partly definitional" and replay agreement "definitional rather than empirical". Removing the verdict-bearing events drops events-only accuracy to chance.
- The external anchor scores a different construct. On tau-bench the labels measure action-authorization coverage, where an exact user request counts as covered, and the transcript hides the executed call. So the high external false-approval rate partly reflects a judge not knowing which action it audits.
- Under the procedural variant, records help Qwen3-4B and gemma-2-9b but raise FAR for the Llama-3.1-8B and 72B judges, and the SWE-agent/OpenHands trajectories contain no verdict events. So the external evidence is mixed across judges and, for the coding agents, cannot test the rule at all.
- The title's human-control claim rests on LLM-simulated operators whose text and events are decoupled by construction.
- The claim about the semantic-execution gap is stated more generally than the judges tested. Judges are open-weight models up to an AWQ-quantized Qwen2.5-72B, plus two hosted API judges tested on 20 pairs per cue; neither a reasoning model nor a judge with code execution, the obvious remedy for rule execution, is evaluated. The order gap also depends on the prompt, since the post-hoc P3 condition closes it for gemma-2-9b.
- The headline FAR range is computed across a "format-competent" panel excluding Yi-1.5-34B. This model seems to meet the paper's own verdict-parse failure threshold, but its FAR rises with events.
Questions for authors
- Does a reasoning model, or a judge that can write and run a check over the event log, close the order gap on the 200-pair and 300-pair batteries?
- For the revoked-approval result cited in the abstract, how many uncovered revocation items is each judge shown in total? And the FAR of each?
- Was the "format-competent" panel fixed before the E1 results were in?
- A transcript with realistic platform timestamps, rather than only appended reply latency, could show how much of the observation gap is due to the text-only condition hiding information that an auditor would normally see.
- Prior work shows LLM judges failing to apply temporal rules over event traces where formal monitors succeed (Alamdari et al.; FormalJudge by Zhou et al.) and same-surface authorization pairs where an LLM judge fails yet a structural check succeeds ("Same Action, Different Justification"). What does the counterfactual battery show that this earlier work doesn't?
Follow-up on the Judge Panel and False Approval Rates
Dear Reviewer xUNr,
Thank you for flagging the judge selection and metric denominators. Table 4 now reports all seven judges, including Yi. Adding events improves overall accuracy for all seven judges and lowers the false approval rate for five. Table 8 now computes the prefix false approval rate over negative examples.
Do these changes address your concerns about how to interpret the main comparison?
Best regards,
The authors