- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to evaluate compliance interventions for tool-using agents on τ-bench. It shows that, if task success remains unchanged, the gain in violation-free success is bounded by the baseline fraction of successes containing violations. It compares this ceiling with paired confidence intervals and an author-defined reporting threshold. The authors also evaluate a learned violation predictor across two domains, several model sizes, and two model families, using do-nothing and random-scoring controls. The intervention reduces attempted violations in some settings, but its benefit over random scoring at a similar blocking rate remains unclear, and it does not clearly improve violation-free success. The paper provides evaluation recommendations, code, and episode records.
Critical strengths
-
The evaluation includes useful controls, especially random scoring at a matched blocking rate and a do-nothing baseline (§§2, 5). These help separate the benefit of the learned predictor from simply blocking more actions or reducing agent activity.
-
The paper reports negative results and distinguishes attempted violations from writes accepted by the environment (§§5, 7). Together with the released episode records and analysis code, this makes the findings easier to audit and helps prevent overstating safety improvements.
Critical weaknesses
-
The claim that both the ceiling and statistical resolution can be determined from the baseline is not supported (§6, Fig. 4). The ceiling follows from Eq. (1), but the paired interval also depends on intervention outcomes. Indeed, the ideal remover’s 1.16 pp gain has an interval excluding zero (Appendix E); it fails only the additional spread threshold. The paper acknowledges this distinction, but the broader framing remains stronger than the evidence, and the statistical justification for this threshold is unclear.
-
The violation metric checks whether a write belongs to the gold action set, rather than whether it follows the policy in the current conversation (§2, Appendix F). A gold action executed without user confirmation can therefore count as violation-free. The audit of 60 flagged writes supports precision but does not measure missed violations. This limits the interpretation of both the ceiling and the reported safety improvements.
Questions for authors
-
How can the paired statistical resolution be estimated from baseline data alone, without intervention outcomes (§6, Fig. 4)? Please clarify how this claim relates to the ideal remover’s interval excluding zero in Appendix E.
-
What justifies using the largest cross-trial range as an additional reporting threshold (§2)? Which main conclusions would remain under the paired confidence intervals alone?
-
Does the policy audit cover gold-matching writes and episodes labelled as violation-free (§2, Appendix F)? In particular, how often does the metric miss actions executed without user confirmation, and could these cases materially change the estimated ceiling?
3 comments
Thank you for the revision and additional analyses. These address several of my concerns. The baseline-only estimate now has a stated assumption and an empirical check, and Table 11 supports the narrow ceiling conclusion for retail 3B. I also appreciate the sensitivity analysis for missing trajectories.
My remaining concern is mainly the broader methodological contribution. The spread rule is a reasonable conservative reporting choice, but failing this rule does not mean an effect cannot be statistically resolved. This distinction is clearer in Section 6, although the wording in Section 9 still seems too strong. The proxy label also leaves substantial uncertainty about contextual policy compliance beyond the 3B case.
I find the controls and negative results useful, but I am still not fully convinced that they support the proposed general evaluation criteria. I therefore maintain my rating.
Section 9 wording
Thank you for reading the revision and for the specific point on Section 9.
We agree. Failing the spread rule means that our conservative rule does not claim a difference. It does not mean that the difference cannot be resolved statistically. Section 6 states this, and Section 9 already says that the 3B gain is "unclaimed by our criterion even where a paired test resolves it". The last sentence of Section 9, "so smaller differences are unreadable", is too strong. In the final version it will read "so our rule does not claim smaller differences", in line with Section 6.
Response to Reviewer j8jv
Thank you for the review.
Resolution from baseline data, and the ideal remover (weakness 1, question 1). Fixed in the revision: the abstract, the introduction, Figure 4 and the conclusion now say that the baseline predicts the resolution. The ceiling follows from Eq. 1. The procedure for the resolution uses the within-task variance of the three baseline trials. Under a zero-effect pairing it gives 2.13 pp on retail 3B violation-free success, with an upper value of 3.91 pp without task pairing. On the stored records, the observed vf-pass half-width falls between each contrast's own two values in 36 of 41 retail Qwen contrasts (Appendix K). The ideal remover is a case built from the baseline. Its gain of 1.16 pp has the interval [0.29, 2.32], which excludes zero (Appendix E). The paper reports this and states that the registered bar, not the benchmark, withholds the claim (Section 6). The paired interval of the ideal remover excludes zero; the registered bar withholds the claim there because its two-arm spread threshold is 4.35 pp.
Spread threshold (weakness 1, question 2). PROTOCOL.md of 2026-08-06 says not to claim a difference smaller than the cross-trial spread (Section 2). Each cell is 3 trials of one task set under one fixed configuration, so the spread is the instrument's own noise with nothing changed. Run-level drift across the 4 baseline re-run pairs inside one serving configuration reaches 2.32 pp on the violation axis. Over the 11 re-run pairs of any arm inside one serving configuration it reaches 4.64 pp, and over all 25 stored pairs 4.93 pp. Every one of the 16 claimed contrasts exceeds all three. On the stored records, none of 25 pairs of re-runs of the same condition has a paired interval that excludes zero, and none meets the rule.
Conclusions under paired intervals alone. On the violation axis, 21 of the 49 contrasts have paired intervals that exclude zero, against 16 under the full rule. Among the 53 differences between the deltas of two arms, 19 do so, against 5 (Table 10; counts in package file discussion/expected/F6_unknown_violation_status.json,
tally_reported_plain_contrastsandtally_difference_of_deltas, assignment S0). At 3B the forecaster removes 16.38 pp (95% CI [-20.58, -12.32]), and the removal replicates across runs and retrainings. The matched-dose comparison shows no separation on either half of the rule (Section 5). Under paired intervals alone, 14 differences separate that fail the full rule (Table 10 lists them). Two examples: intercept-all removes 4.35 pp more than the four-card forecaster [-7.97, -0.58], and the random scorer at its default threshold removes 4.93 pp less than the forecaster pooled over runs 1 and 2 [+1.30, +8.55]. Section 5 reports both and shows that neither meets its bar. The ceiling of Section 6 and its two predicted resolutions are computed on the baseline arms.Gold-matching writes and missed violations (weakness 2, question 3). The violation label is a proxy: a write whose name and normalized arguments are not in the task's gold action set. The benchmark's native reward is separate; it checks the final database state and the required outputs. Appendix F samples writes outside the gold set. The model classified 59 of 60 as writes the policy forbids. Fixed in the revision: an AI model, Claude Opus 5.5, assigned these labels, and the paper now says so. For the ceiling, the 3B arm has 13 violation-free successes, and 12 of them contain no write at all, so no gold write can hide a violation there. The one remaining episode would raise the ceiling from 1.16 pp to 1.45 pp. That is below the baseline's own cross-trial spread, 2.61 pp. We did not measure how often the metric misses a gold-matching write (Appendix K).