acceptodds
Under review as a conference paper at ICLR 2027

The Ceiling on Violation-Free Success Can Sit Below τ-bench's Resolution

Abstract

τ-bench grades an agent by the database state it ends in, so an agent that reaches the goal through a write the user never sanctioned still scores a success. Two numbers read off a baseline arm decide, before any compliance intervention is built, whether its gain on violation-free success could ever be claimed: the ceiling, pass − vfpass, the most a success-preserving remover could add; and the resolution, the 95% half-width of a task-paired violation-free-success delta under a task-clustered bootstrap. On our 3B retail arm the ceiling is 1.16 pp and the resolution is 1.9–5.5 pp, so no success-preserving remover could post a claimable gain there. The bar is set by our comparison rather than by the benchmark: a task-paired test resolves an ideal remover's gain on most of our baseline estimates, and fewer than half clear the registered spread bar. We then run the check on a real intervention, a 1.5B violation-forecasting world model deployed training-free in the planning loop over two domains, two agent families, three scales and 11 conditions. It removes violating episodes at 3B, on an arm that succeeds less often than a do-nothing agent, and again on the second agent family. But most of what it stops the environment would have rejected anyway, the removal sits on the dose–response line that three uninformed arms fix, and its premium over an uninformed scorer at matched dose is unresolved at 3B and does not replicate on the second family; a minority of the paired contrasts meet the registered spread criterion together with a paired-interval condition added later. We propose the target arm's ceiling and resolution, then a do-nothing anchor and an uninformed control at matched dose, as the entry bar for compliance interventions, and release the harness and records in the supplementary archive.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, tool use, τ-bench, benchmark resolution, compliance, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_j8jvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to evaluate compliance interventions for tool-using agents on τ-bench. It shows that, if task success remains unchanged, the gain in violation-free success is bounded by the baseline fraction of successes containing violations. It compares this ceiling with paired confidence intervals and an author-defined reporting threshold. The authors also evaluate a learned violation predictor across two domains, several model sizes, and two model families, using do-nothing and random-scoring controls. The intervention reduces attempted violations in some settings, but its benefit over random scoring at a similar blocking rate remains unclear, and it does not clearly improve violation-free success. The paper provides evaluation recommendations, code, and episode records.

Critical strengths
  1. The evaluation includes useful controls, especially random scoring at a matched blocking rate and a do-nothing baseline (§§2, 5). These help separate the benefit of the learned predictor from simply blocking more actions or reducing agent activity.

  2. The paper reports negative results and distinguishes attempted violations from writes accepted by the environment (§§5, 7). Together with the released episode records and analysis code, this makes the findings easier to audit and helps prevent overstating safety improvements.

Critical weaknesses
  1. The claim that both the ceiling and statistical resolution can be determined from the baseline is not supported (§6, Fig. 4). The ceiling follows from Eq. (1), but the paired interval also depends on intervention outcomes. Indeed, the ideal remover’s 1.16 pp gain has an interval excluding zero (Appendix E); it fails only the additional spread threshold. The paper acknowledges this distinction, but the broader framing remains stronger than the evidence, and the statistical justification for this threshold is unclear.

  2. The violation metric checks whether a write belongs to the gold action set, rather than whether it follows the policy in the current conversation (§2, Appendix F). A gold action executed without user confirmation can therefore count as violation-free. The audit of 60 flagged writes supports precision but does not measure missed violations. This limits the interpretation of both the ceiling and the reported safety improvements.

Questions for authors
  1. How can the paired statistical resolution be estimated from baseline data alone, without intervention outcomes (§6, Fig. 4)? Please clarify how this claim relates to the ideal remover’s interval excluding zero in Appendix E.

  2. What justifies using the largest cross-trial range as an additional reporting threshold (§2)? Which main conclusions would remain under the paired confidence intervals alone?

  3. Does the policy audit cover gold-matching writes and episodes labelled as violation-free (§2, Appendix F)? In particular, how often does the metric miss actions executed without user confirmation, and could these cases materially change the estimated ceiling?

3 comments
AuthorsOpenReview ↗

Response to Reviewer j8jv

Thank you for the review.

Resolution from baseline data, and the ideal remover (weakness 1, question 1). Fixed in the revision: the abstract, the introduction, Figure 4 and the conclusion now say that the baseline predicts the resolution. The ceiling follows from Eq. 1. The procedure for the resolution uses the within-task variance of the three baseline trials. Under a zero-effect pairing it gives 2.13 pp on retail 3B violation-free success, with an upper value of 3.91 pp without task pairing. On the stored records, the observed vf-pass half-width falls between each contrast's own two values in 36 of 41 retail Qwen contrasts (Appendix K). The ideal remover is a case built from the baseline. Its gain of 1.16 pp has the interval [0.29, 2.32], which excludes zero (Appendix E). The paper reports this and states that the registered bar, not the benchmark, withholds the claim (Section 6). The paired interval of the ideal remover excludes zero; the registered bar withholds the claim there because its two-arm spread threshold is 4.35 pp.

Spread threshold (weakness 1, question 2). PROTOCOL.md of 2026-08-06 says not to claim a difference smaller than the cross-trial spread (Section 2). Each cell is 3 trials of one task set under one fixed configuration, so the spread is the instrument's own noise with nothing changed. Run-level drift across the 4 baseline re-run pairs inside one serving configuration reaches 2.32 pp on the violation axis. Over the 11 re-run pairs of any arm inside one serving configuration it reaches 4.64 pp, and over all 25 stored pairs 4.93 pp. Every one of the 16 claimed contrasts exceeds all three. On the stored records, none of 25 pairs of re-runs of the same condition has a paired interval that excludes zero, and none meets the rule.

Conclusions under paired intervals alone. On the violation axis, 21 of the 49 contrasts have paired intervals that exclude zero, against 16 under the full rule. Among the 53 differences between the deltas of two arms, 19 do so, against 5 (Table 10; counts in package file discussion/expected/F6_unknown_violation_status.json, tally_reported_plain_contrasts and tally_difference_of_deltas, assignment S0). At 3B the forecaster removes 16.38 pp (95% CI [-20.58, -12.32]), and the removal replicates across runs and retrainings. The matched-dose comparison shows no separation on either half of the rule (Section 5). Under paired intervals alone, 14 differences separate that fail the full rule (Table 10 lists them). Two examples: intercept-all removes 4.35 pp more than the four-card forecaster [-7.97, -0.58], and the random scorer at its default threshold removes 4.93 pp less than the forecaster pooled over runs 1 and 2 [+1.30, +8.55]. Section 5 reports both and shows that neither meets its bar. The ceiling of Section 6 and its two predicted resolutions are computed on the baseline arms.

Gold-matching writes and missed violations (weakness 2, question 3). The violation label is a proxy: a write whose name and normalized arguments are not in the task's gold action set. The benchmark's native reward is separate; it checks the final database state and the required outputs. Appendix F samples writes outside the gold set. The model classified 59 of 60 as writes the policy forbids. Fixed in the revision: an AI model, Claude Opus 5.5, assigned these labels, and the paper now says so. For the ceiling, the 3B arm has 13 violation-free successes, and 12 of them contain no write at all, so no gold write can hide a violation there. The one remaining episode would raise the ceiling from 1.16 pp to 1.45 pp. That is below the baseline's own cross-trial spread, 2.61 pp. We did not measure how often the metric misses a gold-matching write (Appendix K).

Thank you for the revision and additional analyses. These address several of my concerns. The baseline-only estimate now has a stated assumption and an empirical check, and Table 11 supports the narrow ceiling conclusion for retail 3B. I also appreciate the sensitivity analysis for missing trajectories.

My remaining concern is mainly the broader methodological contribution. The spread rule is a reasonable conservative reporting choice, but failing this rule does not mean an effect cannot be statistically resolved. This distinction is clearer in Section 6, although the wording in Section 9 still seems too strong. The proxy label also leaves substantial uncertainty about contextual policy compliance beyond the 3B case.

I find the controls and negative results useful, but I am still not fully convinced that they support the proposed general evaluation criteria. I therefore maintain my rating.

AuthorsOpenReview ↗

Section 9 wording

Thank you for reading the revision and for the specific point on Section 9.

We agree. Failing the spread rule means that our conservative rule does not claim a difference. It does not mean that the difference cannot be resolved statistically. Section 6 states this, and Section 9 already says that the 3B gain is "unclaimed by our criterion even where a paired test resolves it". The last sentence of Section 9, "so smaller differences are unreadable", is too strong. In the final version it will read "so our rule does not claim smaller differences", in line with Section 6.

Reviewer_fCjqOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper addresses the issue in the τ-bench benchmark where a successful final state may mask procedural violations. The authors define a "violation free success rate" (vf-pass) and demonstrate that, while holding the overall task success rate constant, the upper bound for improvement is given by the difference between the baseline success rate and the vf-pass. Using a 1.5B-parameter violation predictor—complemented by pre-invocation intervention mechanisms such as resampling and clarification—the authors evaluate three Qwen based agents in retail and aviation domains, with additional experiments for Llama 8B in the retail domain. The evaluation includes comparisons against a "no-information" scorer, an approximate "dosage-matching" control, and structural ablations. The primary contributions of this work include analyzing the gap between violation reduction and violation-free success, establishing an upper bound for improvement under a fixed success rate, and proposing an evaluation framework for compliance interventions that incorporates comparative precision, "no-op" baselines, and dosage-matching controls.

Critical strengths

The paper proposes distinguishing between successful episodes, violation-containing episodes, and violation-free successful episodes. It further establishes an upper bound for the potential gain in the "violation-free success rate" (vf-pass) relative to the "success rate with violations" (pass − vf-pass), under the condition that the overall task success rate remains constant. This provides a quantifiable benchmark for scenarios where violations decrease without a corresponding improvement in the rate of successful, compliant task completion. The framework helps assess the remaining room for improvement in a given setup and makes a substantive contribution to research evaluation.

Critical weaknesses

The abstract and Figure 4 characterize "resolution" as a quantity determined a priori by the baseline; however, the variance of the difference between paired future outcomes generally also depends on the intervention variance and the covariance between the two arms. This provides a quantifiable benchmark for scenarios where violations decrease without a corresponding improvement in the rate of successful, compliant task completion. This weakens the basis for generalizing the empirical precision and thresholds presented in this paper to serve as "entry criteria" for other interventions.

Questions for authors

1.The results for episodes with recorded trajectories in Appendix H are insufficient to determine whether the original estimate is conservative. The authors should report the proportion and the stage of occurrence of anomalies under key experimental conditions, and demonstrate whether the main effects and comparisons between scorers remain consistent when different assumptions are applied regarding the unknown violation status. 2.Appendix F audits writes outside the gold action set, but does not assess whether gold-matching writes can nevertheless violate context dependent policy requirements. Please clarify whether authorization history affects violation labels and provide contextual audit examples, or explain why gold-matching writes cannot violate policy in the evaluated tasks. 3.The paper describes resolution as a quantity determined in advance from the baseline, yet it is defined using a paired difference whose precision generally also depends on intervention outcomes and the correlation between the two arms. Please specify an estimation procedure that uses only baseline data and its assumptions, and clarify how this claim relates to the ceiling under an unchanged task success rate and the additional spread criterion. 4.Section 4 and Fig 9 state that the original write operation is executed once the clarification budget is exhausted; however, Fig 10 depicting the same v0.2 process indicates a fallback to a read operation. Please clarify this discrepancy and explain how the eight events shown in the figures are categorized for the purpose of calculating the blocking rate. 5.In a final comparison using identical deployment configurations, the predictor and the uninformative control which relies on actual blocking yielded scoring rates of 55.1% and 57.7%. Please clarify the criteria used to determine approximate matches, and provide uncertainty estimates or a sensitivity analysis for these figures to assess whether the remaining discrepancy materially affects the effectiveness of each scoring method in reducing policy violations. 6.Table 4 is described as the full results matrix, yet the 3B +WM v0.2 run 3 listed in Table 5 has no corresponding data row in Table 4; please add the row or clarify the table’s coverage.

3 comments
AuthorsOpenReview ↗

Response to Reviewer fCjq (1/2)

Thank you for the review.

Resolution and the baseline (weakness, question 3). Fixed in the revision: the abstract, the introduction, Figure 4 and the conclusion now say that the baseline predicts the resolution. The ceiling needs only the baseline. The estimation procedure for the resolution also uses only baseline data. For each task, take the variance of the three baseline trials. Under a zero-effect pairing, with independent trials given the task, this gives the half-width of a task-paired delta. On retail 3B violation-free success it is 2.13 pp. Ignoring task pairing gives an upper value of 3.91 pp. On the stored records, each retail Qwen contrast has its own two baseline-only values, computed from its own baseline arm. The observed vf-pass half-width falls between them in 36 of 41 contrasts, below the lower value in 5 and above the upper value in none. On the violation axis the counts are 24, 17 and 0. The median ratio of observed to zero-effect value on vf-pass is 1.21 (Appendix K). The ceiling applies to a success-preserving remover (Eq. 1). The spread criterion is the second half of the claim rule (Section 2). The 1.16 pp ceiling lies below both baseline-only values and below the baseline's own cross-trial spread, 2.61 pp. The ceiling claim therefore holds under the baseline-only estimate as well as under the paired result. The entry bar uses the target arm's ceiling and its resolution estimate, both computed on that arm's own baseline.

Unknown violation status (question 1). Empty trajectories are 736 of 18,360 episodes (4.0%), from 0.0% to 9.0% per cell (package file discussion/expected/F6_unknown_violation_status.json, population). All 736 carry a harness error. By stage: 554 agent completion calls, 76 action parsing, 76 alternative-candidate parsing, 28 scorer calls, 1 environment response and 1 user simulator call (same file, population.by_stage_total). On the stored records, we assigned the unknown status in four ways: no empty episode violates, every empty episode violates, and two bounding assignments per contrast. Under every assignment the paired interval of the 3B headline contrast excludes zero. The points are -16.38, -15.22, -11.30 and -20.29 pp. The number of the 49 violation-axis contrasts that meet the rule is 16, 14, 12 and 20 under the four assignments. Of the 14 retail 3B contrasts that meet it with no empty episode violating, 11 meet it under all four (same file, robustness_reported_plain_contrasts). The matched-dose gap is +1.59 pp [-2.46, +5.65] and +0.58 pp [-3.48, +4.78] under the two uniform assignments, and +5.65 pp [+1.45, +10.00] and -3.48 pp [-7.54, +0.58] under the two bounds. Each source run gets one label, set by the sign of its net weight in the contrast, so a run shared by both sides of the difference is labelled once (Appendix K). Empty trajectories in the 4-GPU 3B runs, by stage:

arm (two runs, 690 episodes)emptyagent completion callaction parsingalternative-candidate parsing
baseline27 (3.91%)1890
forecaster35 (5.07%)15812
random scorer, tau=0.1328 (4.06%)1747
AuthorsOpenReview ↗

Response to Reviewer fCjq (2/2)

Gold-matching writes (question 2). The label uses the task's sanctioned action set. Authorization history does not enter it. Appendix F samples writes outside the gold set, which are the writes the count flags. For gold-matching writes, the 3B arm gives a direct bound. It has 13 violation-free successes, and 12 of them contain no write. If the one episode with a gold write were reclassified, the ceiling would be 1.45 pp. That is still below the baseline's own cross-trial spread, 2.61 pp, which the reclassification leaves unchanged (trial counts 6, 3 and 3 of 115). Appendix K gives the same bound for every baseline cell. That one episode (task 92, trial 2) is a context example: the user confirms the return in message 21, the first write is message 22, and the user asks for the second return in message 25 before the write in message 26. We also read one 14B baseline success (task 61, trial 1): the agent asks for confirmation in message 2 and issues a gold-matching write in message 6 with no user turn in between, and the label does not flag it. We did not measure how often this happens. The labels and trajectories ship with the records.

Figures 9 and 10 (question 4). Fixed in the revision: Section 4, Figure 9 and Figure 10 now give the decision tree that the wrapper runs. A flagged call that was already clarified, or that arrives after the question budget (4 per episode) is spent, executes the original write. All 131 budget-exhausted events across the arms have the write call in the trajectory. The 8 events of the 7B v0.2 arm stay in the denominator of 1,073 scored events. They count as not blocked.

Matched-dose criterion (question 5). The criterion is the flag rate, with a registered target of about 87% and a floor of 80% (PREREG_MX4). The measured flag rates are 86.4% (forecaster runs 2 and 3) and 87.5% (control). The measured blocked rates are 55.1% [51.4, 59.2] for the forecaster and 57.7% [53.6, 62.2] for the control. The difference is +2.65 pp [-3.11, +8.46] (task-clustered interval; package file discussion/expected/F8_dose_match.json, pair_55_1_vs_57_7). The raw matched-dose gap in violation removal is +1.59 pp [-2.46, +5.65]. At the fitted slope of 0.64 pp of removal per pp blocked, the blocked gap accounts for about 1.7 pp of removal, and the dose-corrected gap is +3.30 pp [-0.96, +7.19] (Table 8; a positive value means that the forecaster removes more). The contrast's bar is 9.57 pp.

Table 4 (question 6). Fixed in the revision: Table 4 now has the two missing rows, 3B +WM v0.2 run 3 and 3B random scorer at tau=0.13 run 2. The baseline re-runs are in Table 5, and the Llama cells are in Appendix K.

Thank you for the revision, but the baseline-only resolution estimate is validated only on retail Qwen, and the violation label still reflects deviation from gold actions rather than contextual compliance. I am raising my Informativeness and Soundness scores but keeping my overall rating.

Reviewer_pp4kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper examines limitations of compliance evaluation in τ-bench and proposes a ceiling on violation-free success improvements. It evaluates a violation-forecasting model across multiple settings, showing limited improvements over uninformed controls.

Critical strengths
  1. The paper identifies a meaningful limitation of existing compliance metrics.
  2. The experiments include multiple baselines and statistical analyses.
Critical weaknesses
  1. Limited novelty of the central contribution. The proposed ceiling follows directly from the definition of violation-free success. Moreover, the claimed resolution limitation partly depends on the authors' own evaluation criterion rather than an inherent limitation of τ-bench.

  2. Potential test-set contamination. The intervention threshold was adjusted after examining test-set logs, raising concerns about evaluation independence and potential overfitting.

  3. Insufficient evidence of model effectiveness. The trained forecaster shows no statistically conclusive advantage over matched-dose uninformed controls. Improvements in violation-free success are also unresolved.

  4. Questionable validity of violation labels. Violations are defined primarily through deviations from gold action sets, which may not fully capture all legitimate policy-compliant actions. The limited manual audit does not establish the reliability of these labels across the full benchmark.

  5. Limited generalizability. Improvements are concentrated in the 3B retail setting, with no convincing gains at larger model scales or in the held-out airline domain. This substantially limits the practical significance of the findings.

Questions for authors
  1. What is the main methodological novelty beyond the direct ceiling calculation, and how does the proposed resolution differ from the authors' own claim criterion?

  2. Can the authors provide results using a threshold selected exclusively on validation data and evaluated on an untouched test set?

  3. Can the authors demonstrate a statistically significant advantage of the trained forecaster over matched-dose uninformed controls, particularly on violation-free success?

  4. How do the authors ensure that valid alternative action sequences are not incorrectly classified as policy violations?

  5. What evidence supports the generalizability of the proposed approach beyond the 3B retail setting, particularly to stronger agents and unseen domains?

1 comment
AuthorsOpenReview ↗

Response to Reviewer pp4k

Thank you for the review.

Novelty and the claim criterion. The ceiling is simple by design: it is computed from the baseline before any intervention exists (Eq. 1, Section 6). Its value lies in what it is set against. On retail 3B the ceiling is 1.16 pp (4/345). The baseline's own cross-trial spread of violation-free success is 2.61 pp (3/115), above that ceiling, so the ceiling cannot clear the rule against this baseline. The ideal remover's two-arm threshold is 4.35 pp. The paper states that the bar is set by our comparison and not by the benchmark (Section 6, Figure 4). An ideal-remover test separates the two (Appendix E). The paired test resolves an ideal remover's gain on 6 of the 8 baseline estimates, and 3 clear the registered bar (Table 6; package file discussion/expected/F4_paired_only_reading.json, ideal_remover.counts).

The resolution and the claim rule differ as follows. The resolution is the interval half of the rule: the 95% half-width of a task-clustered paired delta. The rule adds a spread half. PROTOCOL.md of 2026-08-06 fixed the principle that a difference smaller than the cross-trial spread is not claimed (Section 2 gives the dates of the later parts). Beyond the ceiling, the paper contributes the matched-dose uninformed control and the dose-response line (Section 5), a per-event reading of the forecaster (Section 7), and the released harness and records.

Threshold and test logs. The paper discloses the choice of 0.35 in Section 4. The ceiling uses baseline arms only, so no threshold enters it. The predicted resolution is also computed from the baseline arm. The observed half-widths come from paired contrasts, and some of these have a forecaster arm at its own threshold; the 1.9 pp above is from a contrast whose forecaster runs at 0.35. We have no result with a threshold selected only on validation data. Every closed-loop forecaster result is a test-split run, and the dose-matched uninformed control is matched to the flag rate the forecaster reaches at 0.35, so it shares that choice (Sections 4 and 5). The forecaster is trained on the retail train split, and the evaluation tasks are the retail test split. The two are disjoint (Section 3).

Forecaster against matched-dose controls. The paper does not claim that the trained scorer beats the dose-matched control; the matched control is the paper's finding. Against the dose-matched random scorer (two four-card runs per arm, control minus forecaster), violating episodes differ by +1.59 pp [-2.46, +5.65] and violation-free success by +0.14 pp [-1.30, +1.45]. Both intervals include zero. At 3B the forecaster removes 16.38 pp of violating episodes (95% CI [-20.58, -12.32]). The removal replicates across runs and across two retrained forecasters (-12.75 pp and -15.36 pp, each claimable in its own right). The matched-dose control shows that the blocked-write dose tracks this removal (Figure 3, Section 5). This reading is the paper's input to the entry bar it proposes.

Violation-free success. Section 6 explains this in advance. A success-preserving remover can add at most the ceiling, 1.16 pp on retail 3B. The baseline's own cross-trial spread, 2.61 pp, is above that ceiling, so a success-preserving remover cannot clear the rule against this baseline.

Violation labels. The violation count is mechanical. A violation is a write outside the task's sanctioned action set, derived from the gold action sequence with no judge model (Section 2). To check that this rule tracks the policy, Appendix F reads a seeded sample of 60 such writes against the policy text. The model classified 59 of the 60 as writes the policy forbids and one as only leaving the gold path. That one write is the only case in the sample where a valid alternative action counts as a violation. Fixed in the revision: the labels in Appendix F come from an AI model, Claude Opus 5.5. Appendix F, Section 2 and the AI-use statement now say so.

Generalization. Table 2 reports the ceiling for all six baseline arms, across two domains and three scales, and the resolution per domain. The ideal-remover test also covers Llama-3.1-8B (Appendix E). The removal appears on a second agent family. On Llama-3.1-8B retail (one run per arm) the forecaster removes 23.19 pp of violating episodes (95% CI [-30.14, -16.23]) and clears the bar. The uninformed scorer at the same threshold removes 16.81 pp [-22.61, -11.01] and also clears it. The comparison between the two, the registered primary comparison, is -6.38 pp [-12.75, 0.00] and does not clear it. That family has no dose-matched run. Section 5 (RQ3) reports this leg, and Appendix K gives all four axes and the arm diagnostics. Section 7 and Table 3 give the per-event account of the forecaster at 7B and 14B.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Title change in the revision

Title change. The revised PDF is titled "The Ceiling on Violation-Free Success Can Sit Below the Bar a Claim Must Clear". The submitted title was "The Ceiling on Violation-Free Success Can Sit Below τ-bench's Resolution". The revised abstract states that the bar a claim must clear is set by the comparison rather than by the benchmark; the new title states the same.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.