acceptodds
Under review as a conference paper at ICLR 2027

Before Commit: Purpose-Aware Rehearsal for Tool-Using Agents

Abstract

Tool-using agents can produce actions that are syntactically valid, policy-compliant, and executable, yet persistently alter external state without advancing the user's current purpose. Existing safeguards can improve how agents plan, reason, and recover, but they do not necessarily verify the final reversible decision before an external write is committed. We introduce Purpose-Aware Rehearsal (PA-Rehearsal), a training-free, candidate-bound interface that intervenes at this pre-commit boundary. For each proposed write, PA-Rehearsal binds the exact candidate to the latest user purpose and observable evidence, and uses an LLM to rehearse the action's likely effect and the goal it would serve. A deterministic controller retains final release authority by enforcing mandatory evidence, resolving conflicts between rules and model judgments, and verifying candidate identity. The interface leaves read-only exploration unchanged, rechecks every revised candidate, and can be layered onto execution-time or post-execution assistance. In matched three-run evaluations on -bench Airline and Retail, PA-Rehearsal attains the highest pass@1 point estimates among the compared methods. Notably, it outperforms a prompt-only boundary checklist by 20.0 percentage points on Airline. On WebShop, a benchmark for grounded web interaction, adding pre-purchase rehearsal to a fixed trajectory-regulation scaffold raises pass@1 from 44.5% to 50.0%. Together, these results show that purpose-aware checking at commitment boundaries can improve reliable task completion, positioning pre-commit rehearsal as a distinct, composable control primitive for tool-using agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Tool-Using Agents, Purpose Alignment, Pre-Execution Control, Runtime Oversight

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_HeZ2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes PA-Rehearsal, which checks state-changing actions before a tool-using agent commits them. It binds the candidate action to the current user purpose, observable evidence, and authorization, then combines LLM-based semantic assessment with domain-specific rules to determine whether the action should be released. Rejected actions can be revised and checked again. Experiments cover τ-bench Retail, Airline, and WebShop and report improvements in Pass@1. However, the incremental contribution, implementation details, and attribution of these gains remain insufficiently established, and some conclusions exceed the experimental evidence. I therefore recommend rejection.

Critical strengths
  1. The problem is practically relevant. A valid tool call does not necessarily serve the user’s current purpose, so checking actions before committing external writes has practical value.

  2. The paper includes three repeated evaluations, component ablations, and selected paired confidence intervals. The results demonstrate improvements in task success.

  3. The paper reports token costs and presents a regression case, helping readers understand the method’s overhead and limitations.

Critical weaknesses
  1. The incremental contribution over existing methods is unclear. The core mechanism combines LLM-based semantic assessment, domain-specific rules, and recovery after rejection. Although the paper discusses AgentSpec, ToolSafe, and ShieldAgent, it does not provide direct experimental comparisons or sufficiently establish the distinctive contribution of purpose-aware assessment over conventional pre-execution constraint checking.
  2. Important implementation details are missing. The provided PDF does not include complete judge prompts, output schemas, the mandatory-field registry, or domain-specific hard-rule specifications. Page 4 mentions a recovery budget but does not specify its value or the behavior when it is exhausted. The task identifiers and selection procedure for the fixed evaluation subsets are also unspecified, limiting reproducibility and the assessment of generalization.
  3. The attribution of performance gains lacks sufficient controls. Page 9 acknowledges that the WebShop comparison is not compute-matched, while the conclusion claims that “matched-budget comparisons” establish that the gains are not due to additional retries. The separate resource profile shows approximately 2.47 times the token usage, with actor continuation accounting for roughly 87% of the additional tokens. The current results therefore do not adequately disentangle the contribution of the proposed checking mechanism from additional reasoning and exploration.
  4. Some conclusions are overstated. In Table 1, Airline success across all three runs is 30%, below Life-Harness’s 35%, while Retail success in at least one of three runs is 75%, below EBP’s 82.5%. The highest Pass@1 point estimate does not establish superiority across all reliability metrics. Moreover, without independent action-level correctness labels, the evidence is insufficient to “rigorously validate” the necessity of combining the components.
Questions for authors
  1. Please clarify the methodological differences from AgentSpec, ToolSafe, and ShieldAgent. Could direct comparisons be conducted using the same judge model and compute budget?
  2. Please provide the complete prompts, output schemas, domain-specific rules, recovery budget, and behavior upon budget exhaustion. Please also provide the evaluation task identifiers, subset selection procedure, and rule-development process.
  3. Please resolve the inconsistency regarding compute-budget matching on page 9 and add compute-matched controls to examine whether the gains primarily arise from additional reasoning and continued exploration.
  4. Could interventions be independently annotated to report false approvals, false rejections, and the causes of regression cases? Please also restrict the reliability claims to the metrics supported by the results.
1 comment

Thank you for the direct questions. We address your concerns below.

1: Relation to existing guards and direct control

MethodShared idea with PA-RehearsalKey difference
AgentSpecRuntime constraints over agent actionsEnforces authored specifications; PA also resolves evolving user purpose and tests the exact write’s prospective effect using candidate-time evidence
ToolSafeChecks tool calls before executionPrimarily tool/policy safety; PA targets valid-but-purpose-misaligned writes and binds evidence, counterevidence, and recovery to the candidate
ShieldAgentUses a runtime shield to prevent unsafe actionsShields policy/safety violations; PA separates semantic purpose discovery from deterministic evidence enforcement and proposal-bound recovery

We cite these works to position pre-execution guarding, rather than reporting unmatched reimplementations whose rule authoring, threat models, and stacks differ. Instead, our same-stack control G fixes the Qwen3-8B actor, GLM-5.2 Judge, write trigger, parser, PA controller, evidence projection, and one-retry cap, changing only the Judge instruction from purpose-aware effect comparison to generic validity/policy/current-request checking.

DomainG / Full Avg.G / Full Pass@3G / Full Pass^3
Airline38.33 / 51.6750.00 / 70.0025.00 / 30.00
Retail49.17 / 59.1777.50 / 75.0022.50 / 42.50

Avg. is the mean of all three seeded runs (formerly Pass@1), not a single-seed result. We use Avg. as the primary aggregate while separately reporting at-least-one-of-three and all-three reliability.

2: Implementation, test sets, and action-level audit

The operative contract is: audit writes only; latest explicit user revision wins; compare the candidate’s expected effect with purpose/scope/constraints/policy; return four violation flags plus verifiability, registered field, evidence, counterevidence, precondition, rationale, and correction; enforce rejection only with registered target-bound evidence and no valid counterevidence; allow one correction, then block/clarify except confirmation bound to the held candidate.

Airline uses all 20 official test tasks × 3 seeds, and Retail all 40 × 3. WebShop uses the same frozen 200-task Life-Harness roster and configuration.

Regarding action-level correctness, false rejections/approvals, and regression causes, we are currently finalizing and cross-checking our blinded candidate-time audit (separating Judge vs. Controller decisions and examining discordant/regression cases) so that we report verified findings.

3: Additional reasoning, recovery, and cost

DomainArmActorJudgeDeploy tok./ep.Successes/1M tok.
AirlineG84,5695,20289,7714.27
AirlineFull98,3858,257106,6424.84
RetailG101,8387,466109,3034.50
RetailFull103,1419,533112,6755.25

We withdraw “matched-budget”: the model roles and one-retry cap are matched. Full is more expensive than G and also yields more successful episodes per deployed token in both domains.

Reviewer_uvFwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PA-Rehearsal, a training-free pre-commit control framework for tool-using agents. It combines LLM-based semantic assessment with deterministic, evidence-based release control to ensure that proposed state-changing actions align with the user's latest intent. Experiments on τ-bench and WebShop demonstrate improvements in task completion.

Critical strengths
  1. The paper addresses a practically important problem in tool-using agents: ensuring that executable actions remain consistent with evolving user intentions. The motivation is clear and well illustrated.​

  2. The proposed framework is relatively simple and can be integrated with existing agent systems. The experiments include comparisons with existing methods, component ablations, and token-cost analysis.

Critical weaknesses
  1. Limited methodological novelty and reproducibility. The proposed method overlaps with existing runtime guardrails, intent-aware validation, and rule-based enforcement, while its technical distinctions remain insufficiently clarified. Moreover, key implementation details, including rehearsal prompts, evidence-checking rules, and recovery protocols, are underspecified. No code repository link is provided in the manuscript, further limiting reproducibility.

  2. Insufficient evaluation of the core mechanism. The experiments primarily report task completion metrics without directly evaluating purpose–effect consistency. In particular, the lack of action-level annotations, false rejection rates, and false approval rates makes it difficult to determine whether the improvements stem from accurate intent-conflict detection rather than additional checking and recovery.

Questions for authors
  1. PA-Rehearsal increases token consumption to 2.47× the baseline on WebShop, whereas Retail and Airline show much smaller increases (+16.0% and +4.9%) relative to their respective baselines. What accounts for this variation in token overhead? Could the authors provide comparable actor/judge token breakdowns, rehearsal frequencies, and rejection/recovery statistics to better explain these differences?

  2. The evaluation is limited to three domains across two benchmarks, with only 40 Retail and 20 Airline tasks selected from τ-bench. What motivated the selection of these relatively small subsets, and how representative are they of the full benchmark? Given the limited task and benchmark coverage, it remains unclear whether the observed improvements would generalize to other tool-use tasks, benchmarks, and scenarios involving evolving user intent.

1 comment

Thank you for emphasizing reproducibility and direct evaluation of purpose–effect consistency. We address each of your raised concerns below.

1: Reproducible mechanism, stated directly

ComponentDecision-relevant specification
BoundaryIntercept state-changing candidates only; read-only exploration is unchanged
Purpose updateResolve at every write boundary from actor-visible ordered user turns; the latest explicit revision overrides earlier requests
Judge instructionCompare the exact candidate’s expected effect with latest purpose, authorized scope, active constraints, and mandatory policy prerequisites
Schemapurpose_mismatch, scope_expanded, user_constraint_dropped, policy_precondition_missing; verifiability; field; evidence; counterevidence; precondition; rationale; correction
Registryeligibility, explicit_user_scope, authorization_confirmation, domain_policy
ControllerREJECT only a registered, target-bound, candidate-time-verifiable violation that survives counterevidence; otherwise downgrade and log
RecoveryOne tool-enabled correction; on exhaustion block/clarify, except confirmation bound to the same held candidate

Airline and Retail share this contract. Adapters contain only policy/tool-schema-derived evidence projection and hard rules

2: Same-stack attribution, action-level audit, and cost

Replacing only the purpose-aware instruction with generic checking, while fixing the actor, Judge, trigger, parser, controller, projection, and retry cap, reduces three-seed Avg. from 51.67% to 38.33% on Airline and from 59.17% to 49.17% on Retail.

DomainArmActor tok./ep.Judge tok./ep.Deploy tok./ep.Successes/1M tok.
AirlineG84,5695,20289,7714.27
AirlineFull98,3858,257106,6424.84
RetailG101,8387,466109,3034.50
RetailFull103,1419,533112,6755.25

Overhead across domains: WebShop’s larger token overhead reflects control flow: blocking an early purchase returns the actor to search/navigation, whereas an unguarded purchase terminates immediately; roughly 87% of added tokens come from actor continuation rather than Judge output. By contrast, τ\tau-bench gates sparser state-changing writes and allows at most one correction, resulting in much smaller deployment-token increases (+4.9% on Airline and +16.0% on Retail over No Harness).

Action-level consistency: Evaluating candidate-time correctness requires inspecting the dialogue, visible evidence, authorization, and latest purpose available prior to each proposed write rather than endpoint reward alone. We are currently finalizing and cross-checking our blinded candidate-time audit (separating Judge vs. Controller decisions, false-rejection/false-approval checks, and rehearsal/recovery statistics) so that we report verified figures.

3: Evaluation coverage

TaskAvailable Train TasksTrain TasksTest / Eval TasksTotal Tasks
Airline30302050
Retail747440114
WebShop9,0001002009,200

τ\tau-bench uses all 20 Airline and all 40 Retail official test tasks over three seeds. WebShop uses the same frozen 200-task roster and configuration as Life-Harness.

Reviewer_sEHfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Purpose-Aware Rehearsal (PA-Rehearsal), a training-free mechanism that checks state-changing tool calls immediately before execution. It binds each candidate action to the latest user purpose and observable context, obtains an LLM assessment of potential conflicts, and uses a deterministic controller to enforce domain-specific evidence and release rules. Rejected actions return to the agent for revision or clarification. Evaluations on τ-bench Retail and Airline report the highest mean single-run success among the compared methods. On WebShop, adding rehearsal to a fixed trajectory-regulation scaffold improves success from 44.5% to 50.0%, with increased token usage.

Critical strengths
  1. A practically useful separation between action generation and release control. An executable, policy-permitted action can still target the wrong outcome, particularly when user requirements change. Checking the exact proposed write before commitment addresses this important failure mode. Candidate-specific binding, rechecking revised proposals, and evidence-based resolution make the approach more concrete than a general reminder to verify actions.

  2. The evaluation provides evidence of complementary components and composability. The transactional ablations show higher success for the full method than either semantic judgment or hard rules alone. The fixed-scaffold WebShop comparison also supports an incremental benefit from pre-commit checking, with a task-cluster confidence interval excluding zero. Reporting regressions, repeated-success metrics, and resource overhead helps characterize the intervention’s benefits and limitations.

Critical weaknesses
  1. The distinctive contribution of purpose-aware “rehearsal” is not sufficiently isolated or specified. Existing action guards can already ask an LLM whether a proposed tool call satisfies user requirements and domain policies. To establish a distinct control primitive, the paper needs to show what prospective purpose–effect reasoning adds beyond such candidate-level verification. However, the implemented schemas do not require explicit predicted effects or implied goals (§3.3), and there is no matched comparison against a generic pre-execution verifier using the same judge, context, rules, and recovery budget. Important implementation details—how the latest purpose is maintained, which mandatory conditions are registered, and how evidence and counterevidence are validated—are also insufficiently specified. Consequently, the results do not clearly distinguish the proposed mechanism from a domain-engineered LLM guard with feedback.

  2. The experiments do not adequately separate mechanism-specific gains from additional inference and recovery. Because an intercepted action gives the agent another opportunity to reason, gather information, and revise its proposal, improved task completion alone does not establish that purpose-aware checking is responsible. This concern is especially relevant on WebShop: the separate resource audit reports 2.47× token usage relative to trajectory regulation, with approximately 87% of the increase attributable to additional actor continuation. The paper explicitly acknowledges that this comparison is not compute-matched (§4.5), yet the conclusion claims that matched-budget comparisons rule out superficial retries. A compute-matched critique/retry baseline and independently labeled action-level violations are needed to support that attribution. Moreover, the transactional evaluation covers only 40 Retail and 20 Airline tasks with one actor–judge pairing, and does not specifically isolate revised-intent cases despite their central role in the motivation. These limitations constrain both the causal interpretation and generality of the findings.

Questions for authors

See the weaknesses.

1 comment

Thank you for identifying what purpose-aware rehearsal adds beyond generic checking and whether gains reflect additional recovery. We agree that these points needed clearer isolation.

1: Purpose-aware reasoning versus generic checking

Our new same-stack control G fixes the Qwen3-8B actor, GLM-5.2 Judge, write trigger, parser, PA controller, evidence projection/context construction, and one-retry cap; only the Judge instruction changes from latest-purpose/effect comparison to generic validity, policy, and current-request checking. Thus G versus Full isolates the purpose-aware semantic instruction, not realized compute.

Avg. is the mean over all three seeded runs (formerly Pass@1), not one selected run.

DomainG / Full Avg.G / Full Pass@3G / Full Pass^3
Airline38.33 / 51.6750.00 / 70.0025.00 / 30.00
Retail49.17 / 59.1777.50 / 75.0022.50 / 42.50

Full−G Avg. is +13.33 pp on Airline and +10.00 pp on Retail. Existing internal ablations are consistent with complementarity: removing deterministic rules gives 40.00/55.83 Avg. on Airline/Retail, while removing the semantic Judge gives 41.67/46.67, versus Full’s 51.67/59.17.

Two frozen post-hoc cases make the target concrete. If a user prioritizes a boot exchange over two returns, executing a locally valid return first can preclude the exchange, so it is BLOCKED for clarification. Conversely, a confirmed bottle replacement remains RELEASED despite an unrelated cancelled-order inquiry. PA asks whether this exact write advances the latest priority; it does not block whenever anything is unresolved. These cases did not inform rule development.

2: Retries, compute, and action-level evaluation

Retry attribution. G gives the actor the same one-retry opportunity to gather evidence and revise its proposal, yet trails Full by 13.33 pp on Airline and 10.00 pp on Retail Avg.. Full also yields more task completions per million deployed tokens than G (Airline: 4.84 vs. 4.27; Retail: 5.25 vs. 4.50). This does not imply equal realized compute: Full/G deploy tokens per episode are 106,642/89,771 on Airline and 112,675/109,303 on Retail. We have therefore removed the unsupported “matched-budget” wording. WebShop is also explicitly non-compute-matched: its 2.47× token use mainly reflects additional actor continuation after a blocked premature purchase (~87% of the increase), and is presented as a compositional efficacy test.

Action-level evaluation. A τ-bench reference sequence indicates which operations may eventually be needed, not whether a write is permissible when proposed. A matching tool/target can still precede identity, eligibility, confirmation, or supporting evidence; the same write may become permissible later. We use sequences only for approximate eventual alignment and evaluate candidate-time correctness from prior dialogue, evidence, authorization, and latest purpose. We are cross-checking the blinded audit—separating Judge assessments from Controller decisions and tagging revised-intent trajectories—before reporting rates.

3: Decision-relevant implementation contract

ComponentRuntime behavior
TriggerAudit state-changing candidates at the last reversible boundary; read-only calls pass unchanged
PurposeRe-resolve from actor-visible ordered user turns; the latest explicit correction, narrowing, withdrawal, or authorization wins
Judge/schemaCompare the candidate’s expected effect with purpose, scope, constraints, and policy; return four violation flags plus verifiability, registered field, evidence, counterevidence, precondition, rationale, and correction
ReleaseEnforce REJECT only for a registered class (eligibility, explicit_user_scope, authorization_confirmation, domain_policy) supported by target-bound pre-candidate evidence and surviving counterevidence; otherwise downgrade and log
RecoveryPermit exactly one tool-enabled correction; another rejection blocks without bypass, except explicit confirmation bound to the held candidate

4: Complete evaluation rosters

TaskAvailable Train TasksTrain TasksTest / Eval TasksTotal Tasks
Airline30302050
Retail747440114
WebShop9,0001002009,200

τ-bench uses the complete official test splits—all 20 Airline and 40 Retail tasks over three seeds—not selected subsets. WebShop uses the same frozen 200-task roster and configuration as Life-Harness.

Reviewer_nmDeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the problem that most conventional tool-using agents mechanisms have; Most of them just check if the proposed action is valid, executable, and not against policy, etc., and care less about whether the actions and their consequences are really aligned with the user's objectives, especially in scenarios where the user's intention changes in the middle of the task. So they propose the Purpose-Aware Rehearsal method, using an extra LLM judge which can semantically determine whether the expected consequences of the proposed action are aligned with the user's objective or not. They also propose a deterministic controller to analyze the judge's assessment and make a decision based on that. The objective of this paper is to make the committed actions more focused on the user's objectives and prevent scenarios where doing a wrong action cannot be easily undone or has some costs for the user.

Critical strengths

1- The paper focuses on a practical problem in tool-using agents, especially in scenarios where the agent's actions can change the external environment and making a wrong decision can be costly or hard to undo. Although the problem of verifying actions before execution has been studied before (for example SABER, https://arxiv.org/abs/2512.07850, which also gates state-changing actions on τ-bench), I appreciate the authors' effort to focus more specifically on whether the consequences of an action are really aligned with the user's latest objective, rather than just checking if the action is valid or follows the policies.

2- Good design choice to apply the proposed verification only before state-changing actions, while leaving read-only actions unchanged. This allows the agent to continue exploring the environment without adding extra verification to every step.

Critical weaknesses

1. Lack of fair comparison in terms of inference budget: I think the paper lacks a fair comparison with baselines under the same inference budget. For example, instead of adding an extra LLM judge, we could let the baseline model think and reason more by allocating additional thinking tokens, or use a stronger model with a comparable computational budget. This would help us understand whether the improvement really comes from the proposed LLM judge and deterministic controller, or simply from spending more computation during inference. This matters more here because the judge (GLM-5.2) is a much larger model than the 8B actor. Also, the conclusion says that “matched-budget comparisons” confirm the gains, but I could not find any matched-budget experiment in the paper, and Section 4.5 itself says the WebShop comparison is not compute-matched

Also, since the computational cost of each token depends on factors like model size, architecture, and input/output sequence lengths, comparing only the number of tokens is not enough. It would be better to also compare the total inference FLOPs, computational cost, and end-to-end latency of each episode, from the beginning until the agent completes the task.

2. Ablation studies are not informative enough: Although the paper includes some ablation studies to isolate the effect of each component, I think they are still not informative enough to understand how exactly each proposed component contributes to the performance. Since the main objective is to identify and prevent actions that are not aligned with the user's current purpose, just reporting the final task success rate is not enough. For example, how often does the method correctly reject an action that violates the user's objective, and how often does it incorrectly reject a valid action or allow an invalid one? It would help to report this separately for the judge’s decisions and for the controller’s final decision. The paper says action-level labels are not available, but τ-bench tasks come with reference action sequences, so the rejected and released writes could be checked against them, at least approximately.

3. Reliance on a human-designed deterministic controller: The ablation without evidence resolution shows that the controller helps (Pass@1 drops from 59.17 to 55.83 on Retail and from 51.67 to 40.00 on Airline). However, as I understand, this controller relies on predefined rules, mandatory fields, and domain-specific evidence, which require some human effort to design, and the paper does not actually list them (Section 3.4 only names the categories, and the hard rules are not given). This could make the method less plug-and-play and harder to adapt to other tasks or environments without additional modifications. It would be useful to clarify how much domain-specific configuration is needed and how well the controller generalizes to new tasks.

Another unclear part is what happens when the LLM judge identifies a potential violation that is not covered by the predefined rules or mandatory fields of the deterministic controller. As the paper mentions, concerns without sufficient enforceable evidence are treated as nonblocking. But what if the judge correctly identifies a new type of violation that the controller is not designed to handle? Wouldn't this lead to false negatives, where an incorrect action is still allowed to execute? It would be useful to report how many of the judge’s flags were discarded by the controller, and how many of those were actually correct.

Questions for authors

Please see the 'Critical Weaknesses' section for my questions/concerns.

2 comments

Thank you for highlighting the careful verification design of our method. We appreciate the helpful feedback and address each concern below.

1: Mechanism-matched control and inference cost

We agree that the original “matched-budget” wording was unsupported and have removed it. Our new control G fixes the Qwen3-8B actor, GLM-5.2 Judge, write trigger, parser, deterministic PA controller, evidence projection, and one-retry cap; it changes only the Judge instruction from latest-purpose/effect comparison to a generic validity/policy/current-request check. Thus G versus Full isolates the purpose-aware semantic instruction under the same stack, rather than claiming equal realized compute.

Avg. is the mean over all three seeded runs (formerly Pass@1), not one selected run.

DomainG / Full Avg.G / Full Pass@3G / Full Pass^3
Airline38.33 / 51.6750.00 / 70.0025.00 / 30.00
Retail49.17 / 59.1777.50 / 75.0022.50 / 42.50

Full-minus-G Avg. is +13.33 pp on Airline and +10.00 pp on Retail. Existing internal ablations are consistent with complementarity: removing deterministic rules gives 40.00/55.83 Avg. on Airline/Retail, while removing the semantic Judge gives 41.67/46.67, versus Full’s 51.67/59.17.

DomainArmActor tok./ep.Judge tok./ep.Deploy tok./ep.Successes / 1M tok.
AirlineG84,5695,20289,7714.27
AirlineFull98,3858,257106,6424.84
RetailG101,8387,466109,3034.50
RetailFull103,1419,533112,6755.25

Full uses more tokens but yields more successful episodes per million deployed tokens in both domains. Tokens are not cross-model FLOPs, so we make no exact-compute claim.

2: Action-level evaluation and reference action sequences

We appreciate your observation that τ\tau-bench tasks provide reference action sequences that can be used for approximate checking. We agree that relying solely on final task success leaves action-level behavior under-examined, and we clarify how reference action sequences and candidate-time correctness relate in our setting.

A τ\tau-bench reference action sequence indicates which state-changing operations may eventually be needed, rather than whether a specific candidate write is permissible at the exact step tt it is proposed. A candidate write can match a reference tool and target while still being premature or misaligned at step tt if required identity verification, eligibility checks, explicit user confirmation/authorization, latest-purpose priority, or supporting evidence has not yet been established. Conversely, the same write can become permissible later in the trajectory once those preconditions are observed. Treating reference-sequence inclusion alone as a step-level label would therefore conflate eventual task necessity with step-level permissibility.

Accordingly, we treat reference sequences only as an approximate check of eventual tool/target alignment, while evaluating candidate-time correctness using only the dialogue, visible evidence, authorization, and latest user purpose available prior to each proposed write. We are currently finalizing and cross-checking our blinded candidate-time audit (separating Judge assessments from Controller final decisions) to ensure full consistency before reporting detailed action-level rates, rather than presenting unverified ratios here.

3: What the controller enforces

The controller recognizes four registered classes: eligibility, explicit_user_scope, authorization_confirmation, and domain_policy. A Judge flag blocks only when it names a registered field, is verifiable from target-bound pre-candidate evidence, survives counterevidence, and gives a recoverable condition; otherwise it is downgraded and logged. No rule uses task IDs, hidden answers, endpoint rewards, or audit labels. This is an explicit adaptation interface, not adaptation-free universal transfer.

I want to first acknowledge the effort you put into preparing answers to my concerns/questions.

Some of my concerns are partially resolved (e.g., you show the effectiveness of the purpose-aware instruction by comparing it with a general instruction that is not purpose-aware), but the comparison I asked for with the baseline model (no LLM judge) under a matched thinking budget is still missing, and honestly I’m not convinced that the performance you get is because of the extra LLM-judge phase and not just the extra compute.

For my concerns about FLOPs, latency, real cost (not just tokens), etc., although you didn’t answer them properly, I won’t let this affect my final decision, because it may take a lot of time to do and it’s not fair to consider it in my evaluation.

For the action-level evaluation, I understand the reference action sequences are only approximate, but they can still catch clear errors (for example, an executed write that is not in the reference). Since the audit you mentioned is not provided yet, it’s still not clear how accurate the judge and the controller actually are and how each of them play their roles individually/together in your pipeline.

Also, for the warnings that don’t fit any rule, it seems the method is not robust to new problems that are not defined or expected, which makes it less plug-and-play, and I think this can reduce the value of this method compared to a larger baseline with more thinking tokens.

Also, the comparison with weaker judges with fewer parameters that I asked for was not examined, so it’s not clear whether purpose-aware instructions are beneficial for smaller judges, which is closer to real-world scenarios.

I again appreciate your effort, but I’ll stick with my initial assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.