- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes PA-Rehearsal, which checks state-changing actions before a tool-using agent commits them. It binds the candidate action to the current user purpose, observable evidence, and authorization, then combines LLM-based semantic assessment with domain-specific rules to determine whether the action should be released. Rejected actions can be revised and checked again. Experiments cover τ-bench Retail, Airline, and WebShop and report improvements in Pass@1. However, the incremental contribution, implementation details, and attribution of these gains remain insufficiently established, and some conclusions exceed the experimental evidence. I therefore recommend rejection.
Critical strengths
-
The problem is practically relevant. A valid tool call does not necessarily serve the user’s current purpose, so checking actions before committing external writes has practical value.
-
The paper includes three repeated evaluations, component ablations, and selected paired confidence intervals. The results demonstrate improvements in task success.
-
The paper reports token costs and presents a regression case, helping readers understand the method’s overhead and limitations.
Critical weaknesses
- The incremental contribution over existing methods is unclear. The core mechanism combines LLM-based semantic assessment, domain-specific rules, and recovery after rejection. Although the paper discusses AgentSpec, ToolSafe, and ShieldAgent, it does not provide direct experimental comparisons or sufficiently establish the distinctive contribution of purpose-aware assessment over conventional pre-execution constraint checking.
- Important implementation details are missing. The provided PDF does not include complete judge prompts, output schemas, the mandatory-field registry, or domain-specific hard-rule specifications. Page 4 mentions a recovery budget but does not specify its value or the behavior when it is exhausted. The task identifiers and selection procedure for the fixed evaluation subsets are also unspecified, limiting reproducibility and the assessment of generalization.
- The attribution of performance gains lacks sufficient controls. Page 9 acknowledges that the WebShop comparison is not compute-matched, while the conclusion claims that “matched-budget comparisons” establish that the gains are not due to additional retries. The separate resource profile shows approximately 2.47 times the token usage, with actor continuation accounting for roughly 87% of the additional tokens. The current results therefore do not adequately disentangle the contribution of the proposed checking mechanism from additional reasoning and exploration.
- Some conclusions are overstated. In Table 1, Airline success across all three runs is 30%, below Life-Harness’s 35%, while Retail success in at least one of three runs is 75%, below EBP’s 82.5%. The highest Pass@1 point estimate does not establish superiority across all reliability metrics. Moreover, without independent action-level correctness labels, the evidence is insufficient to “rigorously validate” the necessity of combining the components.
Questions for authors
- Please clarify the methodological differences from AgentSpec, ToolSafe, and ShieldAgent. Could direct comparisons be conducted using the same judge model and compute budget?
- Please provide the complete prompts, output schemas, domain-specific rules, recovery budget, and behavior upon budget exhaustion. Please also provide the evaluation task identifiers, subset selection procedure, and rule-development process.
- Please resolve the inconsistency regarding compute-budget matching on page 9 and add compute-matched controls to examine whether the gains primarily arise from additional reasoning and continued exploration.
- Could interventions be independently annotated to report false approvals, false rejections, and the causes of regression cases? Please also restrict the reliability claims to the metrics supported by the results.
Thank you for the direct questions. We address your concerns below.
1: Relation to existing guards and direct control
We cite these works to position pre-execution guarding, rather than reporting unmatched reimplementations whose rule authoring, threat models, and stacks differ. Instead, our same-stack control G fixes the Qwen3-8B actor, GLM-5.2 Judge, write trigger, parser, PA controller, evidence projection, and one-retry cap, changing only the Judge instruction from purpose-aware effect comparison to generic validity/policy/current-request checking.
Avg.is the mean of all three seeded runs (formerly Pass@1), not a single-seed result. We use Avg. as the primary aggregate while separately reporting at-least-one-of-three and all-three reliability.2: Implementation, test sets, and action-level audit
The operative contract is: audit writes only; latest explicit user revision wins; compare the candidate’s expected effect with purpose/scope/constraints/policy; return four violation flags plus verifiability, registered field, evidence, counterevidence, precondition, rationale, and correction; enforce rejection only with registered target-bound evidence and no valid counterevidence; allow one correction, then block/clarify except confirmation bound to the held candidate.
Airline uses all 20 official test tasks × 3 seeds, and Retail all 40 × 3. WebShop uses the same frozen 200-task Life-Harness roster and configuration.
Regarding action-level correctness, false rejections/approvals, and regression causes, we are currently finalizing and cross-checking our blinded candidate-time audit (separating Judge vs. Controller decisions and examining discordant/regression cases) so that we report verified findings.
3: Additional reasoning, recovery, and cost
We withdraw “matched-budget”: the model roles and one-retry cap are matched. Full is more expensive than G and also yields more successful episodes per deployed token in both domains.