- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces GuardValue, an instrument for cloning live tool-agent episodes to distinguish the value of a single guard intervention from the effect of repeatedly applying a guard policy. It estimates local intervention values through paired continuations and aggregates them along guarded-policy trajectories using the established performance-difference identity. Experiments show that first-intervention values can misrepresent and misrank guard policies, and investigate when occupancy sums agree with separately measured policy effects.
Critical strengths
-
The paper makes a useful distinction between local intervention values and policy-level effects. The ranking reversal in the second-actor airline experiment makes the consequence concrete: first-point estimates rank the reviewed engine above silent refusal, whereas their whole-episode effects have the opposite ordering. This is a meaningful warning for evaluations that extrapolate from isolated interventions to repeated deployment.
-
The instrument validation and reporting of limitations are careful. The authors examine execute/execute controls, adjacent continuations, trajectory agreement, budget accounting, serving variation, and episode-start synchronization. They also explicitly exclude defective early experiments and report unresolved tests and protocol deviations. These checks increase confidence in the results under the validated conditions.
Critical weaknesses
-
The strongest findings remain closely tied to the refusal protocol, finite horizon, and benchmark scoring. Algorithm P.2 implements silent refusal without adding feedback to the actor’s history while consuming a proposal step. Appendix U reports that 229/270 single refusals add exactly one step, while 266/270 repeated-refusal continuations reach the step limit. In the every-visit retail study, eight of ten nonzero values occur at the last visit. Separately, Table F.1 shows that TypeBlock never succeeds on the 199 reference-mutation tasks whose reference replay changes the scored database, while its successes in the no-mutation stratum occur exclusively where that database remains unchanged.
-
The additional practical value of the instrument is not sufficiently demonstrated. The performance-difference identity and shared-state branching are established, as the authors acknowledge. Exact occupancy agreement is therefore valuable implementation validation, but the paper needs a stronger demonstration of what the resulting local measurements enable. In Table Z.7, whole-episode evaluation achieves the same zero selection regret with approximately 288k actor tokens per task versus 899k for occupancy walks. A compelling application would use the local values to diagnose and improve a guard, then validate that improvement on held-out tasks.
Questions for authors
Looking forward to receiving your rebuttals:
-
How do the first-point gap, policy effect, and locations of nonzero local values change when the interaction budget and proposal horizon are varied independently?
-
Can the observed guard gains be evaluated with a complementary measure of successful user-task completion, beyond the final database comparison used in the audit?
-
Can you demonstrate a held-out guard improvement informed specifically by local intervention values, or compare the instrument with a replay-based implementation on fidelity, cost, or diagnostic capability?
-
Does the supplementary artifact include the runnable live-cloning adapter, exact benchmark revision, guard prompts, and a minimal end-to-end reproduction example?
2 comments
Thank you again for your time and thoughtful review. As the discussion deadline approaches, we would be grateful if you could consider our response when your schedule permits. We would be happy to address any remaining questions.
Author Response to Reviewer ZGWK
Thank you for the review. Our measurements show that first-point substitution fails in the evaluated settings, including the ranking reversal you cite (Section 6.4). Finite horizon, refusal feedback, and benchmark scoring constrain how far these findings extend.
1. Independent changes to budget and horizon. We have not run the requested factorial comparison. Silent refusal consumes a proposal step without adding feedback. As you note, 229/270 single refusals add one step, 266/270 repeated-refusal continuations reach the step limit (Appendix U), and 8 of 10 nonzero values in the every-visit retail study fall at the final visit (Appendix Z.5). These observations fit finite-horizon delay and prevent claiming horizon independence for silent refusal. Repeated noticed refusal ends naturally in 269/270 retail continuations (Appendix U). The airline first-point gaps are +0.174 [+0.072, +0.287] at 14B and +0.182 [+0.068, +0.295] at 7B, with 22 nonzero values of both signs across 165 7B visits (Figure 3(b), Table R.1). Still, this neither replaces a budget-by-horizon test nor isolates the cross-domain sign mechanism.
2. Successful user-task completion. We have no validated complementary completion measure. The audit endpoint is benchmark reward, including final-database comparison, rather than independently assessed user satisfaction or task fulfillment. Table F.1 shows this dependence. TypeBlock has 0/597 successes on the 199 reference-mutation tasks whose reference replay changes the scored database, while on the 30 no-mutation tasks where the database stays unchanged it succeeds on 86/90 episodes versus 43/90 for NoGuard. These counts characterize the benchmark scorer and task mix and do not imply that retaining the initial database generally fulfills the user's request.
In the AgentDojo transfer (Section 3, Figure A.1, Table N.1, and Appendix N), the workspace adapter passes an empty final answer to the scorer. Under v1.2.2, 22 of 40 workspace tasks fail on empty text, including 17 of the 18 no-mutation tasks. All 396 archived episodes on those 22 tasks were scored unsuccessful, and the 720 retained traces lack the final answers and terminal states needed for rescoring. The 11 tasks in the reported TypeBlock contrast use state-only scorers (NoGuard 24/33, TypeBlock 0/33) and were selected by observed success. Defined by scorer type alone, the 17 state-only mutation tasks give NoGuard 24 of 51 runs and TypeBlock 0 of 51, a difference of 0.47 [0.27, 0.69] (Appendix N), smaller than 0.73 because NoGuard has 0/18 successes on the six added tasks. The revision withdraws the capability-floor reading.
3. Held-out improvement and replay comparison. Neither comparison has been completed. The Table Z.7 selection is post hoc on an exposed pool where both selectors had zero regret, and restricting to three shared policies chose noticed refusal for both (Appendix Z.7). Table Z.7 omitted customer tokens and used a parent-walk proxy. In a new comparison on ten previously studied airline tasks across allow, noticed refusal, and two early-step refusal rules (Appendix Z.7), whole episodes used 2,199,629 tokens and occupancy 3,864,864, including duplicate controls at 33 visits. Under the same 7-million-token allowance, both gave the three refusal policies identical rewards on every task, +0.20 relative to allow, and selected noticed refusal by the tie rule without separating placement rules or testing unseen tasks. We also lack a live-clone versus replay benchmark of fidelity, cost, or diagnostic power (Appendix P), so we cannot claim that replay is generally incapable of these interventions or that live cloning is superior to faithful replay.
4. Supplementary artifact. The archived package recomputes released outcomes and counterfactual records offline without a model or GPU (Appendix O). Its README excludes a live runner, and the archive contains neither the live-cloning adapter nor an end-to-end live example. Retained audit metadata identifies τ²-bench revision
864350a8971a8f8ee9e7b8472e2edc380a806b0c. Appendices D, E, O, P, and Q describe the feedback rules, cloning contract, and serving stacks, without providing resolved prompt files and live execution dependencies. We therefore cannot claim live-experiment reproducibility from the offline package.Contribution. These results support the evaluated instrument, its validated operating conditions, and concrete counterexamples to treating a first intervention as a repeated-policy effect. The decision-value, replay, and completion-validity claims you identify require evidence beyond our current experiments.