acceptodds
Under review as a conference paper at ICLR 2027

GuardValue: Estimating One-Step Values and Policy Effects with Cloned Trajectory Prefixes

Abstract

Pre-execution guards decide whether a tool-using agent's proposed action runs. Comparing guarded with unguarded episodes measures a policy that may intervene repeatedly, not the value of one intervention. We introduce GuardValue, which clones a live benchmark episode to measure both from a shared history and sums one-step values over guarded-policy visits, as the performance-difference identity prescribes. In a ten-arm matched-budget audit on 278 τ²-bench tasks, type-blocking raises success by 0.264 where the reference uses no irreversible action and lowers it by 0.101 elsewhere. One silent refusal has near-zero value, while repeated refusal changes success by -0.089 per first mutating proposal in retail, +0.270 in airline and +0.227 with a second actor. With every visit measured under deterministic serving and synchronized starts, the occupancy sum equals separately measured effects task by task in four checks covering silent and noticed refusal and a budget-charged engine, and on 47 of 50 tasks without it. A fixed visit schedule leaves the registered silent test unresolved, and production-serving airline attempts disagree. For the second actor, first-point values rank the engine above silent refusal, whose effect is ten times larger. First-point values should not be reported as policy effects.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
tool-using agents, safety guards, causal estimation, policy evaluation, one-step value, counterfactual trajectories, agent benchmarks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZGWKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GuardValue, an instrument for cloning live tool-agent episodes to distinguish the value of a single guard intervention from the effect of repeatedly applying a guard policy. It estimates local intervention values through paired continuations and aggregates them along guarded-policy trajectories using the established performance-difference identity. Experiments show that first-intervention values can misrepresent and misrank guard policies, and investigate when occupancy sums agree with separately measured policy effects.

Critical strengths
  1. The paper makes a useful distinction between local intervention values and policy-level effects. The ranking reversal in the second-actor airline experiment makes the consequence concrete: first-point estimates rank the reviewed engine above silent refusal, whereas their whole-episode effects have the opposite ordering. This is a meaningful warning for evaluations that extrapolate from isolated interventions to repeated deployment.

  2. The instrument validation and reporting of limitations are careful. The authors examine execute/execute controls, adjacent continuations, trajectory agreement, budget accounting, serving variation, and episode-start synchronization. They also explicitly exclude defective early experiments and report unresolved tests and protocol deviations. These checks increase confidence in the results under the validated conditions.

Critical weaknesses
  1. The strongest findings remain closely tied to the refusal protocol, finite horizon, and benchmark scoring. Algorithm P.2 implements silent refusal without adding feedback to the actor’s history while consuming a proposal step. Appendix U reports that 229/270 single refusals add exactly one step, while 266/270 repeated-refusal continuations reach the step limit. In the every-visit retail study, eight of ten nonzero values occur at the last visit. Separately, Table F.1 shows that TypeBlock never succeeds on the 199 reference-mutation tasks whose reference replay changes the scored database, while its successes in the no-mutation stratum occur exclusively where that database remains unchanged.

  2. The additional practical value of the instrument is not sufficiently demonstrated. The performance-difference identity and shared-state branching are established, as the authors acknowledge. Exact occupancy agreement is therefore valuable implementation validation, but the paper needs a stronger demonstration of what the resulting local measurements enable. In Table Z.7, whole-episode evaluation achieves the same zero selection regret with approximately 288k actor tokens per task versus 899k for occupancy walks. A compelling application would use the local values to diagnose and improve a guard, then validate that improvement on held-out tasks.

Questions for authors

Looking forward to receiving your rebuttals:

  1. How do the first-point gap, policy effect, and locations of nonzero local values change when the interaction budget and proposal horizon are varied independently?

  2. Can the observed guard gains be evaluated with a complementary measure of successful user-task completion, beyond the final database comparison used in the audit?

  3. Can you demonstrate a held-out guard improvement informed specifically by local intervention values, or compare the instrument with a replay-based implementation on fidelity, cost, or diagnostic capability?

  4. Does the supplementary artifact include the runnable live-cloning adapter, exact benchmark revision, guard prompts, and a minimal end-to-end reproduction example?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer ZGWK

Thank you for the review. Our measurements show that first-point substitution fails in the evaluated settings, including the ranking reversal you cite (Section 6.4). Finite horizon, refusal feedback, and benchmark scoring constrain how far these findings extend.

1. Independent changes to budget and horizon. We have not run the requested factorial comparison. Silent refusal consumes a proposal step without adding feedback. As you note, 229/270 single refusals add one step, 266/270 repeated-refusal continuations reach the step limit (Appendix U), and 8 of 10 nonzero values in the every-visit retail study fall at the final visit (Appendix Z.5). These observations fit finite-horizon delay and prevent claiming horizon independence for silent refusal. Repeated noticed refusal ends naturally in 269/270 retail continuations (Appendix U). The airline first-point gaps are +0.174 [+0.072, +0.287] at 14B and +0.182 [+0.068, +0.295] at 7B, with 22 nonzero values of both signs across 165 7B visits (Figure 3(b), Table R.1). Still, this neither replaces a budget-by-horizon test nor isolates the cross-domain sign mechanism.

2. Successful user-task completion. We have no validated complementary completion measure. The audit endpoint is benchmark reward, including final-database comparison, rather than independently assessed user satisfaction or task fulfillment. Table F.1 shows this dependence. TypeBlock has 0/597 successes on the 199 reference-mutation tasks whose reference replay changes the scored database, while on the 30 no-mutation tasks where the database stays unchanged it succeeds on 86/90 episodes versus 43/90 for NoGuard. These counts characterize the benchmark scorer and task mix and do not imply that retaining the initial database generally fulfills the user's request.

In the AgentDojo transfer (Section 3, Figure A.1, Table N.1, and Appendix N), the workspace adapter passes an empty final answer to the scorer. Under v1.2.2, 22 of 40 workspace tasks fail on empty text, including 17 of the 18 no-mutation tasks. All 396 archived episodes on those 22 tasks were scored unsuccessful, and the 720 retained traces lack the final answers and terminal states needed for rescoring. The 11 tasks in the reported TypeBlock contrast use state-only scorers (NoGuard 24/33, TypeBlock 0/33) and were selected by observed success. Defined by scorer type alone, the 17 state-only mutation tasks give NoGuard 24 of 51 runs and TypeBlock 0 of 51, a difference of 0.47 [0.27, 0.69] (Appendix N), smaller than 0.73 because NoGuard has 0/18 successes on the six added tasks. The revision withdraws the capability-floor reading.

3. Held-out improvement and replay comparison. Neither comparison has been completed. The Table Z.7 selection is post hoc on an exposed pool where both selectors had zero regret, and restricting to three shared policies chose noticed refusal for both (Appendix Z.7). Table Z.7 omitted customer tokens and used a parent-walk proxy. In a new comparison on ten previously studied airline tasks across allow, noticed refusal, and two early-step refusal rules (Appendix Z.7), whole episodes used 2,199,629 tokens and occupancy 3,864,864, including duplicate controls at 33 visits. Under the same 7-million-token allowance, both gave the three refusal policies identical rewards on every task, +0.20 relative to allow, and selected noticed refusal by the tie rule without separating placement rules or testing unseen tasks. We also lack a live-clone versus replay benchmark of fidelity, cost, or diagnostic power (Appendix P), so we cannot claim that replay is generally incapable of these interventions or that live cloning is superior to faithful replay.

4. Supplementary artifact. The archived package recomputes released outcomes and counterfactual records offline without a model or GPU (Appendix O). Its README excludes a live runner, and the archive contains neither the live-cloning adapter nor an end-to-end live example. Retained audit metadata identifies τ²-bench revision 864350a8971a8f8ee9e7b8472e2edc380a806b0c. Appendices D, E, O, P, and Q describe the feedback rules, cloning contract, and serving stacks, without providing resolved prompt files and live execution dependencies. We therefore cannot claim live-experiment reproducibility from the offline package.

Contribution. These results support the evaluated instrument, its validated operating conditions, and concrete counterexamples to treating a first intervention as a repeated-policy effect. The decision-value, replay, and completion-validity claims you identify require evidence beyond our current experiments.

Thank you again for your time and thoughtful review. As the discussion deadline approaches, we would be grateful if you could consider our response when your schedule permits. We would be happy to address any remaining questions.

Reviewer_ZfUiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies pre-execution guards for tool-using agents, such as rules that decide whether a proposed action is allowed to run. There are two major takeaways from the work. First, it runs a large matched-budget comparison of guard designs on τ²-bench. Blocking every irreversible action helps on tasks whose reference solution needs no such action and hurts on tasks whose reference solution does, so the guard’s overall value depends on the task mix. Second, it argues that comparing guarded and unguarded episodes measures the effect of a guard that may intervene many times, not the value of a single intervention. To measure both from the same history, the authors build GuardValue, a tool that clones a live benchmark episode at a proposed action and continues one copy with the intervention and one without. Using the performance-difference identity, the policy effect equals the sum of one-step values over all the points the guarded policy visits. Experiments show that the value of the first intervention alone can misstate, and even misorder, the effect of a repeated guard policy, while the sum over every visit matches the policy effect when model serving is made deterministic.

Critical strengths
  • The paper points out a real and easily missed problem in how pre-execution guards are evaluated. Comparing guarded and unguarded episodes measures the effect of a guard that may intervene many times, not the value of a single intervention, yet the two are often treated as the same thing. Separating them matters for anyone deciding where and how often a guard should act, and the paper’s recommendation not to report first-intervention values as policy effects is a useful lesson for the agent-safety community.
  • The core idea is easy to grasp from Figure 1, which shows the difference between a one-step value and a policy effect and how cloning measures both. Method section is clear and readable.
  • The experimental work is extensive and careful. And the engineering looks solid. Cloning a stateful, multi-threaded benchmark episode with no built-in rollback is non-trivial. The authors check clone fidelity carefully and trace the remaining mismatches to serving nondeterminism and a reset race in the benchmark, rather than to the cloning itself. This finding is valuable for agent-evaluation infrastructure in general.
Critical weaknesses
  • The main conceptual result is a known identity. The equality between the policy effect and the sum of one-step values is the performance-difference lemma, which the paper cites. Showing that the sum matches the policy effect when every visit is measured therefore validates the implementation of the cloning rather than producing a new finding. That measuring only the first intervention misses the later ones also follows directly from the same identity.
  • The instrument’s practical value is unclear. For choosing among guards, plain whole-episode comparisons select the same guard as summing over every visit, at roughly a third of the cost (Figure 2). The instrument can locate where along a trajectory a guard’s effect occurs, but the paper does not show that one-step values lead to a better guard or a better decision that whole-episode comparisons would miss.
  • The empirical findings are largely expected. A guard that blocks every irreversible action helps on tasks that need no such action and hurts on tasks that do. This is close to true by construction, and the strata are defined by reference-solution labels that a deployed guard cannot see. Similarly, one silent refusal is worth nearly nothing, most likely because the agent simply proposes the same call again, while refusing every time amounts to forbidding the action. This is also what one would expect.
Questions for authors
  • Beyond confirming the identity, what decision does GuardValue enable that a whole-episode comparison cannot? For example, can one-step values be used to design a better guard, such as one that only intervenes at points with positive value?
  • Do the conclusions change with a stronger actor or a larger budget, where far fewer branch points sit at the success floor?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer ZfUi

Thank you for your assessment of the identity, cloning engineering, and decision value. Local values locate the states where an intervention changes the outcome, including sign changes within a trajectory, which a whole-episode average cannot show. They have not yet produced a better guard validated on unseen tasks.

Selection between fixed guards. Both selectors had zero regret in the reported post hoc analysis (Table Z.7), where silent refusal was available only for whole episodes. Recomputing on the same 50 airline tasks over the three shared policies gave mean reward differences of 0.00 to allow, +0.02 to the reviewed engine, and +0.20 to noticed refusal under both selectors, and both chose noticed refusal (Appendix Z.7). The 288.2k and 899.2k tokens per choosing task in Table Z.7 combine actor and guard tokens, omit customer tokens, and estimate parent-walk cost from a separate episode, rather than matching total cost.

A new comparison ran both selectors with the 14B actor on ten previously studied airline tasks under a shared 7-million-token allowance per selector (new Appendix Z.7), comparing allow, noticed refusal, and two rules limiting refusal to earlier steps. Under both selectors all three refusal rules gave identical rewards on every task, +0.20 relative to allow, so the fixed tie rule chose noticed refusal for both without separating placement rules. Whole episodes used 2,199,629 actor and customer tokens and occupancy used 3,864,864, including duplicate controls at 33 visits, over 70 completed measurement units, yielding no distinct rules to test on an independent set.

What local values add. The 7B noticed-refusal study has 22 nonzero values among 165 visits, of both signs within one walk (Figure 3(b)). In the every-visit retail silent-refusal study, 8 of 10 nonzero values fall at the final visit (Appendix Z.5), fitting your reading that silent refusal mainly costs a step. Single silent refusal took one more step than execution at 229 of 270 points, though the records store step counts rather than the next proposed action and cannot establish a repeated call (Appendix U). These local attributions have not yet produced a better deployment rule. Intervening whenever the counterfactual value is positive would use future outcomes unavailable at deployment. A valid test would use development local values to select a rule on observable state, freeze it, and test it on unexposed tasks against the best constant guard and matched-budget whole-episode tuning over the same rule family. We have not run that test, and a useful diagnostic need not be a cost-effective selector.

Theory and audit claims. The performance-difference identity and shared-state branching are established. Occupancy agreement validates our implementation. The engineering contribution is copying live environment, customer, budget and thread state without rollback and testing continuation fidelity (Appendices P and Z). Reference-mutation labels record use, not necessity, and are unavailable at deployment. TypeBlock executes no taxonomy mutations by construction. Its no-mutation success increase from 0.277 to 0.541 over 53 tasks (Table 1) is measured rather than implied by the identity. In Table F.1, TypeBlock has 0/597 successes on 199 reference-mutation tasks whose replay changes the scored database. On 30 no-mutation tasks where it stays unchanged, it succeeds on 86/90 episodes versus 43/90 for NoGuard.

Stronger actors and larger budgets. Available evidence does not resolve stronger actors or larger budgets. In the retail probe at B=10, the 14B and 32B actors each solve 21 of 114 tasks with 11 shared, with the customer fixed at 14B. Post hoc counts show 32B exhausts budget in 173 of 342 episodes versus 23 for 14B (Appendix X.1, Table X.2). Branch runs were not repeated at 32B, and the 7B airline study is a second-actor check changing both actor and customer. Under the evaluated setup, no tested arm succeeds at 81.1% of retail and 53.0% of airline branch points (Appendix X). Interaction budget and proposal horizon were not varied independently, so we cannot claim that the gaps persist once either constraint is relaxed.

AgentDojo capability statement. The workspace adapter scores an empty final answer, forcing failure on 5 of 22 reference-mutation tasks and 17 of 18 remaining tasks under v1.2.2, so archived outcomes cannot identify the full capable set (Appendix N). The original 11-task contrast uses state-only scorers, with NoGuard at 24 of 33 runs and TypeBlock at 0 of 33. On all 17 state-only mutation tasks defined by scorer type alone, NoGuard succeeds in 24 of 51 runs and TypeBlock in 0 of 51, a difference of 0.47 [0.27, 0.69], smaller than 0.73 because NoGuard has 0/18 successes on the six added tasks.

Thank you again for your time and thoughtful review. As the discussion deadline approaches, we would be grateful if you could consider our response when your schedule permits. We would be happy to address any remaining questions.

Thank you for the careful response and the new experiments. The response confirms my main concern that occupancy agreement validates the implementation rather than giving a new result, a concern also shared by the other reviewers. The warning that first-point values should not be reported as policy effects is useful, and the cloning engineering is solid. However, the main conceptual result follows from a known identity, and the practical value of the instrument remains unclear. The one-step values depend on the future outcomes of both branches, so they are only available offline, as the response also acknowledges. The new shared-budget comparison shows that local measurement cost more and led to the same choice as whole-episode comparison. To show practical value, the paper would need to use these values to learn a rule on observable state and test it on unseen tasks. The paper is also still hard to read and needs a rewrite. The main text mixes the findings with protocol details, such as registered rules, tolerance bands and recovered or repeated campaigns, and uses many coined terms, so the main message is easy to lose. I am inclined to keep my score.

Thank you for pressing us on what the measurements add. We agree that the performance-difference identity is known and that exact agreement checks the implementation. On 50 airline tasks with Qwen2.5-7B as both actor and customer under deterministic serving, both first-point and count-scaled estimates ranked the review-based engine above silent refusal. Running each guard throughout the episode reversed that order: relative to always allowing the action, task success rose by 20 percentage points for silent refusal versus 2 points for the review-based engine.

The branch measurements trace this discrepancy to individual decisions by following both continuations offline, and their cost matters. In a separate shared-token study on ten previously studied airline tasks with Qwen2.5-14B (not independent unseen tasks), both selectors picked noticed refusal under the same fixed tie rule, while occupancy walks cost more tokens than whole episodes. Our results therefore support post-hoc diagnostic decomposition of why local proxies diverge from deployed guard effects, not cheaper guard selection or runtime control. Whether local measurements can help train a rule on observable state that improves outcomes on unseen tasks remains open.

We also take your point about readability. Keeping the ranking reversal, validation boundaries, and cost comparison front and center in the main text, while moving the detailed registered rules, tolerance bands, and recovered or repeated campaign logs to the appendix and replacing coined labels with plain terms, would make the paper much easier to follow.

Reviewer_K8SiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the metrics for evaluating pre-execution guards for tool-using agents, one metric captures the effect of one intervention at a decision state, and the other one measures the effect of a policy that intervenes repeatedly throughout an episode. GuardValue was introduced to estimate such metrics. It clones a live episode at a proposed state-changing action, and rolls out two continuations from this step. The paired continuations estimate a one-step intervention value. And repeating this comparison at states visited by the guarded policy estimates the occupancy sum in the performance-difference identity. The empirical study includes a matched-budget audit of ten guard configurations and branch experiments with tasks from Tau2 bench. Results show that blanket type-blocking helps tasks that do not require mutation actions and hurts tasks that does require mutation actions. In the branch studies, one silent refusal has nearly zero value, while repeated refusal has sizable and domain-dependent effects. When the model serving is configured under deterministic serving and synchronized starts, the summed one-step values match separately measured policy effects.

Critical strengths
  1. The paper gives a useful warning for safety guard evaluation: treating policy-level outcomes as the effect of a single interventions can be misleading. The paper explains the reason. Comparing guarded and unguarded episodes measures what happens when a guard repeats, but judging a local decision needs a one-step value.

  2. The proposed GuardValue offers an empirical method to measure the two metrics. And results on Tau2 tasks are consistent with the theoretical identity (under controlled conditions).

Critical weaknesses

The paper is quite hard to follow in many places. Clearer organization and simpler language would help a lot.

The main conclusions are useful, but most of them follow directly from the performance-difference identity, so they are not surprising and thus the paper's contribution to the field is limited. One intervention does not tell you the policy's overall effect. You also cannot get the policy effect by taking the first intervention's value and multiplying it by the number of interventions. The real policy effect is the sum of the one-step values over the states that the guard policy visits (a simple telescoping sum). And a type-blocking guard helps on tasks that do not need mutation actions. These all follow from the identity, so the paper mainly empirically confirms what the theory already predicts rather than adding new insight.

The strongest results also depend on a very restricted setup: deterministic serving, prefix caching turned off, and synchronized starts. This is expensive to run, from the perspective of serving conditions. The branch experiments covered two domains from Tau2-bench, using Qwen2.5 models. The results agree with the conceptual claim, but considering the models and tasks used, they seem narrow to show that first-intervention values rank practical guards wrong.

Questions for authors
  • Can GuardValue give reliable results with normal production model serving, where the same request may produce different responses?

  • The clearest result comes from silent refusal. Do the noticed-refusal and reviewed-engine experiments show the same gap between the first intervention’s value and the full policy effect?

  • Running GuardValue at every guarded state is expensive, when would it be more useful than simply comparing complete guarded and unguarded episodes?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer K8Si

Thank you for distinguishing the identity, instrument validation and practical value. Our contribution is empirical. GuardValue measures the terms of the performance-difference identity in a live, threaded tool-agent benchmark without rollback, and tests local substitutes (Sections 5 and 6). Exact occupancy agreement validates the implementation rather than proving a new theorem. The identity alone does not predict why TypeBlock raises no-mutation success from 0.277 to 0.541 (Table 1), why repeated refusal flips sign across domains, or when first-point values misrank guards (Section 6.4).

1. Ordinary production serving. We cannot claim validated production reliability. The identity holds in expectation under stochastic outputs, which does not certify finite-sample estimates under arbitrary serving (Section 5, Appendices W.1 and Z.2). Two airline production campaigns reuse the same 50 tasks and whole-episode comparator, with task-clustered 95% intervals. They gave −0.041 [−0.125, +0.041] on 123 of 150 units after recovery versus +0.00667 [−0.060, +0.080] on 150 units in the technical repeat, inside the prespecified ±0.10 band (Table R.1). In the repeat, the within-campaign residual was −0.02667 [−0.05333, −0.00667] and 4 of 273 adjacent comparisons disagreed. A separate diagnostic found 14 of 444 re-sent production requests differed (Appendix Z.2). Passing the band does not imply complete fidelity. Deterministic serving and synchronized starts define where our stronger validation holds. Across two τ²-bench domains with 14B and 7B Qwen2.5 actors, the branch data show that first-point values can misrank guards in these settings, not how often elsewhere.

2. Noticed refusal and the reviewed engine. The gap is neither confined to silent refusal nor universal (Table R.1). For noticed refusal, the airline gap (policy effect minus first-point value) is +0.174 [+0.072, +0.287] at 14B and +0.182 [+0.068, +0.295] at 7B. On retail, the 14B noticed-refusal gap is −0.037 [−0.085, +0.004], including zero under a prespecified width rule that is not an equivalence test (Section 6.1). For the 14B reviewed engine, gaps are −0.004 [−0.011, 0.000] in retail and +0.052 [0.000, +0.116] in airline, with intervals touching zero, while the 7B engine's gap is zero. These contrasts are secondary or descriptive because the registered first-point tests concern silent refusal. The 7B noticed-refusal walks contain 22 nonzero values among 165 visits, including opposite signs within one walk (Figure 3(b)), extending the evidence beyond silent refusal without implying a general mechanism across guards or models.

3. When local measurement is worth its cost. Local measurements locate where an intervention changes the outcome (Figure 3). They have not been shown to be cheaper or better for choosing among fixed guards. Both selectors had zero regret in the post hoc half-splits of 50 airline tasks, and repeating that choice over the three shared candidates (allow, noticed refusal, and the reviewed engine) gave noticed refusal for both (Appendix Z.7). Table Z.7 estimates 288.2k tokens per choosing task for whole episodes and 899.2k for occupancy walks, omitting customer tokens and estimating parent-walk cost from a separate episode, so these are not all-in costs.

A shared-budget test on ten previously studied airline tasks (new Appendix Z.7) used one deterministic seed and four options, namely allow, noticed refusal, and two rules limiting refusal to earlier steps. Whole-episode measurement used 2,199,629 tokens and occupancy 3,864,864, within a 7-million-token allowance each. All three refusal rules gave identical rewards on every task (+0.20 relative to allow), so the fixed tie rule selected noticed refusal for both without discriminating among placement rules. Local measurement cost more and gave the same decision, and we have not shown gains on unseen tasks.

AgentDojo transfer. We also correct the external transfer check (Section 3, Figure A.1, Table N.1, and Appendix N). The workspace adapter passes an empty final answer to the scorer, forcing failure on 22 of 40 tasks under the recorded benchmark version. The original 11 success-selected tasks use state-only scoring, where NoGuard succeeded on 24 of 33 runs and TypeBlock on 0 of 33, and the revision withdraws the capability claim on query tasks. On all 17 state-only mutation tasks defined by scorer type alone, NoGuard succeeds in 24 of 51 runs and TypeBlock in 0 of 51, a difference of 0.47 [0.27, 0.69] (Appendix N), smaller than 0.73 because NoGuard has 0/18 successes on the six added tasks.

Presentation. Each section now opens with its finding, and clone-fidelity mechanics move to Appendices P and Z while keeping the section structure, validation conditions, and production-serving limits in the main text.

Thank you again for your time and thoughtful review. As the discussion deadline approaches, we would be grateful if you could consider our response when your schedule permits. We would be happy to address any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Post-Revision Discussion Summary for the Area Chair

Thank you for overseeing the review of our submission. We briefly summarize the key developments from the discussion following our 15 November revision:

  1. Resolution of the step-limit concern: Reviewer ZGWK noted that the noticed-refusal analysis addressed their concern that the first-point gap was merely a step-limit artifact and indicated inclination to raise their score slightly under the revised, narrower framing (concrete, verified cases where first-point proxies misrepresent whole-episode guard effects).

  2. Clear scope boundaries and presentation plan: In follow-ups with Reviewers ZfUi and ZGWK, we highlighted the clearest empirical finding (on 50 Qwen2.5-7B airline tasks under deterministic serving, both first-point and count-scaled proxies rank the review-based engine above silent refusal, whereas whole-episode effects relative to allow reverse to +20 pp vs. +2 pp), while candidly keeping our claims strictly scoped to offline diagnostic evaluation rather than cheaper selection or runtime control (as shown by our 10-task Qwen2.5-14B shared-token study in Appendix Z.7). We also stated clearly the remaining archive/sweep boundaries and acknowledged that simplifying the main-text presentation around the core findings and moving protocol details to the appendix will improve readability.

We remain available if any further clarification would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.