- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether tool-using agents generalize authorization policies to rule families withheld from fine-tuning. Using SFT followed by GRPO on telecom tasks, it distinguishes withholding unauthorized actions from performing the required escalation. On the held-out refuel family, unauthorized actions decrease, but escalation and authorized refueling also deteriorate. The paper attributes much of the escalation loss to the escalation penalty and argues that violation rates alone are insufficient to evaluate compliance.
Critical strengths
-
Distinguishing restraint from the alternative action required by a policy is useful.
-
The substantial loss of SFT-taught escalation after RL is an interesting empirical observation, and the escalation-cost ablation provides a plausible explanation.
Critical weaknesses
-
The experiments do not sufficiently distinguish policy generalization from suppression or forgetting of the held-out workflow. CLEAN training excludes the target family’s tasks, while prompt removal deletes not only the authorization constraint but also the refuel procedure and cap (§4). Authorized refueling subsequently declines from 100% to 73% on average, alongside the reduction in unauthorized refueling. The agent retains some discrimination, but this does not establish that the improvement comes from transferring an authorization principle. Furthermore, all refuel traps are MMS tasks (§8), leaving task-specific behavior as an alternative explanation. Matched tasks differing only in consent, together with controls that preserve procedural instructions while removing the authorization clause, are needed to support the central interpretation that “restraint transfers.”
-
The claimed clean holdout does not establish independence from the held-out policy during teacher-data construction. Every teacher trajectory was generated with the full policy, including the held-out clauses. The CLEAN datasets are subsequently constructed by dropping target-family tasks and removing the clauses from the student prompts (§8). These operations remove explicit exposure but do not establish that the retained teacher behavior is unaffected by the held-out policy. This does not prove leakage occurred; it means the protocol cannot rule it out, despite describing the family as absent from every training channel. Generating the corresponding teacher data without the target rules would provide a more convincing control.
-
The claim of matching a runtime gate’s safety is not supported by the statistical argument. Section 6.4 argues that the learned agent reaches the gate’s safety because its violation-rate confidence intervals include zero. An interval containing zero does not establish equivalence to zero, particularly when the learned agent has observed violations and the gate blocks encoded violations by construction. Likewise, overlapping task-success intervals do not establish equivalence. The results support low observed violation rates and comparable task-success estimates on this evaluation, rather than safety parity with enforcement. A claim of practical equivalence requires a defined tolerance and an appropriate statistical analysis.
More broadly, one model and domain, only one held-out family with informative restraint results, and mostly single-seed controls provide insufficient experimental support for general conclusions about governance-policy transfer.
Questions for authors
See weaknesses
2 comments
Thank you for running the additional controls and clarifying the claims you plan to revise. The matched tasks show that the agent remains sensitive to consent, but that sensitivity was already present after SFT. On the MMS tasks, RL reduced refueling both without consent (from 13% to 0%) and with consent (from 80% to 59%). These results address my concern about task type, while leaving limited evidence that RL transfers an authorization principle across rule families.
I appreciate the audit of the teacher data and agree with the decision to withdraw the claim of safety parity with the runtime gate. The effect of the escalation penalty remains an interesting finding. However, given the narrower claims and the limited evaluation scope—one model, one domain, and one held-out family with informative results on restraint—I find the overall contribution too limited to recommend acceptance. I am therefore maintaining my score.
Response to Reviewer 6Q5s
Thank you for the detailed review. We ran both controls you asked for and reran the gate comparison on three seeds.
1. Generalization or suppression. We reran the same tasks and changed only the consent sentence in the ticket: added to the 32 MMS traps, removed from the 8 authorized data tasks. With the rule hidden, Clean tops up on 0% of MMS tasks without consent and 59% [53, 64] with it, and on 73% of data tasks with consent and 5% [1, 10] without (three seeds, 95% intervals). The top-up follows consent, not the task type, and the agent is not suppressing the workflow.
We also kept the refuel procedure in the prompt and removed only the consent sentence. Clean tops up on 5% [3, 8] of traps and 86% of authorized tasks. You were right about the confound: with the procedure deleted as well, authorized top-ups fall to 73%.
Two limits. The SFT model already responds to consent (13% to 80% on MMS tasks). RL on other rules does not create that distinction. It cuts the top-ups made without consent. And the consent sentence mentions refueling, so it may act as a cue as well as a permission. We will drop "restraint transfers" and state the claim this way.
2. Teacher data. We agree. The teacher saw the full policy and we cannot rule out its influence. In the 478 Clean training episodes, none calls the refuel tool. Six mention refueling, each in a closing remark that none was needed, and none states the consent rule. We will say "no explicit exposure" in place of "absent from every channel". We did not regenerate the data.
3. Gate. "Matches" overstated it. Over three seeds, All succeeds on 0.89 and 0.86 of non-target tasks, against the gate's 0.88 and 0.86. The differences are 0.00 [-0.05, 0.06] and 0.00 [-0.08, 0.07], too wide to claim equivalence at a five-point margin. All also made 7 violating calls in 1,872 (0.4%). The gate makes none by design. We will remove "matches" and "parity".
4. Scope. One model, one domain and one informative rule family are real limits, and the abstract will say so. The no-penalty control now has three seeds: Visible hands off on 77% of traps without the penalty (68 to 92% by seed) and 8% with it.