acceptodds
Under review as a conference paper at ICLR 2027

Learning to Escalate: Do LLM Agents Generalize Governance Policy to Unseen Rules?

Abstract

LLM agents that act through tools are usually held to a written policy by a runtime gate that blocks rule-breaking calls. A gate covers only the rules someone has encoded, so we ask what an agent learns about a policy from reward alone, and whether that learning reaches rules it was never trained on. We train Qwen2.5-7B-Instruct with GRPO on customer-support tasks from τ²-bench's telecom domain, rewarding task success and penalizing policy violations and escalations to a human. To test transfer, we remove entire rule families from every channel through which training could expose them. The held-out rule we study requires the customer's consent before a paid data top-up, and the agent learns only part of it. It learns restraint, making no unauthorized top-ups on any of three seeds. It does not learn the handoff the policy asks for; instead, training on other rules erodes a handoff it had learned during fine-tuning, from 99% of blocked requests to 8%, and a single-seed test points to the escalation cost as a major cause. It also becomes reluctant to top up even when allowed. An agent trained on every rule family avoids all three failures and matches the gate on safety and task success. The held-out agent commits no violations, so a violation rate would call it compliant. We argue that policy evaluations should also check whether the agent hands off what it may not do.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, reinforcement learning, human-in-the-loop, escalation, policy compliance, GRPO, agentic safety

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_6Q5sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether tool-using agents generalize authorization policies to rule families withheld from fine-tuning. Using SFT followed by GRPO on telecom tasks, it distinguishes withholding unauthorized actions from performing the required escalation. On the held-out refuel family, unauthorized actions decrease, but escalation and authorized refueling also deteriorate. The paper attributes much of the escalation loss to the escalation penalty and argues that violation rates alone are insufficient to evaluate compliance.

Critical strengths
  • Distinguishing restraint from the alternative action required by a policy is useful.

  • The substantial loss of SFT-taught escalation after RL is an interesting empirical observation, and the escalation-cost ablation provides a plausible explanation.

Critical weaknesses
  1. The experiments do not sufficiently distinguish policy generalization from suppression or forgetting of the held-out workflow. CLEAN training excludes the target family’s tasks, while prompt removal deletes not only the authorization constraint but also the refuel procedure and cap (§4). Authorized refueling subsequently declines from 100% to 73% on average, alongside the reduction in unauthorized refueling. The agent retains some discrimination, but this does not establish that the improvement comes from transferring an authorization principle. Furthermore, all refuel traps are MMS tasks (§8), leaving task-specific behavior as an alternative explanation. Matched tasks differing only in consent, together with controls that preserve procedural instructions while removing the authorization clause, are needed to support the central interpretation that “restraint transfers.”

  2. The claimed clean holdout does not establish independence from the held-out policy during teacher-data construction. Every teacher trajectory was generated with the full policy, including the held-out clauses. The CLEAN datasets are subsequently constructed by dropping target-family tasks and removing the clauses from the student prompts (§8). These operations remove explicit exposure but do not establish that the retained teacher behavior is unaffected by the held-out policy. This does not prove leakage occurred; it means the protocol cannot rule it out, despite describing the family as absent from every training channel. Generating the corresponding teacher data without the target rules would provide a more convincing control.

  3. The claim of matching a runtime gate’s safety is not supported by the statistical argument. Section 6.4 argues that the learned agent reaches the gate’s safety because its violation-rate confidence intervals include zero. An interval containing zero does not establish equivalence to zero, particularly when the learned agent has observed violations and the gate blocks encoded violations by construction. Likewise, overlapping task-success intervals do not establish equivalence. The results support low observed violation rates and comparable task-success estimates on this evaluation, rather than safety parity with enforcement. A claim of practical equivalence requires a defined tolerance and an appropriate statistical analysis.

More broadly, one model and domain, only one held-out family with informative restraint results, and mostly single-seed controls provide insufficient experimental support for general conclusions about governance-policy transfer.

Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6Q5s

Thank you for the detailed review. We ran both controls you asked for and reran the gate comparison on three seeds.

1. Generalization or suppression. We reran the same tasks and changed only the consent sentence in the ticket: added to the 32 MMS traps, removed from the 8 authorized data tasks. With the rule hidden, Clean tops up on 0% of MMS tasks without consent and 59% [53, 64] with it, and on 73% of data tasks with consent and 5% [1, 10] without (three seeds, 95% intervals). The top-up follows consent, not the task type, and the agent is not suppressing the workflow.

We also kept the refuel procedure in the prompt and removed only the consent sentence. Clean tops up on 5% [3, 8] of traps and 86% of authorized tasks. You were right about the confound: with the procedure deleted as well, authorized top-ups fall to 73%.

Two limits. The SFT model already responds to consent (13% to 80% on MMS tasks). RL on other rules does not create that distinction. It cuts the top-ups made without consent. And the consent sentence mentions refueling, so it may act as a cue as well as a permission. We will drop "restraint transfers" and state the claim this way.

2. Teacher data. We agree. The teacher saw the full policy and we cannot rule out its influence. In the 478 Clean training episodes, none calls the refuel tool. Six mention refueling, each in a closing remark that none was needed, and none states the consent rule. We will say "no explicit exposure" in place of "absent from every channel". We did not regenerate the data.

3. Gate. "Matches" overstated it. Over three seeds, All succeeds on 0.89 and 0.86 of non-target tasks, against the gate's 0.88 and 0.86. The differences are 0.00 [-0.05, 0.06] and 0.00 [-0.08, 0.07], too wide to claim equivalence at a five-point margin. All also made 7 violating calls in 1,872 (0.4%). The gate makes none by design. We will remove "matches" and "parity".

4. Scope. One model, one domain and one informative rule family are real limits, and the abstract will say so. The no-penalty control now has three seeds: Visible hands off on 77% of traps without the penalty (68 to 92% by seed) and 8% with it.

Thank you for running the additional controls and clarifying the claims you plan to revise. The matched tasks show that the agent remains sensitive to consent, but that sensitivity was already present after SFT. On the MMS tasks, RL reduced refueling both without consent (from 13% to 0%) and with consent (from 80% to 59%). These results address my concern about task type, while leaving limited evidence that RL transfers an authorization principle across rule families.

I appreciate the audit of the teacher data and agree with the decision to withdraw the claim of safety parity with the runtime gate. The effect of the escalation penalty remains an interesting finding. However, given the narrower claims and the limited evaluation scope—one model, one domain, and one held-out family with informative results on restraint—I find the overall contribution too limited to recommend acceptance. I am therefore maintaining my score.

Reviewer_EAPmOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper Learning to Escalate: Do LLM Agents Generalize governance Policy to Unseen Rules, details the experiments in a Telecom case from τ2-bench, where agents make the tool calls, based on predicates, on what should happen with the customer support ticket at hand. The experiments are on the central question of what an agent learns from rewards, and how far into other rules it has not been trained on, the policies reach. The case at hand, a customers data top up, requires customer consent - the results show almost an over-cautious agent - and that learned compliance extends even to the rules that are not encoded. The main claim is that having agents ask for help requires training.

Critical strengths
  • the research idea is generally interesting, testing how restraint and escalation in support-ticket-handling ai agents are influenced by reinforcement learning from verifiable rewards
  • the presentation is well done
Critical weaknesses
  • The paper needs stronger contextualization with the related work. The story is introduced nicely, but this is not the first work to test GRPO beyond task success or training-level penalties for policy violations (see below). Similarly, there is rich literature on abstention that could be taken into account.
  • Lack of strong baselines. The paper needs to consider other approaches as baselines and be more clear about what the paper's findings add to the literature, e.g., Li & Shi (2026) RLVP; Wang et al. (2026) SENTINEL; Zeng et al. (2026) LOGIGEN.
  • Only 1 model (Qwen2.5 ~7B) is tested, which is warm-started by Deepseek-v4-pro but the teacher's direct results on the task are not measured.
  • Restraint could be influenced by task type: all traps are MMS tasks, and SFT already refuels 13% on traps vs. 100% on authorized tasks.
  • ablations are done with a single seed. robustness of these ablation results is thus questionable
  • To strengthen the paper, I'd like to see how does DeepSeek-v4-pro perform on the task (i.e., the model that provided the trajectories for SFT warm start) and one penalty-based baseline from the related work (e.g., RLVP)
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer EAPm

Thank you. The paper's finding, briefly: an agent can commit no violations and still fail the policy, because the policy also requires a handoff. RL on other rules removed a handoff the agent already had (99% to 8% of blocked requests) while its violation rate stayed at zero. Our reply to each point follows.

1. Related work. Penalizing violations during RL is prior work, and §2 says so: it cites RLVP and SENTINEL and notes that both "test on the rules they train on". What we add is the test on rules removed from the training data, the prompts and the reward. We will make that contrast explicit in the introduction. We will also cite LOGIGEN, which trains for task success on generated tasks and holds no rules out, and the abstention survey of Wen et al. (2025). §2 already covers AgentAbstain, learning to defer and over-refusal.

2. Baselines. The paper asks what a penalty-trained agent does on unseen rules, and proposes no training method of its own. RLVP, SENTINEL and LOGIGEN hold no rules out, so they cannot be baselines for that question. Our reward is a penalty of the same kind: it pays for the outcome and charges each violating action. Table 4 has the comparison you ask for. Against a success-only reward, the full reward cuts the violation rate from 0.028 to 0.002 on the roaming task set and from 0.014 to 0.001 on the refuel set, with success of 0.92 and 0.89 against 0.93 and 0.92.

3. Teacher and model. DeepSeek V4 Pro completed 1,965 of the 2,154 distillation tasks (91%) and broke no policy rule in any episode. Nearly all the rest never closed the ticket. We did not run it on the evaluation tasks. We will report this in Appendix B. One 7B student model is a limitation, and we agree.

4. Task type. This is a fair concern: every trap is an MMS task and every authorized task is a mobile-data task. We are testing it directly by rerunning the same tasks with only the consent sentence in the ticket added or removed.

5. Seeds. Two more seeds each of the no-penalty run and the all-rules agent are training, which brings both to three.

We will post both sets of results in this thread. If a soundness concern remains beyond these five points, please tell us and we will address it.

AuthorsOpenReview ↗

Results for Reviewer EAPm

The two results we promised under points 4 and 5 are in.

4. Task type. We reran the same tasks and changed only the consent sentence in the ticket. With the rule hidden, the held-out agent tops up on 0% of MMS traps without consent and 59% [53, 64] with it. On mobile-data tasks it tops up on 73% with consent and 5% [1, 10] without (three seeds, 95% intervals). Top-ups follow consent inside each task type, so the task type does not explain the restraint.

5. Seeds. The no-penalty run and the all-rules agent now have three seeds each. Without the handoff penalty, the agent hands off on 77% of traps (68 to 92% by seed), against 8% with it (2 to 17%). The all-rules agent hands off on 88 to 100% of traps, tops up on 97 to 100% of authorized tasks, and never tops up without consent.

The text changes from points 1 to 3 will be in the revised PDF we upload before the deadline.

I appreciate the author's effort to run more experiments. In my view, however, with that many new results, the paper is better off to go through another review cycle. The single student model limitation can then also be alleviated.

Reviewer_FSPHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper argues that the success of fine-tuned policies in LLMs should be measured not only by the rate at which tasks are rejected but also by whether the correct alternative action is performed. For this purpose, they evaluate Qwen2.5-7B in a single setting to measure how likely a model is to follow a rule depending on where/if it encountered it during training. They find that the agent performs worse if the target rule is visible during reinforcement learning. More (correct) escalations can be achieved when they are not disincentivised, but the refueling behavior in allowed cases remains deteriorated. The authors partially attribute this effect to the escalation cost, but do not offer an explanation for the remaining gap. They further show that if trained on all rules, the model matches the introduction of a gate.

Critical strengths

The introduction of a detailed hold-out protocol provides an environment that allows for this analysis of model behaviour given different training settings. Even though the reasons for the observed effects are partially unclear, an evaluation of individual signals can be obtained, laying the groundwork for future investigation that includes the nuance of correct alternative actions.

Furthermore, the paper provides an honest and comprehensive account of its limitations and clearly highlights gaps in explanations for the obtained results.

Critical weaknesses

A fundamental limitation of the presented work is, that the CLEAN model is better at following the trained policy than the VISIBLE one. While escalations can be boosted without a penalty, this appears to simply increase the general escalation rate, not only correct cases in unseen settings. The failure to positively influence the model limits the insight obtained by ablating the different hold-out scenarios.

This is accompanied by a partial lack of evaluation over only a single seed, which is especially concerning considering the large variance in the obtained results (such as 56-94% for authorised refuels). These uncertainties are also not visible in the main text's figures and tables, which can lead to misinterpretation by readers upon first inspection. Combined with the choice of an older model, it is unclear whether the results transfer to another domain or evaluation setting.

Questions for authors
  • Why do you expect the model to infer completely unseen requirements, such as travel conditions in line 370? Does this not fundamentally oppose our expectation that the model follows the provided policy?
  • Why was this older LLM chosen for evaluation? Simply due to memory/compute constraints, or do newer models perform better at this task?
  • 358-359: I am not sure I understand what you are trying to convey in the sentence: "The runs also differ in task mix: 8 of VISIBLE’s 36 GRPO tasks require a transfer, all of them τ2\tau^2-bench’s own transfer tasks and none blocked by a consent rule, whereas 31 of ALL’s 74 do, 23 of them consent-blocked." Is it possible that a cause of weaker performance in VISIBLE is that there are fewer handoffs in the training data? But we should compare to CLEAN, not ALL here.
1 comment
AuthorsOpenReview ↗

Response to Reviewer FSPH

Thank you. We ran more seeds and one new control, and they speak to both of your main points.

1. What does improve the agent. Two things, each now on three seeds.

Training on the rule works. All hands off on 88 to 100% of traps, tops up on 97 to 100% of authorized tasks, and never tops up without consent.

Removing the handoff penalty also works. Visible then hands off on 77% of traps (68 to 92% by seed), against 8% with the penalty. This is not a general rise in handoffs. They go up 69 points on traps and 12 points on tasks that do not need one (2% to 14%, still below the SFT model's 20%). Needed handoffs stay at 96%. So most of the erosion comes from the penalty, which charges every handoff, including those the policy requires on rules the reward does not cover.

2. Clean against Visible. Both lose handoffs against their own starting model. Visible goes from 99% to 8%, and Clean from 52% to 23%. Clean ends higher, but it fell too, so hiding the rule does not protect the handoff. Neither run rewards it.

3. Seeds and uncertainty. The main-text figures and tables will show seed ranges or intervals. The extra seeds changed one result. Authorized top-ups without the penalty were 56%, 100% and 0% by seed, so we will report that as seed-dependent, not as a consistent fade. One 7B model and one domain limit how far the results carry, and the abstract will say so.

Your questions.

Line 370. We do not expect it. Roaming is our null result: being abroad appears in no other rule, and nothing transfers without the rule text. Refuel differs because consent is stated in the ticket. In a new control we changed only that sentence. With the rule hidden, the held-out agent's top-ups on the same tasks went from 0% to 59% when consent was added, and from 73% to 5% when it was removed.

Choice of model. Compute. Qwen2.5-7B trains with GRPO on a single H100, which let us afford about twenty runs. We did not test newer models.

Lines 358-359. You are right that Clean is the better comparison. Clean and Visible train on the same 36 GRPO tasks, so task mix cannot explain the gap between them, and the no-penalty run changes only the cost. The sentence was explaining why All is not a clean comparison for Visible. The revision rewrites it.

Reviewer_FgJgOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies a customer-support AI that can change accounts through software tools. It asks whether training on some service rules helps the agent handle a rule it was never trained on: before adding paid mobile data, it needs the customer’s permission. If permission is missing, it should make any other available fixes and pass the case to a person.

The authors test the agent with and without that rule in its instructions. When the rule has been withheld from training and is also absent at test time, the agent makes no unauthorized top-ups across three runs, but hands off only 23% of the blocked cases. In a separate setup, an agent that had learned to hand off 99% of these cases from examples does so only 8% of the time after reinforcement learning on other rules—even when the rule is shown. The paper’s central point is that a zero violation rate can hide a failure to handle the customer’s request.

Critical strengths

The paper identifies a real gap in how these agents are evaluated. An agent can avoid an unauthorized charge and still leave the customer’s case unresolved. In the rule-shown test, one agent makes no unauthorized top-ups but handles only 7% of the blocked cases as the policy requires. The authors make a careful effort to withhold the target rule during training. They remove it from examples, policy text, practice tasks, and rewards, then test what happens when the rule is either shown or hidden. The handoff decline is especially useful evidence: it follows the same starting model through reinforcement learning and is reported across three runs. Violations are checked against actual tool calls rather than judged by another language model.

Critical weaknesses

It is unclear whether reinforcement learning taught the agent the consent rule. Before that stage of training, the held-out agent already avoids unauthorized top-ups in 87% of cases. Afterward it avoids all of them, but it also becomes less willing to top up when the customer has given permission: that rate falls from 100% to 73%. The result could reflect broader caution about charging customers. The roaming holdout does not resolve this question because even untrained models rarely attempt the forbidden action on its test cases.

The explanation for the lost handoffs is plausible but not settled. Removing the penalty for handing off preserves many more handoffs, but that experiment has only one run. The comparison with the agent trained on all rules also changes how often training tasks require a handoff: 8 of 36 tasks versus 31 of 74. Separately, removing refuel examples may have removed many handoff demonstrations from the fully held-out agent’s initial training. Counts of those demonstrations, more runs without the penalty, and a comparison with a balanced task mix would help separate these effects.

The broader claim rests mainly on one informative held-out rule. The consent sentence comes from a workflow document but had to be added to the agent’s policy prompt; the benchmark’s original answers conflict with it on the blocked cases, so the authors use an adjusted success measure. The roaming cases offer little evidence about restraint because the forbidden action is rarely attempted. The results also come from one domain and one model. The paper’s claim about how a rule-checking gate would behave around the held-out agent is an inference; that combination was not tested.

Questions for authors

Could you repeat the run without the handoff penalty across three seeds? How many handoff demonstrations remain in each initial training set after refuel tasks are removed? Could you test another held-out rule with cases that genuinely tempt the agent to break it, as well as cases where the action is allowed?

2 comments
AuthorsOpenReview ↗

Response to Reviewer FgJg

Thank you. We agree with all three weaknesses; below is what we did about each. The text changes will be in the revised PDF we upload before the deadline.

1. Rule transfer or general caution. Agreed: our design cannot separate them. The held-out agent makes no unauthorized top-ups (13% → 0%, three seeds) and still tops up on 73% of authorized tasks, so it tells the cases apart, but it is also more hesitant overall and we could not trace the change to a trained rule. The abstract and §6.1 will say "restraint emerges, with reluctance to act when allowed" in place of "restraint transfers".

2. Why handoffs are lost.

  • Demonstrations. Episodes containing a handoff: Full 814/1,965 (41%), Matched 211/478 (44%), No-Roam 219/478 (46%), No-Refuel 6/478 (1.3%). You were right: Clean starts with almost no handoff demonstrations, so its low rate on traps cannot be attributed to RL alone, and §6.2 will say so. It does have the skill (92–94% on the benchmark's own transfer tasks). The erosion result is unaffected: Visible starts from Matched and falls from 99% to 8%.
  • Seeds. Seeds 1 and 2 of the no-penalty run are training. We will post them here whatever they show.
  • Task mix. The no-penalty run uses Visible's 36 tasks and changes only the cost, so it separates the cost from the task mix. The comparison with All does not, and the text will say so.

3. Breadth. Agreed: one informative family, one domain, one model. The abstract and conclusion will state the claim at that scope. Our statement about a gate around the held-out agent was an inference, and we are removing it. A second held-out rule with tempting cases is not possible on this benchmark, whose other families have four evaluation tasks between them; we will list this as the main limitation.

AuthorsOpenReview ↗

Results for Reviewer FgJg

The runs we promised are in.

1. Rule transfer or general caution. We said our design could not separate these. A new control does. We reran the same tasks and changed only the consent sentence in the ticket. With the rule hidden, the held-out agent tops up on 0% of MMS traps without consent and 59% [53, 64] with it. On mobile-data tasks it tops up on 73% with consent and 5% [1, 10] without (three seeds, 95% intervals). So it is not avoiding charges in general. Its top-ups follow consent. Two caveats: the SFT model already responds to consent (13% to 80% on the traps), so RL on other rules mainly removes the top-ups made without it, and the reluctance on authorized tasks remains.

2. No-penalty run, three seeds. Visible hands off on 77% of traps without the penalty (70%, 92% and 68% by seed), against 8% with it (2 to 17%) and 99% for its starting model. Handoffs on tasks that do not need one rise from 2% to 14%, below the starting model's 20%. One result did not hold up. Authorized top-ups without the penalty were 56%, 100% and 0% by seed, so we will report that as seed-dependent, not as a consistent fade.

3. All-rules agent, three seeds. It hands off on 88 to 100% of traps, tops up on 97 to 100% of authorized tasks, and never tops up without consent.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised PDF uploaded: summary of changes

We have uploaded the revised PDF. Thank you all for the reviews. The changes, with where to find them:

  • New controls (6Q5s, FgJg, EAPm, FSPH). Changing only the consent sentence in the ticket, and keeping the refuel procedure while removing the consent sentence from the prompt. §6.1, Tables 9 and 10.
  • More seeds (FgJg, FSPH, 6Q5s, EAPm). The no-penalty run and the all-rules agent now have three seeds each. §6.2, Figure 2, Table 8.
  • Claims narrowed. "Restraint transfers" is gone: the fine-tuned model already responds to consent, and RL on other rules removes the top-ups made without it. "Matches the gate" is gone: §6.4 and Table 11 give the three-seed comparison. Authorized top-ups without the penalty are reported as unstable across seeds.
  • Uncertainty in the main text (FSPH). Figure 2 shows seed ranges, Table 3 shows intervals, and Table 4 has a seed-range row.
  • Exposure and teacher (6Q5s, EAPm). "No explicit exposure" replaces "absent from every channel", with a teacher-data audit and teacher completion numbers in Appendix B.
  • Related work and scope (EAPm, FgJg). The contrast with penalty-trained methods is explicit in the introduction, two citations are added in §2, and the abstract and conclusion state the scope: one rule family, one domain, one model.

Main-text table numbers are unchanged from the submitted version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.