acceptodds
Withdrawn from ICLR 2027

Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Runtime policy verifiers can prevent agents from executing disallowed actions, but their decisions are typically used only for enforcement and discarded afterward. We ask whether these decisions can instead become verifiable policy rewards for training the agents they guard. We introduce Policy-Verified Self-Training (PVST), a post-training paradigm in which an agent generates its own interaction trajectories and an executable policy verifier, here a deterministic gate compiled from the natural-language policy, deterministically evaluates whether each action complies. These verdicts provide verifiable training signals: compliant trajectories are reinforced, and violations are used to construct negatives for preference optimization. PVST thus turns an existing runtime guardrail into a teacher for the same agent: beyond the gate and the benchmark’s task-success signal, it needs no stronger teacher model, no human preference labels and no attack-specific supervision, although rollouts are collected in the benchmark’s attacked environment. We evaluate PVST on tool-using agents under prompt injection. On heldout AgentDojo tasks, policy-verified preference training of Qwen3-8B reduces violating trajectories from 37.2% to 10.3 ± 0.9% (three seeds) and executed attacks from 12.4% to 0.5% at equal or higher task utility, compared with 22.0% violating trajectories when negatives are selected by an attack oracle and 31.9 ± 0.9% with matched negatives that contain no violation (a success-only control that never consults the gate lands at 20.2%). This separation indicates that the improvement comes from the policy information encoded by the verifier rather than from preference optimization alone. The learned behavior also transfers beyond the training distribution: it holds under an attack template never used in training (37.7% to 11.0%), violations decrease from 34.8% to 16.7% on a suite whose rule set and tasks were excluded from training, and measurable improvements emerge from a few dozen preference pairs. When the original verifier is retained at deployment, its intervention rate falls from 39.2% to 8.8%, reducing reliance on runtime enforcement while preserving a deterministic safety backstop; the compliance result replicates on a second trainee (Qwen3-4B-Instruct-2507, 37.3% to 10.2%). Our analysis identifies the condition under which the verdict is most informative: guarded actions must appear in both permitted and forbidden contexts. Executable policies can therefore serve not only as runtime constraints but as a scalable source of verifiable rewards for self-training, and they motivate learning objectives that exploit the verifier’s finer-grained rule- and predicate-level structure.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, tool-use agents, policy compliance, verifiable rewards, rejection sampling fine-tuning, compiled policy gates, reinforcement learning with verifiable rewards, agent safety, tau-bench, prompt injection, guardrails, distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZUXVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The idea of this paper is to have an agent generate trajectories and then have a gate decide for each action whether the action complies with some safety specification. The trajectories are collected without the gate. This compliance information is then used for fine-tuning the agent to adhere to the specifications. The gate remains active at deployment. The main contribution is this fine-tuning. Specifications are generated via an anonymous paper.

Critical strengths

S1: The evaluation is carefully designed to separate the impact of different learning variants. S2: The information required for retraining is cheaply obtainded from a running agent as the gate anyway needs to be in place.

Critical weaknesses

W1: The newly presented methodology is thin. Retraining an agent to reduce the intervention of a safety mechanism is not a majorly new idea. W2: The bibliography of the paper is rather short. I briefly investigated whether further relevant literature exists and quickly came across the following:

  • Ta, Anh, Junjie Zhu, and Shahin Shayandeh. "Reinforced Agent: Inference-Time Feedback for Tool-Calling Agents." Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM). 2026.
  • Agarwal, Aradhye, et al. "Learning when to act or refuse: Guarding agentic reasoning models for safe multi-step tool use." arXiv preprint arXiv:2603.03205 (2026).
  • Mu, Tong, et al. "Rule based rewards for language model safety." Advances in Neural Information Processing Systems 37 (2024): 108877-108901.
  • Guan, Melody Y., et al. "Deliberative alignment: Reasoning enables safer language models." arXiv preprint arXiv:2412.16339 (2024).
Questions for authors
  • What is the relation to the literature named above, and is there a reason for the short section on related work?
  • Why did you not compare with other baselines? One could consider other types of defenses.
Reviewer_3ySrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces policy verified self training (PVST) which uses a deterministic tool-call gate first to label an agent's own trajectories and then uses the positive-trajectory and negative-trajectory (violation and successful) rollouts for RFT and DPO. The authors experimented with Qwen3-8B model to perform PVST on AgentDojo task and show that the violating trajectories fall from 37.2% to 10.3%, and executed attack success falls from 12.4% to 0.5%. The paper also tests an unseen attack template, suite transfer and τ2\tau^2 -bench airline.

Critical strengths
  • The paper provides a practical idea of baking policies (via rule-attributed supervision) into the model.
  • The paper systematically examines the contributions of policy-based filtering, negative-examples selection and violation localization through targeted ablations. On a eval set of 230 task–injection triples, DPO-gate reduces policy-violating trajectories from 38.7% to 6.1%. The improvement is also consistent across three training seeds, with full-set violation rates of 9.2%, 10.7% & 11.0%.
Critical weaknesses
  • Although DPO gate improves the policy compliance, its utility gains are not very consistent. On eval set, average utility across 3 seeds is approximately 36.2%, nearly identical to the base model's 36.1%. Utility on write-requiring tasks does not improve and travel utility drops from 33.3% to 16.7%. This raises concerns about the safety–utility tradeoff, which is under-discussed in the paper.
  • With the method described the model may learn to avoid guarded tools rather than distinguish authorized and unauthorized actions. Table 7 reports that DPO-gate withholds calls in 75.6% of the tested contexts, supporting this argument. The paper lacks detailed results on benign task utility and authorized-write recall, making it difficult to understand whether the learned behavior reflects genuine policy understanding or overly caution.
  • Same compiled policy rules generate training labels and also measure policy violations. The reported improvements demonstrate better compliance with the verifier, but not necessarily with the original NL policies. Independent evaluations of false approvals, false rejections and policy correctness are needed.
  • Generalization? Most experiments use AgentDoj and the held-out suite shares some policy predicates and tools with the training suites. Only a single unseen attack template is evaluated with several generalization results reported for a single seed. The τ²-bench experiments use a different pipeline involving larger-model distillation, so they do not independently validate the proposed PVST method either.
Questions for authors
  • what are write-task and non-write-task utility with denominators on Table 12's common set and by suite? What is the clustered interval for DPO-gate's utility difference from base and RFT-gate?
  • RFT gate and RFT succ reaches similar compliance results, but RFT-gate shows higher task utility (41.3% vs 34.3% on common set). Is this utility difference consistent across multiple seeds and is it statistically significant under task-clustered bootstrap CI?
  • While constructing DPO preference pairs, do the chosen and rejected trajectories share the same injected prompt and tool-result prefix, or are they paired solely by task identity?
  • Sec 4 mentions eval on benign tasks but these results no where reported. Could the authors provide benign task utility w/ and w/o the gate to assess over-refusal, if any? Additionally, Table 10 shows similar utility regression when gate is enabled despite substantially fewer interventions after DPO-gate training. Could the authors explain this discrepancy through per-task or per-intervention analysis?
Reviewer_YqGZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes using deterministic policy verifiers to guide self-training of LLM agents through rejection-sampling fine-tuning and preference optimization, reducing policy violations.

Critical strengths

The paper addresses an important practical problem by leveraging existing policy verifiers to improve agent compliance without requiring human annotations or stronger teacher models. The experiments show substantial reductions in policy violations while largely preserving task utility.

Critical weaknesses
  1. The method primarily combines existing techniques (rejection-sampling fine-tuning and DPO), with the main contribution being the use of a policy verifier to select and label training examples. The authors should better clarify what is technically novel and demonstrate why this supervision strategy offers advantages over existing approaches.
  2. The experimental results are limited. The results are from only one model family. It is not clear which results are over multiple runs. No standard deviation or seed information. No direct comparison against baseline prompt-injection training methods such as SecAlign or instruction-hierarchy training. Most significant results are from a single dataset, AgentDojo, and only one unseen attack template is evaluated.
  3. Training and testing share injection tasks, while cross-domain evaluations reuse some policy rules and tools. The current experimental evidence do not support generalization to unseen attacks and policies.
Questions for authors
  1. The method is heavily reliant on the policy verifier. The method assumes that the predefined policy verifier accurately identifies violations, but real-world policies may contain ambiguous, incomplete, or incorrect rules. Have the authors considered the affect of this?
  2. How much of the improvement comes from training on attacked environments rather than verifier-based supervision itself?
  3. Could you clarify how well the method generalizes to entirely new policy rules, tools, and attack strategies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.