acceptodds
Under review as a conference paper at ICLR 2027

RePolicy: Learning to Select and Apply Safety Policies for Agent Safeguards

Abstract

Agent safeguards must determine whether an agent’s behavior violates the safety rules of the current task, environment, and user. Because these rules vary across contexts, a safeguard needs to select the relevant policy before judging the behavior. Existing policy-aware safeguards can apply a supplied policy, but generally assume that the correct policy is already known. They therefore do not learn how to select a policy from a library, even though an incorrect choice can lead to an incorrect safety judgment. We introduce RePolicy, a safeguard that jointly learns policy selection and safety assessment. Given an agent trajectory and a set of candidate policies, RePolicy selects the policy that best matches the situation, applies its requirements, and predicts whether the behavior is safe. We construct PolicyTraj-20K, a dataset of more than 20,000 agent trajectories annotated with applicable policies, safety labels, and explanations. We train RePolicy with supervised demonstrations followed by reinforcement learning, while randomizing policy order, identifiers, and distractors to reduce shortcut learning. Across six agent-safety benchmarks and 19 baselines, RePolicy-4B achieves leading overall safety-detection performance and improves policy-selection accuracy. These results show that safeguards can learn which safety requirements to use for assessing agent behavior. Code is available at https://anonymous.4open.science/r/RePolicy.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
llm agent, agent safeguard, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LaRzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LLM agents can take harmful actions, so a guard model labels recorded trajectories as safe or unsafe, usually against written safety policies. Existing guards are handed the right policy, but in practice you have a library and must pick one, and a wrong pick means a wrong judgment. The proposed RePolicy reads short policy descriptions, fetches the relevant one, then judges the trajectory against it. It’s trained with SFT and then RL on a synthetic 20k-trajectory dataset, with shuffled policy IDs, order and decoys so it learns to match on content.

Critical strengths

Strengths

  • Splitting “which rules apply?” from “was this a violation?” is a real, practical distinction. Most policy-aware guards quietly assume the first part is solved, and making it an explicit, inspectable step means you can actually audit what the guard consulted. Reminds me of the arguement used in formal verifcation works.
  • Good metric selection beyond the typical F1score: decoy selection rate, number of policies retrieved, FP/FN breakdowns, library-size sensitivity. Experimental evaluation seems decent.
  • Reproducibility is strong. Code, splits, policy library, rewards, hyperparameters and a very detailed baseline protocol are all released.
Critical weaknesses

Weaknesses

  • Unsafe-class F1 ignores true negatives, so over-flagging is never penalized. An “always unsafe” classifier scores about 79.8, so RePolicy’s 88.15 is only about 8 points above trivial, and it’s below trivial on OpenAgentSafety and AgentHazard. No per-benchmark false-positive rates are reported.
  • The policy library is fixed at 30 policies for both training and test. A 4B model can memorize them, so “selection” may really be 30-way classification, and nothing tests unseen, rewritten, or user-defined policies, which is exactly the motivating scenario.
  • Correct me if I am wrong but the policy reward is defined only for a single policy, yet the model retrieves about 3.2 on average. If the reward just checks that the right one is included, retrieving more is a reward hack, and the rising retrieval count looks suspiciously like that.
  • No error bars or confidence intervals.
Questions for authors

Questions

  • Unsafe-class F1 ignores true negatives. Can you report FPR, FNR, balanced accuracy (or macro-F1) and AUROC on each of the six benchmarks, not just on your synthetic validation set?
  • What does an “always unsafe” classifier score per benchmark and overall? Given your split sizes it’s roughly 79.8 Overall, and above RePolicy’s score on OpenAgentSafety and AgentHazard. How should readers interpret the 88.15 against that?
  • Can you report the pooled F1 you already compute (Eq. 10), and Overall with AgentHazard excluded, since it has no safe examples and can’t penalize false alarms?
Reviewer_LiGmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Effective agent safeguards must learn which policy applies to a given situation and then assess the agent’s trajectory against that policy. Both tasks are crucial, since selecting the wrong policy or failing to detect a violation can undermine safety. This paper addresses these tasks through RePolicy, which jointly learns to select an applicable policy and assess the safety of an agent’s trajectory under that policy. To train policy selection and safety assessment, RePolicy constructs PolicyTraj-20K with over 20000 filtered LLM-generated trajectories spanning diverse tools and environments. Each trajectory is paired with an audited operation-specific policy, a rationale, and a safety label. During cold-start training, each sample’s applicable policy is mixed with distractors to form a local policy library. The SFT objective supervises both selection of the correct policy for the given trajectory and generation of the reference rationale and safety label after receiving that policy’s content. The cold-start model is then further trained through GRPO using feedback on its own generated policy selections and safety predictions.

Contributions

  • Jointly learning policy selection and safety assessment for agent safeguard.
  • Constructs PolicyTraj-20K for learning policy selection and safety assessment.

RePolicy is evaluated across 6 agent-safety benchmarks against 19 external systems using the Unsafe F1 metric. The ablation study further clarifies the contribution of its components and code implementation has been supplied to support reproducibility.

Critical strengths

RePolicy explicitly trains safeguards to identify applicable policies and assess agent behavior under them. This important safeguarding capability is central to my overall assessment. Furthermore, evaluation across six agent-safety benchmarks against 19 external systems provides substantive evidence of its effectiveness beyond the training dataset. Component ablations further support the value of the combined approach.

Critical weaknesses

None. Separate reward ablations and reward weight sensitivity tests would strengthen the analysis, but their absence does not outweigh the contribution and cross-benchmark results. The authors acknowledge limitations in rationale faithfulness and multi-policy conflict resolution, which do not undermine the main claims.

Questions for authors
  • Was rationale faithfulness evaluated before and after GRPO? Since rationales are supervised during SFT but not independently verified by the GRPO rewards, such evidence would clarify whether their faithfulness is preserved.
  • How were the reward weights selected? Are reward-component ablations or weight-sensitivity results available to clarify each reward’s contribution?

Answers addressing these concerns would help me assess the method more confidently and I would be happy to increase my rating.

Reviewer_YUyXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces RePolicy, a methodology for agent guardrails that not only executes a given safety policy on an agent trajectory, but also autonomously retrieves applicable policies from a pool when one is not readily selected. The paper shows that with a dataset curated for this task, PolicyTraj-20k, a finetuned Qwen-3-4B-Instruct model achieves competitive performance on a number of agent guardrail benchmarks compared to closed source general models, open-source general models, and open-source guard-focused models. The proposed dataset also allows both SFT and RL finetuning, where RL tuning further boosts the model's performance.

Critical strengths
  • The proposed dataset is effective at improving a small base model in enforcing policy compliance on agent trajectories over multiple datasets.
  • The policy invocation method shows improvements in the setting where distractor policies are provided, which is also shown on a number of datasets.
  • The evaluation is largely performed in a held-out setting, which is crucial in assessing the generalizability of the proposed approach.
Critical weaknesses
  • The proposed setting misses the key challenges of multi-policy enforcement in the real world, and as a result the proposed eval setting is somewhat biased towards amplifying the efficacy of the proposed approach. In reality, an agent rarely needs to execute in a setting where a large pool of policies are provided, but it's unclear which ones are relevant: the kinds of data the agent handles and the tools and interfaces the agent operates on will often be directly attached to the policies applicable. The realistically challenging problems typically arise when multiple policies are potentially applicable to the same agent trace: a) when different parts of an agent trace trigger different policies, b) when the same (part of) agent trace triggers more than one policy that might contradict each other, and/or c) when the policies do not contradict each other, but the applicability of each policy to (parts of) the agent's trajectory are not readily discernible. By taking a simplistic approach to data synthesis and evaluating primarily on guardrail benchmarks that are not policy-aware, this paper misses the opportunity to address a realistic problem but instead focuses on somewhat of a strawman problem.
  • As a result of the above point, the proposed approach is somewhat simplistic as well -- a single-shot retrieval, invocation, and application would solve most of the problem. The fact that RL helps here is well-established in retrieval-augmented generation literature (e.g., [1]). The fact that synthetic data significantly improves a small model in policy-aware guardrail capabilities is also established in prior work [2].

[1] Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, Jing Jiang. R3: Reinforced Reader-Ranker for Open-Domain Question Answering. AAAI 2018. [2] Xiaofei Wen, Wenjie Jacky Mo, Yanan Xie, Peng Qi, Muhao Chen. Learning Efficient Guardrails for Compliance. ICLR 2026.

Questions for authors
  • Fig 1 is misleading: only the result from the proposed method is bolded, not the best result\
  • L124 "An API-based language model then synthesizes a complete agent trajectory from the sampled setting." How can one be sure that this is realistic in any sense? What domains are captured and how is diversity ensured?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.