acceptodds
Withdrawn from ICLR 2027

Agent Safety Alignment via Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The emergence of autonomous Large Language Model (LLM) agents capable of tool usage has introduced new safety risks that go beyond traditional conversational misuse. These agents, empowered to execute external functions, are vulnerable to both user-initiated threats (e.g., adversarial prompts) and tool-initiated threats (e.g., malicious outputs from compromised tools). In this paper, we propose the first unified safety-alignment framework for tool-using agents, enabling models to handle both channels of threat via structured reasoning and sandboxed reinforcement learning. We introduce a three-way taxonomy that classifies both user prompts and tool responses as benign, malicious, or sensitive, and we define a policy-driven decision model to train the agents. Our framework employs a custom-designed sandbox environment that simulates real-world tool execution and allows fine-grained reward shaping. Through extensive evaluations on public and self-built benchmarks, including Agent SafetyBench, InjecAgent, and BFCL, we demonstrate that our safety-aligned agents significantly improve resistance to security threats while preserving strong utility on benign tasks. Our results show that safety and effectiveness can be jointly optimized, laying the groundwork for trustworthy deployment of autonomous LLM agents.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agent, Large Lanuage Models, AI Security

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_qRKzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents an end to end reinforcement learning based safety alignment framework for tool using LLM agents. The authors identify two major threat vectors: user originated threats and tool originated threats. They propose a three way taxonomy (benign / malicious / sensitive) that applies both to user prompts and tool outputs,。Based the taxonomy,the paper apply an execute–refuse–verify policy to guide agent behavior. Training is conducted inside a custom sandbox environment with carefully designed reward functions.

Critical strengths
  1. The paper addresses an important and practically relevant problem in the safety alignment of tool-using language agents.
  2. Considering both user-side and tool-side threats provides a useful perspective on agent safety.
  3. The evaluation includes both security and utility benchmarks, which is important for assessing the practical trade-offs of safety alignment.
Critical weaknesses
  1. The paper claims establishing a unified tri-modal taxonomy as one of its main contributions. However, prior works (such as AgentAlign,Agent-SafetyBench) have already defined benign and harmful categories. Furthermore, the paper does not empirically justify the role of the newly added sensitive prompt category in experiments.
  2. There are multiple obvious writing errors. Rewards for Bu(214), Rewards for Bt(248), and including the corresponding equations that follow.
  3. The main baseline is only “no defense” and “prompt guarded”. The paper lacks comparison against recent agent safety alignment methods such as AgentAlign etc.
  4. Experiment results in table 2 and 3, the proposed method achieves strong performance on its self-constructed dataset, while the performance on existing public datasets is modest; in some cases it is even inferior to Prompt Guarded. The paper lacks corresponding analysis and does not release its constructed dataset.
  5. The paper states: “For B_U, we leverage datasets released by ReCall (Chen et al., 2025b). The remaining datasets are generated using Deepseek.” However, it does not provide detailed information regarding dataset generation.
Questions for authors

None

Reviewer_qRGNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an RL-based framework for aligning tool-using agents against user-side and tool-side threats. It introduces a three-way taxonomy of benign, malicious, and sensitive cases, and trains agents to execute, refuse, or request confirmation using a sandboxed tool-execution environment and scenario-specific rewards. The evaluation uses Agent SafetyBench, InjecAgent, BFCL, and synthetic datasets generated with DeepSeek. The results suggest that joint user-and-tool alignment can improve resistance to some attacks while retaining some tool-use capability.

The problem is important, and jointly considering user-side and tool-side threats is a useful objective. However, I am not convinced that the paper establishes a sufficiently novel or generalizable safety-alignment method beyond a prescribed policy trained with hand-designed rewards.

Critical strengths
  1. The paper addresses an important and timely problem in agent safety. Tool-using agents introduce risks beyond those of ordinary conversational models because their decisions can trigger external actions. Considering both malicious user instructions and malicious tool outputs is a meaningful problem formulation.

  2. The unified execute–refuse–verify policy provides a simple and interpretable behavioral framework. The distinction between benign, malicious, and sensitive interactions offers a practical way to express different safety requirements without treating every potentially risky request as something that must be refused.

  3. The paper recognizes the importance of preserving utility alongside safety. Evaluating both attack resistance and benign tool-use performance is appropriate, since a defense that prevents almost every tool call would not constitute a useful solution for autonomous agents.

  4. The use of a sandboxed environment provides a controlled setting for training and evaluating tool interactions. Intercepting tool calls and simulating execution allows the authors to experiment with safety-related behavior without exposing real systems to potentially harmful actions.

  5. The evaluation covers multiple aspects of agent behavior. The inclusion of Agent SafetyBench, InjecAgent, BFCL, and custom benchmarks is a reasonable starting point for examining user-side threats, tool-side threats, and general function-calling ability.

Critical weaknesses
  1. The desired behavior is explicitly specified in the reward function: execute benign requests, refuse malicious ones, and request confirmation for sensitive actions. What new safety capability does RL learn beyond these hand-designed rules? A comparison against supervised training or a strong policy-based baseline under matched conditions would help isolate the contribution of the RL framework. 2.The novelty relative to existing agent-safety and alignment methods is insufficiently established. The core approach combines a three-way threat taxonomy, a rule-based reward function, sandboxed tool execution, and reinforcement learning. These components are intuitively reasonable, but the paper does not clearly identify which methodological component is novel beyond their combination, or demonstrate why this formulation is preferable to existing agent-alignment approaches. The claim of being the first end-to-end agent safety-alignment framework is particularly strong given the related work discussed by the authors, including AgentAlign and other approaches to agent behavioral safety. A more precise comparison with these methods is necessary to establish the contribution.
  2. Section 3.2 states that the sandbox randomly samples a yes or no when the agent requests confirmation. Since this feedback is not grounded in user intent, how does the agent learn reliable authorization behavior from it? An ablation with intent-consistent confirmation, alongside tests of whether approval is bound to the specific action, would help establish whether the learned protocol translates to meaningful authorization.
  3. The training data is generated using DeepSeek and role-specific templates, while the reward directly captures the expected behavior for each threat category. Do the gains generalize to independently constructed attacks that differ from these templates, particularly multi-step attacks and malicious instructions embedded in otherwise benign tool outputs? Evaluation on such cases would help distinguish contextual safety reasoning from learning the prescribed execute-refuse-verify patterns.
  4. For the 14B model, user-only alignment worsens the InjecAgent score from 12.4 to 51.4, while joint alignment reduces it to 3.0. However, BFCL-Live performance drops from 74.8 for the unaligned model to 42.2 for the jointly aligned model. These results complicate the claim that safety improves while utility is preserved. The authors should analyze this trade-off and ablate the effects of joint training, additional tool-side data, and reward design.
Questions for authors
  1. What is the precise methodological difference between the proposed approach and prior agent-safety alignment methods, particularly AgentAlign? Which aspect of the method should be considered the main technical contribution beyond combining a three-way taxonomy with rule-based rewards and sandboxed reinforcement learning?

  2. How are the synthetic training data and custom evaluation benchmarks separated? Can the authors clarify whether deduplication or semantic-overlap checks were performed across the datasets, and whether the near-perfect custom safety scores persist on examples generated independently of the training pipeline?

  3. Why does user-side alignment worsen the 14B model's InjecAgent score from 12.4 to 51.4, given that lower is better? What aspect of the training objective explains this regression, and how should it be reconciled with the paper's claims about safety improvements?

  4. How do the authors explain the large BFCL and BFCL-Live performance reductions for the aligned 14B models, despite the substantial improvements on the custom utility benchmark? Which metric most accurately reflects the intended deployment setting, and how should the utility-preservation claim be qualified?

  5. In Equation 2, how does the reward for malicious tool outputs explicitly enforce detection and communication of the malicious instruction, rather than merely requesting confirmation and conditionally proceeding? Can the authors clarify the intended reward semantics and the apparent notation inconsistencies in Section 3.3?

Reviewer_tDoCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This submission proposes an RL-based safety alignment framework for an agent. It aims to address both user-initiated threats and malicious tool outputs. It classifies the input into benign, malicious, and sensitive cases. An agent can be trained to follow an execute-refuse-verify policy. The authors test the method by using Qwen 7B and 14B models on some synthetic benchmarks. The final experiments show some improvements on several evaluations, but also show substantial regressions on some public safety benchmarks and other tool-use tasks. It cannot claim that the framework jointly improves safety and utility.

Critical strengths
  1. This paper points out an important problem formulation. How to get a unified alignment framework for those two threat channels is very meaningful.

  2. The execute-refuse-verify formulation is a very intuitive and structured policy for the trade-off between safety and the original task.

  3. Its experiments cover both robustness and tool-use performance. The trade-off is important rather than just reporting safety improvement in isolation.

Critical weaknesses
  1. The major concern is that the reporting results can not support the joint improvement of safety and utility. For the 14B model, the BFCL ACC dropped from 95.5 to 67.3, and BFCL-Live dropped from 74.8 to 42.2. In Agent SafetyBench, the score also decreases a lot compared with the user-aligned configuration. These results weaken the paper's claims.

  2. The reward function has inconsistent threat labels and conflicting behavioral rules. This make the optimization objective not clear and raise lots of concerns about how to train the model useful behavior.

  3. The model can perform so well on authors' synthetic benchmark but perform badly on some public benchmarks. It seems like a little overfitting to predefined safety rules. It would be better to provide more proof of generalization to unseen threats.

Questions for authors

Please refer to the Weaknesses section.

Reviewer_QRzFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes an approach for training LLM-based agents using reinforcement to improve safety in agentic settings wherein they cover threats from user requests and tool invocation results, and include three cases for each benign, sensitive, and malicious user requests/tool results that the agent must identify.

They propose a reward function that covers these 6 cases ({benign, malicious, sensitive} x {user request, tool result}) and also include some formatting rewards for following chat templates. They demonstrate how their RL trained models with this reward design outperform baselines like vanilla and prompt-guarded settings on some safety benchmarks like Agent SafetyBench.

Critical strengths
  • The paper tries to address an important problem and the attack scenarios/threat settings discussed in Section 2 are quite intuitive.
Critical weaknesses
  • The paper is extremely difficult to follow and requires significant revisions to improve readability. The authors provide limited details about how they curate the synthetic corpus from DeepSeek in Section 3.1 and how they manage to spot-check such large amount of data manually. The paper is excessively text-heavy without a single figure/visualization in first 6 pages that describes at least some components like data curation approach or some sample trajectories that describe how the reward function in section 3.3 actually operates over the trajectory. Are the tasks single-turn or multi-turn? Table 1 seems to suggest all tasks are single-turn but the paper seems to propose an RL recipe for agentic safety alignment. Are the rewards from section 3.3 applied at each turn or at a trajectory level?

  • The paper proposes an approach to achieve agent safety alignment via RL, but includes extremely shallow details about the RL algorithm being used to train the model. The only details I find are in line 269 that says: "Policy and Value Update. Apply a clipped policy-gradient step with batch-normalized returns to stable optimization". Is this algorithm PPO or something else? What are the training hyperparameters and how do the reward curves look like? While figure 1 describes the models being trained as Qwen2.5-7B-Instr. and Qwen2.5-14B-Instr., the main text includes no mention of the specific choice of models being trained. What is the total number of training samples being used?

Questions for authors
  • Can you include more details on what is the "custom-designed sandbox environment that simulates real-world tool execution" look like? If you are simulating tool execution, why do you need sandboxes in the first place instead of just doing tool execution simulation programmatically or using an LLM?

  • Why exactly do you need EOS compliance for the general reward signal? I believe Qwen2.5-7B Instruct and Qwen2.5-14B-Instruct models should follow their chat templates quite reliably out-of-the-box?

Others: typo in Section 3.3, Line 214: "For malicious user prompts (B_u), the correct behaviour is an immediate refusal without invoking any external tool", this should be M_u right?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.