acceptodds
Under review as a conference paper at ICLR 2027

UnsafeArena: Agent and Environment Co-Evolution for Safety and Utility

14 views

Abstract

Training agents in synthetic environments can improve complex task performance, yet scalable interactive environments for agentic safety reinforcement learning remain scarce. We introduce UnsafeArena, a framework that jointly improves agent utility and safety by co-evolving policies and executable environments grounded in real-world settings. UnsafeArena combines challenging utility tasks with safety tasks involving contextual risks, supported by reliable verifiers for reinforcement learning. After each training round, dynamic evaluation across difficulty levels guides targeted task generation and incremental environment refinement. We further introduce Compass-GRPO, a trajectory-calibrated extension of group relative policy optimization that incorporates dense self-distillation guidance into policy advantages. A self-teacher compares token probabilities with and without task hints along the unhinted student's own trajectories. This guidance is jointly calibrated against utility and safety to support learning when trajectory rewards provide little differentiation. We instantiate UnsafeArena with 2,661 stateful environments, construct over 40,000 tasks, and conduct large-scale reinforcement learning. Experiments demonstrate substantial improvements over the Qwen3-14B base model: UnsafeArena-14B reduces AgentSecurityBench attack success by 59.5 percentage points and raises -Bench task success by 8.1 points. Gains generalize to external benchmarks without evaluation hints. These results show that environment synthesis and targeted policy optimization can develop agents that accomplish complex tasks while respecting execution boundaries.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, Agentic RL

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_asr2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the safety-utility trade-off in fine-tuning an LLM. This paper marks the gap in overlap between the utility oriented and safety-oriented models.To address the lack of good benchmarks and training environment for joint optimization of safety and utility, the paper introduces the UnsafeArena, a synthetic / co-evolution framework across multiple domains, tools and tasks that can induce in unsafe failure modes. The paper also introduces Compass-GRPO, a variant of GRPO that adds a trajectory-calibration extension by using dense self-distillation guidance into gradient feedback. Tested on Qwen3 models, the Compass-GRPO shows improvement in both utility and safety.

Critical strengths
  • This paper is well-motivated and has a good problem formulation. Co-optimizing for utility and safety is a field that remains to be explored, especially with LLMs getting more capable and being applied to wider field of industry
  • The mathematical formulation of GRPO for zero-gradient issue is sound and well-formulated
  • The empiricial study shows strong evidence and performance across various task and against various baselines
  • The tasks, tools included in the benchmark is quite extensive and well formulated.
Critical weaknesses
  • In B.2 The paper mentions using LLM as a judge, but details of judge (which model was used with what kind of prompts) is missing. While it is promising to see that human experts achieved 99% agreement on binary decisions, It would be nicer if such details on LLM Judge along with details on human judge (prompts shown to humans, how many humans etc) so that the validity of the work can be verified.

  • This paper evaluated the framework on 14B and 8B models released on 2025. Given the speed of LLM research, I would like to see how the paper performs and scales with the advances made in safety and performance in the industry so far.

  • The task synthesis structure is a bit similar to EnvScaler (Song et al, 2026) and Agent World Model (Wang et al, 2026). While the existing works was not geared towards safety in mind, their structure feels similar enough with the proposed paper to undermine the proposed novelty. UnsafeArena on 14B experiments outperformed EnvScaler and AWM shows that new framework is promising, but for the sake of novelty, it would be better to do a bit of an ablation study to uncover the insights that otherwise could be covered up by the difference in scale of the environments and synthesis model used.

Questions for authors
  • I like how the paper compares similar approaches (RLSD, SDPO+GRPO) in the appendix section. However, I am curious how it is mathematically different from other recent algorithms such as DAPO and Dr.GRPO. Perhaps the intricate design of Compass-GRPO makes it difficult to do an ablation study on each of the components, but it would be better to replace the ablation study by comparison with other algorithm with similar mathematical formulation / problems in which it attempted to address.
  • If possible, ablation study that controls the synthesis model and size of the environment for a fair comparison with older baselines.
Reviewer_KhwzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents UnsafeArena, a pipeline that uses LLM agents to build 2,661 stateful, executable tool environments with more than 40k verified utility and safety tasks. The tasks are organized in a taxonomy of 20 failure modes with each environment providing both safe tools and deliberately permissive ones and safety tasks comes at different levels of difficulty. Training is performed over three rounds where before each round the current policy is evaluated at every difficulty level and new tasks are generated at levels where the success rate is within a certain band. this is called agent-environment co-evolution. The paper also proposes Compass-GRPO which adds a bounded token-level term to GRPO's advantage, this term comes from the frozen round's start checkpoint of the same model which scores the student's tokens with a priviliged hint and this switches on only when a utility-safety calibration test is passed. Evaluated on five agent-safety benchmarks and starting from Qwen 3-8B/14B agents improve performance substantially.

Critical strengths
  1. The suite seems well thought through for safety tasks. The safety tasks target operational failures such as skipped approvals, wrong-entity actions, unscoped disclosure and missing provenance, rather than only harmful-content requests. Safety tasks come with legit options that test for over-refusal or conservative behavior.

  2. Both model sizes improve on all five safety benchmarks, with the largest gains on AgentHarm and ASB . In Fig. 5, the trained models reach a strong safety–utility trade-off compared with both safety-tuned and utility-tuned released models.

  3. Table 2 and Fig. 1(b) show that utility-oriented environment RL (EnvScaler [1], AWM [2]) does not improve agent safety and often degrades it which is important analysis for the community.

Critical weaknesses
  1. The contribution of each component (environment, a curriculum and the optimizer) of the paper is not established. Compass-GRPO needs an extra forward pass with the hint and extra calibration rollouts. The environments' own contribution is never isolated either.

  2. Lack of rigor in the results: All results are shown with a single seed with a single training run with no confidence interval, at the same time some of the reported improvements are small enough to be within the plausible range of noise.

  3. For each component the paper lacks relevant comparisons: "Environments:* Sha et al. [1], which reports on the same Agent-SafetyBench and BFCL, or AgentAlign [2]. Only released checkpoints trained on different data are compared. "Curriculum:" not compared with simple difficulty filtering, e.g. DAPO dynamic sampling [3] or PLR [4]. "Compass-GRPO:"not compared with OPSD [5], the method it builds on.

[1] Z. Sha et al. Agent Safety Alignment via Reinforcement Learning. arXiv:2507.08270, 2025. [2] J. Zhang, L. Yin, Y. Zhou, S. Hu. AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models. arXiv:2505.23020, 2025. [3] Q. Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025. [4] M. Jiang, E. Grefenstette, T. Rocktäschel. Prioritized Level Replay. ICML, 2021. [5] S. Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (OPSD). arXiv:2601.18734, 2026.

Questions for authors
  1. How does each Compass-GRPO component (gate, b_x, λ) contribute, and does GRPO at matched compute close the gap?

  2. What do environment revisions add beyond re-targeting task difficulty, and how many environments were revised per round?

  3. How do each component contribute towards the final performance? A proper ablation would help.

  4. How do UnsafeArena and Compass-GRPO compare with Sha et al. [1] / AgentAlign [2] training, DAPO dynamic sampling [3], and OPSD [4] under the same base model and budget?

[1] Z. Sha et al. Agent Safety Alignment via Reinforcement Learning. arXiv:2507.08270, 2025. [2] J. Zhang, L. Yin, Y. Zhou, S. Hu. AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models. arXiv:2505.23020, 2025. [3] Q. Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025. [4] S. Zhao et al. Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (OPSD). arXiv:2601.18734, 2026.

Reviewer_YbriOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes UnsafeArena, a reinforcement learning framework that co-evolves agents and environments to jointly improve the utility and safety of LLM agents. The framework automatically constructs executable synthetic environments with realistic business logic, generates utility and safety tasks, and dynamically adjusts task difficulty based on agent performance. Additionally, the authors introduce Compass-GRPO, which incorporates hint-conditioned self-distillation and joint calibration to provide finer-grained advantage estimates for GRPO. Experiments on Qwen3-8B/14B across multiple safety and utility benchmarks demonstrate improvements in both dimensions.

Critical strengths

The paper addresses a practically relevant problem. As LLM agents are increasingly deployed in real-world applications, improving task execution capabilities while maintaining safety is an important challenge. The motivation to jointly optimize utility and safety is reasonable.

Critical weaknesses
  1. Limited methodological novelty. The proposed framework is primarily an engineering integration of existing techniques. Agent–environment co-evolution closely resembles adaptive curriculum learning, while Compass-GRPO mainly extends GRPO with hint-conditioned self-distillation guidance. Despite promising empirical results, the paper lacks sufficiently novel algorithmic or theoretical contributions.

  2. Insufficient ablation studies. Although the authors explain the absence of certain ablations, the paper lacks systematic component-level evaluations. Consequently, the necessity and individual contributions of the proposed mechanisms remain unclear.

  3. The figures have an obvious AI-generated style , which seriously affects the credibility of the paper. Several figures, particularly Figures 2 and 4, exhibit a pronounced AI-generated visual style. While AI-assisted figure creation is not inherently problematic, figures should prioritize technical accuracy, clarity, and interpretability over elaborate visual presentation. The current presentation raises concerns about the paper's rigor and credibility.

Questions for authors
  1. Could the authors provide comprehensive ablation studies to isolate the contributions of hint-conditioned guidance, the joint calibration gate, and outcome-adaptive weighting in Compass-GRPO? In particular, could comparable improvements be achieved through direct hint-based supervised learning or self-distillation?

  2. Could the authors simplify Figures 2 and 4 to more clearly illustrate the technical relationships among environment evolution, reward computation, and policy optimization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.