- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the safety-utility trade-off in fine-tuning an LLM. This paper marks the gap in overlap between the utility oriented and safety-oriented models.To address the lack of good benchmarks and training environment for joint optimization of safety and utility, the paper introduces the UnsafeArena, a synthetic / co-evolution framework across multiple domains, tools and tasks that can induce in unsafe failure modes. The paper also introduces Compass-GRPO, a variant of GRPO that adds a trajectory-calibration extension by using dense self-distillation guidance into gradient feedback. Tested on Qwen3 models, the Compass-GRPO shows improvement in both utility and safety.
Critical strengths
- This paper is well-motivated and has a good problem formulation. Co-optimizing for utility and safety is a field that remains to be explored, especially with LLMs getting more capable and being applied to wider field of industry
- The mathematical formulation of GRPO for zero-gradient issue is sound and well-formulated
- The empiricial study shows strong evidence and performance across various task and against various baselines
- The tasks, tools included in the benchmark is quite extensive and well formulated.
Critical weaknesses
-
In B.2 The paper mentions using LLM as a judge, but details of judge (which model was used with what kind of prompts) is missing. While it is promising to see that human experts achieved 99% agreement on binary decisions, It would be nicer if such details on LLM Judge along with details on human judge (prompts shown to humans, how many humans etc) so that the validity of the work can be verified.
-
This paper evaluated the framework on 14B and 8B models released on 2025. Given the speed of LLM research, I would like to see how the paper performs and scales with the advances made in safety and performance in the industry so far.
-
The task synthesis structure is a bit similar to EnvScaler (Song et al, 2026) and Agent World Model (Wang et al, 2026). While the existing works was not geared towards safety in mind, their structure feels similar enough with the proposed paper to undermine the proposed novelty. UnsafeArena on 14B experiments outperformed EnvScaler and AWM shows that new framework is promising, but for the sake of novelty, it would be better to do a bit of an ablation study to uncover the insights that otherwise could be covered up by the difference in scale of the environments and synthesis model used.
Questions for authors
- I like how the paper compares similar approaches (RLSD, SDPO+GRPO) in the appendix section. However, I am curious how it is mathematically different from other recent algorithms such as DAPO and Dr.GRPO. Perhaps the intricate design of Compass-GRPO makes it difficult to do an ablation study on each of the components, but it would be better to replace the ablation study by comparison with other algorithm with similar mathematical formulation / problems in which it attempted to address.
- If possible, ablation study that controls the synthesis model and size of the environment for a fair comparison with older baselines.