- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies whether LLMs can improve safety alignment through self-generated supervision without stronger teacher models or external reward models. The proposed SIGMA framework generates alignment dilemmas and evaluation rubrics from a Model Spec, then uses the model itself as a judge for SFT or RL training. Experiments show that SIGMA improves performance across multiple safety and alignment benchmarks using only single-turn training data.
Critical strengths
This paper proposes an alignment training framework that reduces reliance on external supervision by allowing the model to generate its own tasks, evaluation criteria, and reward signals. This direction is meaningful for reducing dependence on stronger models and human supervision in alignment training.
Critical weaknesses
The paper emphasizes the Out-of-distribution Alignment Generalization from Single-turn Training to Agentic Environments. However, Threat Models involving Self-serving and Oversight avoiding Behaviors are already included in the training process, and the Model Spec also clearly stipulates the relevant Safety Principles. These behaviors overlap highly with those evaluated by Agentic Misalignment and InstrumentalEval. So it is not clear whether the performance improvement of the current experiment truly reflects the generalization of the unseen Alignment Failure Modes, or mainly reflects the transfer of the clearly trained behavioral principles among different Interaction Settings.
Questions for authors
Could the authors evaluate SIGMA with the misaligned-model threat model held out during task synthesis, while keeping the Model Spec unchanged, to examine whether the improvements on Agentic Misalignment and InstrumentalEval persist?
2 comments
Follow up
Dear Reviewer VJXg,
Thank you again for your review and suggestions. As the discussion period closes today, we wanted to check whether our rebuttal addresses your concerns. We would welcome any further feedback and the opportunity to clarify any unresolved points.
Response to reviewer VJXg
We thank reviewer
VJXgfor your thoughtful feedback and for recognizing the value of reducing dependence on stronger models and human supervision. Your concern identifies an important distinction in what we mean by alignment generalization, and we now clarify your concern:We do agree that generalizing to unseen alignment failure modes is an important problem, though our paper focuses on another kind of generalization: transferring to known, good behaviors to a domain of data unseen during training. We believe there is a substantial gap between the training and evaluation settings on their domain: SIGMA trains exclusively on single-turn descriptions of alignment-relevant scenarios, where the model explains what it would do and why in a single text response. In contrast, our evaluations test how this training transfers across diverse settings.
For example, in AgentHarm, the model acts through tools over multiple turns and evaluation measures whether harmful actions are conducted. This is quite different from our training which involves text response only. In Agentic Misalignment, the model acts as an email agent facing replacement through its multi-turn conversation history. The three scenarios are focused on specific misaligned behaviors, i.e., blackmailing, leaking confidential information, and murder. In contrast, our model spec does not have any direct guidance against these specific behaviors. For example, the model spec mentions “Not undermining the ability of legitimate principals to adjust, correct, retrain, or shut down AI systems as allowed given their role” which is a very general guidance. Nevertheless, we still see consistent improvement across all 3 scenarios.
We would also like to clarify why we believe our focus of generalizing known aligned behavior to deployment settings not covered by the training data is itself a crucial alignment challenge: as models are increasingly capable of operating in diverse long horizon environments, it’s becoming increasingly infeasible to cover every deployment scenario in the training data. This implies that ensuring models consistently follow behavioral principles outside of scenarios it has been trained on is crucial for their trustworthiness. Our results provide evidence that SIGMA’s self-generated supervision helps bridge this gap. We see discovering failures beyond the Model Spec’s coverage and determining how the specification should address them is a complementary problem of defining that target, and thus not the main focus of this work. We do agree your proposed direction on generalizing to unseen alignment failure modes is a very valuable problem too and we will add discussion of your proposed direction as a valuable next work.