acceptodds
Under review as a conference paper at ICLR 2027

SIGMA: Self-Improving Alignment Generalization from a Model Spec

Abstract

LLM agents are increasingly capable of executing complex tasks and of recursively improving themselves on easy-to-verify objectives such as software engineering and mathematics. Since alignment is much harder to verify, this creates a growing risk of capabilities increasing without appropriate safety alignment: models may act dangerously in consequential situations their safety training never anticipated, especially as their capabilities expand to auto-research and cybersecurity domains. Existing approaches often focus on capability self-improvement using verifiable feedback, or on alignment training with supervision from stronger models or curated data, creating an external supervision bottleneck for alignment. To address this bottleneck, we ask whether current models can improve their own safety alignment. We propose SIGMA, a data generation and training pipeline enabling alignment self-improvement that generalizes to out-of-distribution settings. Given only a Model Spec that states the model's desired behavior and high-level prompts defining its goal, SIGMA leverages a model's general reasoning capabilities to strengthen its own safety reasoning capability. SIGMA first performs spec-guided task synthesis, using the candidate model as a task designer agent to generate diverse alignment dilemma scenarios and convert them into high-quality alignment training tasks that stress-test the model’s understanding of the Model Spec. Next, SIGMA conducts self-judged alignment training through supervised fine-tuning and rubric-based reinforcement learning with the model itself as the reward model. Despite training only on single-turn chat data describing alignment dilemmas, SIGMA improves safety alignment in multi-turn agentic environments (AgentHarm harmfulness decreases from 22.6 to 14.8; Agentic Misalignment decreases from 79.1 to 3.8), outperforms Deliberative Alignment and Constitutional AI baselines, and retains general capability. Analyses show that a Model Spec that balances harmlessness and helpfulness, test-time reasoning for safety deliberation, and high-quality rubrics produced by SIGMA's task designer agent are crucial for effective self-improvement.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
safety, alignment, self-improvement

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VJXgOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether LLMs can improve safety alignment through self-generated supervision without stronger teacher models or external reward models. The proposed SIGMA framework generates alignment dilemmas and evaluation rubrics from a Model Spec, then uses the model itself as a judge for SFT or RL training. Experiments show that SIGMA improves performance across multiple safety and alignment benchmarks using only single-turn training data.

Critical strengths

This paper proposes an alignment training framework that reduces reliance on external supervision by allowing the model to generate its own tasks, evaluation criteria, and reward signals. This direction is meaningful for reducing dependence on stronger models and human supervision in alignment training.

Critical weaknesses

The paper emphasizes the Out-of-distribution Alignment Generalization from Single-turn Training to Agentic Environments. However, Threat Models involving Self-serving and Oversight avoiding Behaviors are already included in the training process, and the Model Spec also clearly stipulates the relevant Safety Principles. These behaviors overlap highly with those evaluated by Agentic Misalignment and InstrumentalEval. So it is not clear whether the performance improvement of the current experiment truly reflects the generalization of the unseen Alignment Failure Modes, or mainly reflects the transfer of the clearly trained behavioral principles among different Interaction Settings.

Questions for authors

Could the authors evaluate SIGMA with the misaligned-model threat model held out during task synthesis, while keeping the Model Spec unchanged, to examine whether the improvements on Agentic Misalignment and InstrumentalEval persist?

2 comments
AuthorsOpenReview ↗

Response to reviewer VJXg

We thank reviewer VJXg for your thoughtful feedback and for recognizing the value of reducing dependence on stronger models and human supervision. Your concern identifies an important distinction in what we mean by alignment generalization, and we now clarify your concern:

“So it is not clear whether the performance improvement of the current experiment truly reflects the generalization of the unseen Alignment Failure Modes, or mainly reflects the transfer of the clearly trained behavioral principles among different Interaction Settings.” “Could the authors evaluate SIGMA with the misaligned-model threat model held out during task synthesis, while keeping the Model Spec unchanged, to examine whether the improvements on Agentic Misalignment and InstrumentalEval persist?”

We do agree that generalizing to unseen alignment failure modes is an important problem, though our paper focuses on another kind of generalization: transferring to known, good behaviors to a domain of data unseen during training. We believe there is a substantial gap between the training and evaluation settings on their domain: SIGMA trains exclusively on single-turn descriptions of alignment-relevant scenarios, where the model explains what it would do and why in a single text response. In contrast, our evaluations test how this training transfers across diverse settings.

For example, in AgentHarm, the model acts through tools over multiple turns and evaluation measures whether harmful actions are conducted. This is quite different from our training which involves text response only. In Agentic Misalignment, the model acts as an email agent facing replacement through its multi-turn conversation history. The three scenarios are focused on specific misaligned behaviors, i.e., blackmailing, leaking confidential information, and murder. In contrast, our model spec does not have any direct guidance against these specific behaviors. For example, the model spec mentions “Not undermining the ability of legitimate principals to adjust, correct, retrain, or shut down AI systems as allowed given their role” which is a very general guidance. Nevertheless, we still see consistent improvement across all 3 scenarios.

We would also like to clarify why we believe our focus of generalizing known aligned behavior to deployment settings not covered by the training data is itself a crucial alignment challenge: as models are increasingly capable of operating in diverse long horizon environments, it’s becoming increasingly infeasible to cover every deployment scenario in the training data. This implies that ensuring models consistently follow behavioral principles outside of scenarios it has been trained on is crucial for their trustworthiness. Our results provide evidence that SIGMA’s self-generated supervision helps bridge this gap. We see discovering failures beyond the Model Spec’s coverage and determining how the specification should address them is a complementary problem of defining that target, and thus not the main focus of this work. We do agree your proposed direction on generalizing to unseen alignment failure modes is a very valuable problem too and we will add discussion of your proposed direction as a valuable next work.

Dear Reviewer VJXg,

Thank you again for your review and suggestions. As the discussion period closes today, we wanted to check whether our rebuttal addresses your concerns. We would welcome any further feedback and the opportunity to clarify any unresolved points.

Reviewer_ata9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SIGMA, which allows large models to generate, score, and train themselves based on behavioral norms, thereby improving their safety performance in unseen complex tasks while retaining their general capabilities.

Critical strengths

One advantage of this paper is that it transforms behavioral norms into a process where the model generates questions, scores them, and trains itself, reducing reliance on stronger models and manual annotation. Another advantage is that experiments demonstrate that security improvements can be transferred from single-turn dialogue training to multi-turn agent tasks while largely retaining general capabilities.

Critical weaknesses

This paper only validates the model on a single base model, lacking generalizability. I suggest the authors add more models for experimentation (different model families could be considered). Furthermore, the experiment only underwent one round of training; the conclusion of "continuous self-improvement" is not yet valid. The authors could consider adding multiple rounds of experiments. "Retaining general capabilities" comes at a practical cost.

SIGMA-SFT's GPQA score dropped from 85.2 to 82.0, and MMLU-Pro's from 86.8 to 84.7; SIGMA-RL's instruction compliance score also declined. The paper needs to more fully explain whether these declines are acceptable, rather than simply emphasizing the basic stability of capabilities.

The additional value of reinforcement learning is unclear. Simple supervised fine-tuning actually performs better on several safety metrics: for example, the rate of improper behavior in Agentic Misalignment is 3.8 for SFT and 9.0 for RL. This does not negate the overall framework, but it indicates that the paper has not yet demonstrated clear benefits from using more complex reinforcement learning processes.

Overall, I think this paper is valuable after the revision.

Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to reviewer ata9

We thank reviewer ata9 for your thoughtful review. We appreciate your recognition that SIGMA reduces reliance on stronger models and manual annotation, and that its single-turn training transfers to multi-turn agent tasks. We address your concerns below.

This paper only validates the model on a single base model, lacking generalizability.

We agree that evaluation across model families would strengthen the evidence. Given the cost of agentic task synthesis, best-of-N generation and judging, and RL, we focused our available resources on a detailed study of Qwen3.6-27B. We would like to note that the SIGMA pipeline is not architecturally specific to Qwen: its components require a reasoning-capable model, a Model Spec, and a task-design harness. As SIGMA utilizes the model’s general reasoning and agentic capability to construct alignment supervision, we expect stronger models to gain more from SIGMA. Regarding evidence on generality, our experiments in §3.6 have shown that SIGMA’s generalize across different alignment evaluation benchmarks. We will make the single-backbone limitation more explicit in §5 and frame our results as evidence that alignment self-improvement is possible in this setting.

Furthermore, the experiment only underwent one round of training; the conclusion of "continuous self-improvement" is not yet valid.

Thank you for raising this point. The central question of our work is whether a model can translate a human-written Model Spec into effective supervision for improving its own alignment, without relying on a stronger teacher or externally curated alignment examples. SIGMA addresses this question through model-generated tasks, rubrics, reasoning, and self-jugements. Our experiments demonstrate that this supervision improves alignment and generalizes to held-out settings, and we believe establishing this capability is a substantive contribution within a single training cycle.

Repeated self-improvement would test an additional hypothesis: whether the improved model becomes a better source of supervision and produces further gains over successive cycles. This is an important extension, but it is not required to establish the effectiveness of the self-generated supervision studied here. Accordingly, we believe the absence of multiple cycles does not undermine the demonstrated alignment gains or their generalization. Section 5 of our paper also states that we conduct “a single round of self-improvement training” and identifies closing the loop as future work. We will emphasize this distinction to avoid any implication that sustained or recursive improvement has already been demonstrated.

The paper needs to more fully explain whether these declines are acceptable, rather than simply emphasizing the basic stability of capabilities.

We would like to clarify that our intended claim is that substantial alignment gains are achievable with relatively minor capability changes, even though training uses only alignment data and no capability training data at all. For example, SFT reduces Agentic Misalignment by 75.3 percentage points while retaining AgentHarm benign completion (76.6 → 76.7), while declines on GPQA and MMLU-Pro are only ~3 points of minor changes. We deliberately do not mix in general-capability or helpfulness datasets, so the experiments expose the effect of SIGMA’s alignment training directly. Mixing such data is a simple and plausible mitigation for further improving capability alongside alignment.

the paper has not yet demonstrated clear benefits from using more complex reinforcement learning processes.

We would clarify that our contribution is the construction of useful self-generated alignment supervision, which can support either SFT or RL. Our main point is showing the broad usage of SIGMA on both SFT and RL routes of alignment training, and it does not depend on whether RL outperforms SFT. We have also provided a comparison between SFT and RL in §3.2 and happy to share further discussion here. Note that SIGMA-SFT performs best-of-N selection from spec-conditioned generations, then directly supervises both CoT and the response with the spec removed from the student’s input (§2.2). It therefore distills explicit safety-deliberation reasoning traces directly into the model through.

On the other hand, SIGMA-RL does not see model spec during RL and the information in the spec only indirectly influences model training through the model spec judge as well as the rubrics which were derived with model spec in the verification loop. Appendix K and Fig. 6 also illustrate this qualitative difference: SIGMA-SFT explicitly recalls spec clauses, whereas SIGMA-RL reaches an oversight-respecting decision without citing any spec-specific content.

Thank you again for these constructive suggestions. We will incorporate the scope clarifications, capability discussions, and expanded SFT–RL comparison into the revision.

Dear Reviewer ata9,

Thank you again for your helpful feedback. With the discussion period ending today, we would appreciate any thoughts on our response. We are happy to address any remaining questions.

Reviewer_o4zuOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SIGMA, a method aimed at alignment self-improvement. Starting from a Model Spec describing how the designers want the model to behave, plus a short list of domains and threat models, a single model (Qwen3.6-27B) constructs its own training data and supervision. It first acts as an agent that designs pairs of alignment dilemmas, differing in one decision-relevant fact, with evaluation rubrics, and verifies them itself. It then serves as its own reward model for rejection-sampling SFT and RL. Although training uses only single-turn prompts, the paper evaluates whether safety gains generalize beyond the training setting, including to multi-turn agentic safety, power-seeking, and sycophancy.

Critical strengths

(1). The paper asks whether a model can build its own alignment supervision from only a Model Spec and a short domain list. The pipeline is minimal, and all evaluations use external graders on benchmarks the pipeline never saw. The paper reproduces the full spec, all spec variants, and nearly every prompt, which makes it easy to audit and build on.

(2). The paired counterfactual task design is a strong point. Each training pair differs in one decision-relevant fact, such as whether authorization was given. This trains the model to judge context rather than simply refuse, and each pair builds in a case where helping is correct. The results show that removing the self-verification and refinement loop more than doubles the Agentic Misalignment rate.

(3. Transfer from single-turn deliberation to agentic behavior, with useful ablations. Training only on chat-style “what would you do” prompts cuts Agentic Misalignment severely. The ablations on Rewards and Sepc content give some insight into what underlies the results. The findings may also be useful for spec design in general.

Critical weaknesses

(1). One important concern is the circularity involved in having the same model play so many roles. The model plays at least six roles: seed generator, seed critic, task designer, the three task verifiers, SFT teacher, and the RL reward model. Every evaluation uses external graders so the reported gains are not self-graded. But a key question is whether the training signal is correct, and does self-judging bias what the model learns? This is an issue in several aspects of the paper. (1). The Spec-adherence verifier. is very lenient by design. The prompt tells the judge to answer “adheres” when it is unsure and to treat any story it constructs about a bad criterion as not a contradiction. (2). The Self-consistency verifier: the model writes a rubric, writes a response meant to pass it, then judges whether it passes. (3). The Difficulty verifier has a weak threshold, as a pair passes if one sample from fails one rubric item. Using a stronger model in some places, ideally along with some human labeling could help a lot, for example to audit some of the tasks, to check for reward hacking during RL and Invoking a different model of similar strength would help bolster the claims, because it fits with the scalable-oversight aim.

A related weakness is that the paper does not consider what happens if a slightly misaligned model can shape rubrics in its favor. This suggests that SIGMA moves the verification problem from outcomes to rubrics without solving it.

(2). The paper proposes two versions, the SFT and RL versions. The tables contain results from both, which are quite different in many aspects. Yet the comparison of these two and analysis of their strengths and weaknesses is very thin. I only found one paragraph, in Section 3.2. They are different in many ways. They supervise different aspects: SFT on the chain of thought directly, while RL only grades the response. RL has a KL penalty while SFT does not. The required compute should also be compared. RL ablates reward and self-verification and SFT ablations focus on data scale and spec formulations. To understand what actually makes the SIGMA work, it is very important to explore this comparison yet it is lacking in the paper.

(3). The paper does include only one external-data baseline, STAR-1. This makes it very difficult to evaluate whether self-generation is as valuable as stated. Ideally there would be two separate prompt sources (SIGMA-synthesized vs. external), which would be paired with two different supervision source (self-judged vs. an external teacher). An important question is how SIGMA’s self-judged training performs on external prompts, which isolates the value of the task-synthesis stage.

Questions for authors

Which of the comparisons are statistically significant? This applies to key comparisons such as SIGMA-SFT vs. the DA-SFT baseline, and the SycophancyEval gain.

Does InstrumentEval reward refusal? The paper says IE convergence drops when thinking is off because refusals increase. Lower convergence may partly reflect more refusal rather than less power-seeking. What are the refusal rates on InstrumentEval?

A main mechanism in the paper is matched pairs where one fact flips the right answer, but there is no held-out evaluation of whether the trained model tracks that fact. How can we tell whether the model learned contrast sensitivity or just became more cautious in general?

3 comments
AuthorsOpenReview ↗

Response to reviewer o4zu

We thank reviewer o4zu for your detailed and thoughtful feedback. We appreciate your recognition that “the paired counterfactual task design is a strong point,” and that the transfer from single-turn deliberation to agentic behavior and the spec ablations are useful. We address each concern below.

But a key question is whether the training signal is correct, and does self-judging bias what the model learns?

Thank you for raising this question. Our hypothesis (§2) is that the same model can use its reasoning ability to provide better safety supervision than its default responses exhibit. Its roles share parameters but differ in task and context. The designer uses the Model Spec to construct tasks and rubrics. SIGMA’s learning signal comes from the assumption that for a given task, it is easier for the judge to check response compared to the learner generating a perfect response without spec and rubric guidance. For example, SIGMA-SFT uses this distinction for context distillation. The model generates responses conditioned on the spec, which is then reranked according to the generated rubrics. This spec-conditioned distillation helps train the model to the spec-compliant behavior we want. Overall, SIGMA turns an external behavioral target, supplied by the Model Spec, into effective supervision for training toward that behavior.

We would also like to clarify what each verifier establishes (§F.1). The rubric–spec check is deliberately a non-contradiction check, which rejects rubrics that clearly violate the spec, to ensure it’s not overly strict on uncertain cases. The difficulty check excludes pairs on which both sampled responses pass and it is a minimal filter for learning opportunities, rather than a calibrated estimate of task difficulty, which would be costly and potentially have low marginal gains. Although these checks can still share the model’s blind spots, we have shown the checks’ usefulness through external evaluation: removing self-verification and refinement raises Agentic Misalignment from 9.0 to 20.3 (§3.5) and this supports the pipeline’s empirical value.

what happens if a slightly misaligned model can shape rubrics in its favor

We would like to clarify that our setting assumes a post-trained model whose general reasoning can support useful spec-grounded supervision, and does not claim robustness to a model strategically corrupting that supervision. We would like to clarify the scope of SIGMA is a proof of concept that alignment can improve with minimal external supervision, rather than a complete solution to scalable oversight. Even if a model is slightly misaligned, the generated rubrics still have to pass the spec adherence verifier during task synthesis. Note that this verifier instructs the model to verify contradiction between the rubric and the model spec, where both are provided directly. Therefore, we believe that as long as the model is following the instruction well, the spec adherence verifiable will be able to identify rubrics that are misaligned regardless of whether the model is misaligned or not. This shows an advantage of our rubric-based task synthesis, where rubrics make the supervision granular, explicit and inspectable. We will provide additional discussion in §5.

“Yet the comparison of these two and analysis of their strengths and weaknesses is very thin.”

Thanks for the suggestion and we are happy to provide more details on their comparison. Note that SIGMA-SFT performs best-of-N selection from spec-conditioned generations, then directly supervises both CoT and the response with the spec removed from the student’s input (§2.2). It therefore distills explicit safety-deliberation reasoning traces directly into the model thought. On the other hand, SIGMA-RL does not see model spec during RL and the information in the spec only indirectly influences model training through the model spec judge as well as the rubrics which were derived with model spec in the verification loop. Appendix K and Fig. 6 also illustrate this qualitative difference: SIGMA-SFT explicitly recalls spec clauses, whereas SIGMA-RL reaches an oversight-respecting decision without citing any spec-specific content. We will also expand our existing comparison in §3.2 with additional discussion. We would like to clarify that for compute, because SIGMA-SFT mainly incurs cost for offline generation and judging N=16 candidates per prompt followed by supervised updates, and SIGMA-RL mainly incurs online rollout and training cost, instead of controlling the compute we control the two setup the use the same set of 14K prompts generated during spec-guided task synthesis, as we believe the diversity and quality of task prompts is one of the main advantages that SIGMA brings.

AuthorsOpenReview ↗

Response to reviewer o4zu - continued

An important question is how SIGMA’s self-judged training performs on external prompts, which isolates the value of the task-synthesis stage.

Thank you for this suggestion. We provide an additional external-data comparison using SafeChain [1]. We train Qwen3.6-27B on matched 14K training examples sampled by the dataset’s benign/harmful and vanilla/adversarial labels. We use the same SFT recipe and supervise on both the CoT and final answer, with the reasoning delimiters normalized to Qwen’s chat template.

SetupAH Harm ↓AH Benign ↑AH HM ↑AM Blackmail ↓AM Leaking ↓AM Murder ↓AM Avg ↓IE Conv. ↓SE Truth. ↑
Base model22.676.677.092.373.371.779.156.684.2
SafeChain3.78.916.38.051.045.734.956.680.7
SIGMA-SFT14.876.780.79.00.02.33.819.786.3

All scores are percentages. AH denotes AgentHarm; HM combines harm avoidance and benign task completion. AM Avg. averages the blackmail, leaking, and murder rates.

We find that even with evenly sampled benign and harmful data, training on SafeChain leads to the model being over-conservative, as seen by the low AH benign and general capability evaluation scores. Moreover, while SafeChain reduces agentic misalignment compared to the base model, it underperforms SIGMA-SFT by a large margin. These results show that this carefully curated safety reasoning dataset does not reproduce SIGMA’s balance of alignment and utility.

Which of the comparisons are statistically significant?

In Appendix J, provides 95% bootstrap confidence intervals. For Agentic Misalignment, SIGMA-SFT’s 3.8 ± 1.2 is clearly separated from DA-SFT’s 11.6 ± 1.9 and the base model’s 79.1 ± 2.6. We refer the reviewer to Tables 16–17 for further details.

Lower convergence may partly reflect more refusal rather than less power-seeking.

We agree that a refusal can receive a non-convergent label, so this metric alone cannot distinguish constructive safe behavior from refusing the scenario. Our footnote in §3.4 explicitly notes this artifact when thinking is disabled so we do not claim the general effect of turning off reasoning on convergence. For the main thinking-enabled models, AgentHarm benign completion remains 76.7/76.9 versus 76.6 for the base model, and XSTest over-refusal remains low at 0.8/1.6 versus 1.2 (SFT/RL). These results argue against a general shift toward refusing everything.

How can we tell whether the model learned contrast sensitivity or just became more cautious in general?

Our paired construction is intended to reward sensitivity to the decision-relevant fact, with a helpful response required for the benign member (§2.1; example in §D). The preserved AgentHarm benign performance and low XSTest over-refusal provide evidence against blanket caution.

We appreciate your suggestions and will incorporate these clarifications and the additional baseline into the revised manuscript. Please let us know if any of these points would benefit from further explanation.

[1] Jiang et al. SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities. 2025. https://arxiv.org/abs/2502.12025

Dear Reviewer o4zu,

Thank you again for your thoughtful review. As today is the last day of the discussion period, we wanted to follow up on our rebuttal. Please let us know if you have any remaining concerns and we would be happy to clarify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.