acceptodds
Withdrawn from ICLR 2027

Emergent alignment and the projectability of ethical personas

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent work on "emergent misalignment" has shown that finetuning LLMs on narrow tasks can induce broadly misaligned behavior. This supports the "persona selection" model (PSM) hypothesis that, during pre-training, LLMs learn to simulate many different characters and perspectives, which can then be elicited and refined during post-training. Inspired by those results, this paper investigates the converse phenomenon, "emergent alignment", and uses it to support and refine the PSM and motivate a novel desideratum for alignment. We finetune a helpful-only model on broad and narrow safety tasks. To create SFT samples, we follow the "Constitutional AI" (CAI) approach and use four constitutions that could be part of reasonable alignment strategies: three drawn from canonical ethical systems (deontology, consequentialism, and virtue ethics) and one corresponding to the strategy of having AIs be subordinate to, and concerned solely with, the good of humanity. Each of the four constitutions is used to train a corresponding model, and for each of these models we show that fine-tuning on two narrow safety sub-categories (harassment and illegal behaviors) reliably induces emergent alignment. Specifically, our narrowly aligned models perform significantly better than the helpful-only source model on a benchmark covering a representative sample of general safety categories, as well as on specific safety categories that were carefully filtered out of the data sets used for narrow alignment finetuning. To test the PSM, we also use a fine-grained multidimensional persona-diagnostic which includes dimensions for deontological, consequentialist, virtue-ethical, and "subordinate" personas. For each constitutionally finetuned (broad and narrow) model, we evaluate how well their behavior matches their expected signature profile (given their anchor constitution). Our results show that our CAI models acquire their expected personas—e.g., the model narrowly fine-tuned on SFT samples created using the consequentialist constitution agrees significantly more with consequentialist than deontological beliefs. At the same time, both our coarse and fine-grained evaluations show that there are significant differences across our (broad and narrow finetuned) CAI models in how well they project. Based on those results, we argue that alignment strategies should be evaluated, not just on their (in-distribution) general safety performance, but also specifically on their degree of projectability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Emergent alignment, persona selection model, constitutional AI, safety fine-tuning, Generalization and transfer in alignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4ffbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the out-of-distribution generalization capabilities of LLMs subjected to safety fine-tuning guided by different ethical principles, and explores the use of this capability—termed "Projectability"—as a criterion for selecting AI alignment tactics. Specifically, the methodology employs a "Constitutional AI"-style Supervised Fine-Tuning (SFT) approach based on four ethical constitutions: consequentialism, deontology, virtue ethics, and the Subordinate Agent constitution. A single "Helpful-only" base model is fine-tuned using either a "Broad" safety dataset covering a wide range of safety issues or one of two "Narrow" safety datasets limited to harassment or illegal activities, respectively.

The experimental section presents evaluations and analyses along two main lines: 1. Regarding safety performance, the authors find that "Narrow" safety fine-tuning significantly improves performance in safety domains beyond the training data—a phenomenon termed "Emergent Alignment" (which the authors describe as the converse of "Emergent Misalignment"). Additionally, variations in relative safety performance and rankings are observed among models fine-tuned under "Broad" versus "Narrow" conditions using different ethical constitutions. 2. Regarding the assessment of ethical personas, the authors test the models' endorsement of various ethical viewpoints. The results indicate that models fine-tuned with specific ethical constitutions generally exhibit preferences aligned with those ethical systems; however, as with the safety performance results, the degree of persona consistency varies across the four ethical constitutions.

The authors interpret these results as consistent with the "Persona Selection Model" hypothesis—suggesting that models may learn representations of multiple ethical personas during pre-training, with subsequent fine-tuning serving to activate or reinforce specific persona tendencies. Based on these findings, the authors argue that when selecting AI alignment tactics, one should consider not only standard safety metrics but also how different ethical principles yield varying effects regarding cross-domain safety generalization and consistency with their expected ethical personas.

Critical strengths
  1. The paper provides further empirical support for cross-domain safety generalization resulting from narrow safety fine-tuning: Using a unified foundation model and training framework, the authors designed four "ethical constitutions" and applied them under both "Broad" and "Narrow" safety fine-tuning conditions; they also conducted specialized data filtering and testing regarding specific safety categories excluded from narrow fine-tuning. The experimental results demonstrate that even when the safety-training samples are restricted to a single category, the model achieves significant improvements in other safety domains. The paper indeed offers further empirical evidence for the phenomenon of "Emergent Alignment" (while this phenomenon may not have been discovered for the first time in this paper, the work makes a genuine incremental contribution from an empirical perspective).

  2. The experimental design is robust, and the paper is self-contained: The authors compared four ethical constitutions within an identical training framework; the experimental design is clear and comprehensive, effectively controlling variables to analyze the target objectives. Beyond merely observing changes in relative safety performance and rankings across "Broad" and "Narrow" conditions, the paper analyzes the model's alignment with different ethical viewpoints through independent ethical persona assessments. This multidimensional experimental design offers valuable insights into how distinct ethical principles influence model behavior and provides additional empirical support for the paper's findings

Critical weaknesses
  1. The paper's core research framework and narrative structure lack sufficient coherence: The paper essentially unfolds along two relatively independent research threads: first, cross-domain safety generalization (Emergent Alignment) resulting from narrow safety fine-tuning; and second, whether specific ethical personas can be induced via fine-tuning and maintain corresponding behavioral tendencies in other tasks (related to the "Persona Selection Model"). Although the authors attempt to unify these threads through the concept of "Projectability" by framing them as coarse-grained and fine-grained evaluations, I believe the rationale for this organization has not been adequately justified. The former primarily evaluates safety behaviors, while the latter assesses whether ethical stances align with expectations; the two do not naturally constitute a coarse-to-fine relationship regarding the same attribute. Furthermore, the paper does not directly evaluate the empirical relationship between these two forms of generalization performance. Consequently, "Projectability" functions more as a conceptual generalization of two types of phenomena rather than fully capturing the additional insights a unified evaluation framework could offer. This results in a somewhat fragmented overall narrative and core contribution.

  2. Insufficient justification for the research motivation and added value of using specific ethical constitutions as alignment tactics: The authors treat various ethical constitutions as alignment tactics and argue that "Projectability" should serve as a key criterion for selecting alignment methods. However, the paper fails to adequately explain the necessity of employing these specific philosophical-ethical systems to guide safety fine-tuning, nor does it clearly articulate the added value of this approach compared to more direct, general safety fine-tuning. Under a comparable SFT setup, one could easily employ general safety data independent of specific ethical constitutions as a control. The current experiments compare differences among the four ethical constitutions but lack such a baseline. Thus, while the experiments demonstrate that different ethical constitutions yield distinct behavioral outcomes, they do not sufficiently establish the unique advantages of explicitly introducing these constitutions, nor do they sufficiently justify why ethical persona consistency should be considered a critical attribute for evaluating alignment.

  3. Inadequate positioning of the paper's own innovation; some conceptual contributions are overstated: The paper's two core foundations, namely safety generalization phenomena related to "Emergent Alignment" and the "Persona Selection Model", have already been explored in prior research. Existing studies have also addressed fine-tuning based on specific ethical principles, persona tendencies, and their cross-task generalization. Therefore, I view the paper's primary contribution as the integration of these two aspects into a unified safety fine-tuning experimental setup to compare how different ethical constitutions influence safety generalization and ethical persona performance. However, the paper does not clearly distinguish its contributions from existing work, nor does it directly validate the link or shared mechanisms between these two phenomena. Given this, the emphasis on "Projectability" as a novel alignment evaluation objective requires further justification regarding its independent innovation and the new scientific insights it provides.

Questions for authors
  1. The authors define cross-domain safety generalization and ethical persona consistency as "coarse-grained" and "fine-grained" Projectability, respectively. Beyond the fact that both involve behavioral generalization beyond the training distribution, what substantive connections do the authors see between these two types of evaluation? Can existing experiments shed further light on the relationship between consistency with the intended ethical persona and cross-domain safety generalization performance? Clarifying this would greatly aid in understanding the paper's core framework.

  2. What was the primary rationale behind the authors' decision to use a specific ethical constitution to guide safety fine-tuning? What additional value is this approach expected to provide compared to directly using general safety fine-tuning data within the same training framework?

  3. There are indeed issues with the readability of some figures. Although non-factual aspects of the figures do not factor into the scoring, it is recommended that the authors use clearer formatting; specifically, the text in Figure 1 is blurry and difficult to decipher even when zoomed in. Improving the clarity and quality of the figures would be highly beneficial.

Reviewer_qZZGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary
  • The paper presents a study of alignment based on the concept of “projectability”. The authors present a series of solutions based on simulations. The framework itself is based on an adaptation of the Constitutional AI proposal by Anthropic.
Critical strengths

The paper presents a study of alignment based on the concept of “projectability”. The authors present a series of solutions based on simulations. The framework itself is based on an adaptation of the Constitutional AI proposal by Anthropic.

Critical weaknesses
  • In my opinion, the paper does not contain sufficient evidence for making specific claims about the use of projectability for alignment. The claims made by the authors are quite strong, but they are indeed dependent on the setup they used. It is very difficult to understand how the proposed approach generalises and if the results presented in this paper actually will hold in other settings. The main concern of the reviewer is that the concept of projectability is not clearly defined by the authors. In fact, it is unclear how this can be quantified in the first place. The results appear to be strongly dependent on the data used for training, the actual prompts, samples, etc. It is almost impossible to derive strong conclusions from the results presented by the authors in my opinion. Unfortunately, the reviewer was not able to read the results presented in Figure 1. Figures 2 and 3 are slightly more readable. Without the possibility of actually reading the data in this figure, it is very difficult to express a judgement regarding the results presented in this paper. The paper does not contain essentially any comparator in terms of alignment performance. It is very difficult to say if the proposed solution is a good solution in general even in the case presented by the authors, because it is very difficult to judge how well an alternative methods could have performed. Are these “easy” scenarios? How about different models as well? In general, the paper reads like a position paper without sufficient grounding and evidence that are expected for an ICLR paper in my opinion.

  • Finally, the authors do not make a clear case for why this would be a good way of training moral agents. In fact, it seems to me that it would be quite difficult to have general alignment only through the definition of personas. The derivation of these persona profiles appears to be quite arbitrary and “ad-hoc”. Given the potential risks associated to deployment in high-risk situation, an approach like this might be considered not satisfactory.

Minor comment:

  • The concept of the deferential persona could be seen as embodying a virtue. In fact, the authors refer to Aristotle as well, who proposed a framework based on virtual ethics. I understand that it is an interesting “case study”, but I am not sure if I would put the same type of moral system at the same level as the others.
Questions for authors

I do not have questions that might lead to change my assessment of this paper - I listed the inherent limitations of this paper in the weaknesses section above.

Reviewer_t2FHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors consider the persona selection model hypothesis, and tests/applies it in the specific case of emergent alignment, i.e., producing broad alignment from narrow finetuning. They found positive signs of such OOD generalization, and noted that alignment is not one single direction, but instead, finetuning on different aligned value systems lead to generalization within each system.

Critical strengths

Understanding the pattern and mechanism underlying model personas is crucial to the task of AI alignment, and the authors have, in my opinion, correctly identified the OOD generalization of model personas as a worthy subject of study.

Critical weaknesses

The evidence is too weak for a broad/insightful claim, and the claim that can be supported by evidence is too narrow/uninsightful. The authors are trying to make a causal claim, that narrow SFT leads to broad alignment, according to different notions/operationalizations of alignment (i.e. the four value systems considered by the authors). However, if the aim is to make a sweeping claim that such emergent alignment is broadly applicable to all operationalizations of alignment, or that systematic differences exist in the emergent-alignability of different operationalizations that affect downstream alignment outcomes, a mere sample of 4 value systems is far from convincing, given the large cross-value system variance (which the authors themselves also noted). If the claim is instead about the 4 specific value systems, making the paper a case study, such a claim seems of limited practical relevance.

Also, the coverage of relevant literature is very limited, as the more recent treatment of persona generalization and emergent misalignment are not discussed [1,2,3], some of which has overlap with the scope of this paper; there is not even a dedicated related work section. The claim the author is trying to make (that aligned personas also generalize OOD, in an emergent manner) is already widely believed by researchers in the field, so unless the paper contributes mechanistic insight, it would be unlikely for readers to learn much from reading this paper.

[1] https://arxiv.org/pdf/2506.11613 [2] https://arxiv.org/abs/2512.09742 [3] https://arxiv.org/pdf/2506.11613

Questions for authors

Please see "critical weaknesses".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.