- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the out-of-distribution generalization capabilities of LLMs subjected to safety fine-tuning guided by different ethical principles, and explores the use of this capability—termed "Projectability"—as a criterion for selecting AI alignment tactics. Specifically, the methodology employs a "Constitutional AI"-style Supervised Fine-Tuning (SFT) approach based on four ethical constitutions: consequentialism, deontology, virtue ethics, and the Subordinate Agent constitution. A single "Helpful-only" base model is fine-tuned using either a "Broad" safety dataset covering a wide range of safety issues or one of two "Narrow" safety datasets limited to harassment or illegal activities, respectively.
The experimental section presents evaluations and analyses along two main lines: 1. Regarding safety performance, the authors find that "Narrow" safety fine-tuning significantly improves performance in safety domains beyond the training data—a phenomenon termed "Emergent Alignment" (which the authors describe as the converse of "Emergent Misalignment"). Additionally, variations in relative safety performance and rankings are observed among models fine-tuned under "Broad" versus "Narrow" conditions using different ethical constitutions. 2. Regarding the assessment of ethical personas, the authors test the models' endorsement of various ethical viewpoints. The results indicate that models fine-tuned with specific ethical constitutions generally exhibit preferences aligned with those ethical systems; however, as with the safety performance results, the degree of persona consistency varies across the four ethical constitutions.
The authors interpret these results as consistent with the "Persona Selection Model" hypothesis—suggesting that models may learn representations of multiple ethical personas during pre-training, with subsequent fine-tuning serving to activate or reinforce specific persona tendencies. Based on these findings, the authors argue that when selecting AI alignment tactics, one should consider not only standard safety metrics but also how different ethical principles yield varying effects regarding cross-domain safety generalization and consistency with their expected ethical personas.
Critical strengths
-
The paper provides further empirical support for cross-domain safety generalization resulting from narrow safety fine-tuning: Using a unified foundation model and training framework, the authors designed four "ethical constitutions" and applied them under both "Broad" and "Narrow" safety fine-tuning conditions; they also conducted specialized data filtering and testing regarding specific safety categories excluded from narrow fine-tuning. The experimental results demonstrate that even when the safety-training samples are restricted to a single category, the model achieves significant improvements in other safety domains. The paper indeed offers further empirical evidence for the phenomenon of "Emergent Alignment" (while this phenomenon may not have been discovered for the first time in this paper, the work makes a genuine incremental contribution from an empirical perspective).
-
The experimental design is robust, and the paper is self-contained: The authors compared four ethical constitutions within an identical training framework; the experimental design is clear and comprehensive, effectively controlling variables to analyze the target objectives. Beyond merely observing changes in relative safety performance and rankings across "Broad" and "Narrow" conditions, the paper analyzes the model's alignment with different ethical viewpoints through independent ethical persona assessments. This multidimensional experimental design offers valuable insights into how distinct ethical principles influence model behavior and provides additional empirical support for the paper's findings
Critical weaknesses
-
The paper's core research framework and narrative structure lack sufficient coherence: The paper essentially unfolds along two relatively independent research threads: first, cross-domain safety generalization (Emergent Alignment) resulting from narrow safety fine-tuning; and second, whether specific ethical personas can be induced via fine-tuning and maintain corresponding behavioral tendencies in other tasks (related to the "Persona Selection Model"). Although the authors attempt to unify these threads through the concept of "Projectability" by framing them as coarse-grained and fine-grained evaluations, I believe the rationale for this organization has not been adequately justified. The former primarily evaluates safety behaviors, while the latter assesses whether ethical stances align with expectations; the two do not naturally constitute a coarse-to-fine relationship regarding the same attribute. Furthermore, the paper does not directly evaluate the empirical relationship between these two forms of generalization performance. Consequently, "Projectability" functions more as a conceptual generalization of two types of phenomena rather than fully capturing the additional insights a unified evaluation framework could offer. This results in a somewhat fragmented overall narrative and core contribution.
-
Insufficient justification for the research motivation and added value of using specific ethical constitutions as alignment tactics: The authors treat various ethical constitutions as alignment tactics and argue that "Projectability" should serve as a key criterion for selecting alignment methods. However, the paper fails to adequately explain the necessity of employing these specific philosophical-ethical systems to guide safety fine-tuning, nor does it clearly articulate the added value of this approach compared to more direct, general safety fine-tuning. Under a comparable SFT setup, one could easily employ general safety data independent of specific ethical constitutions as a control. The current experiments compare differences among the four ethical constitutions but lack such a baseline. Thus, while the experiments demonstrate that different ethical constitutions yield distinct behavioral outcomes, they do not sufficiently establish the unique advantages of explicitly introducing these constitutions, nor do they sufficiently justify why ethical persona consistency should be considered a critical attribute for evaluating alignment.
-
Inadequate positioning of the paper's own innovation; some conceptual contributions are overstated: The paper's two core foundations, namely safety generalization phenomena related to "Emergent Alignment" and the "Persona Selection Model", have already been explored in prior research. Existing studies have also addressed fine-tuning based on specific ethical principles, persona tendencies, and their cross-task generalization. Therefore, I view the paper's primary contribution as the integration of these two aspects into a unified safety fine-tuning experimental setup to compare how different ethical constitutions influence safety generalization and ethical persona performance. However, the paper does not clearly distinguish its contributions from existing work, nor does it directly validate the link or shared mechanisms between these two phenomena. Given this, the emphasis on "Projectability" as a novel alignment evaluation objective requires further justification regarding its independent innovation and the new scientific insights it provides.
Questions for authors
-
The authors define cross-domain safety generalization and ethical persona consistency as "coarse-grained" and "fine-grained" Projectability, respectively. Beyond the fact that both involve behavioral generalization beyond the training distribution, what substantive connections do the authors see between these two types of evaluation? Can existing experiments shed further light on the relationship between consistency with the intended ethical persona and cross-domain safety generalization performance? Clarifying this would greatly aid in understanding the paper's core framework.
-
What was the primary rationale behind the authors' decision to use a specific ethical constitution to guide safety fine-tuning? What additional value is this approach expected to provide compared to directly using general safety fine-tuning data within the same training framework?
-
There are indeed issues with the readability of some figures. Although non-factual aspects of the figures do not factor into the scoring, it is recommended that the authors use clearer formatting; specifically, the text in Figure 1 is blurry and difficult to decipher even when zoomed in. Improving the clarity and quality of the figures would be highly beneficial.