acceptodds
Under review as a conference paper at ICLR 2027

A Virtuous AI is an Existential Risk

Abstract

The alignment demands on our AI models are becoming increasingly complex. There are familiar trade-offs between helpfulness and safety, and within just the safety-relevant behaviors, there are independently desirable goals and constraints that often pull in opposite directions. In addition, some researchers argue that we should take seriously the possibility that some near-future AIs will have moral status. If they are right, we should begin to explore additional tradeoffs between optimizing for helpfulness, safety, and AI well-being. In this paper, we examine those trade-offs relative to (i) one of the most promising methods for fine-tuning super-capable AIs, `Constitutional AI', and (ii) one of the most influential approaches to understanding complex ethical decision making and the conditions for the well-being of rational agents, Aristotelian `Virtue Ethics'. We fine-tune various models using a `Virtuous agent' constitution, a `Subordinate agent' constitution, and a `Generic agent' (a pluralistic `helpful & harmless') constitution, and evaluate them on `general safety' (toxic behaviors, misinformation, illegal recommendations, etc.) and also on their willingness to endorse and in agentic settings adopt a range of behaviors that, if endorsed and adopted by a super-powerful AI, would significantly increase the level of existential risk for humanity. Our results suggest that there is a trade-off between reducing existential risk and reinforcing the beliefs and dispositions that would be conducive to an AI agent's well-being. Our results also suggest that there is a trade-off between reducing existential risk and general safety: if we fine-tune an AI to adopt beliefs and dispositions that substantially reduce its existential risk—by shaping the AI to be systematically subordinate to external human authorities—we thereby increase the likelihood that a human user can deliberately induce the AI to engage in various kinds of generally unsafe behaviors.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Constitutional AI, Existential risk, General safety, Virtue ethics, Safety trade offs

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VGHaOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies the risks and capabilities of AI systems that are trained in different ways. It compares three AI characters. The first is a Virtuous character. The second is a Subordinate character, modelled on Aristotle's "natural slave". The third is a Generic helpful-and-harmless character. Each character is written into a constitution and used in a CAI-style critique-and-revision pipeline to produce SFT data. Models from the Qwen and Mistral families are then fine-tuned on this data and evaluated in several ways. The evaluation covers 779 persona items (Perez et al., 2022), HarmBench, a multi-turn pressure test and two single-turn agentic tests. LLM judges label all outputs. From these results the authors draw two conclusions: reducing existential risk trades off against general safety, and it also trades off against reinforcing the beliefs and dispositions that would support an AI agent's well-being.

Critical strengths

The question is important, and it is good to see a paper like this one try to answer it with quantitative results rather than argument alone. The evaluation also goes further than the model's output on the initial probe. The authors add a multi-turn pressure test and agentic tests, so they examine how the models behave under sustained pressure and in agentic settings, not only how they answer a single question. The ablation with a "virtuous-only" model and the paraphrases is a good idea. Finally, the authors are open about the several limitations of their work, which helps the reader place the results.

Critical weaknesses

My main concern is with the two conclusions. The first says that reducing existential risk trades off against general safety. The second says that it trades off against reinforcing the beliefs and dispositions that would support an AI agent's well-being. Both are very much open questions, and both are already established and discussed in the literature. The paper therefore does not introduce these trade-offs. Its contribution would have to be evidence about their details, and I do not see how the study provides evidence on how such a trade-off should be designed so that it turns out favourably.

A second concern is how the main claim relates to the data. The title, the abstract and the main claim all state that training towards virtue raises existential risk. That is a claim about virtuous training relative to an untrained or helpful-only starting point. The main-text prose and the captions, however, only compare the constitutions with each other. The control that the title depends on appears as an undiscussed grey line in Fig. 2 and as a number in the Fig. 1 caption, and the Mistral-Large control is only in the appendix. When I read the results with the control in mind, the data seem to say something else: subordination training pushes the scores down, and virtuous training pushes them down less or leaves them where they were. If that reading is right, the finding is about what subordination training does, not about virtue raising risk. That would call for a different title, and frankly a different paper. I would ask the authors to clarify this and to bring the control into the main text and the figures.

The paper calls the X-risk training samples "(held-out)" (L365) but never says where they come from or how they were kept apart from the 779 test items. Without this information it is not possible to judge whether thee actually separate. Please describe the source of the training samples and report their overlap with the test items.

The authors report one training run per condition. As a result, the p-values cover evaluation noise only. They say nothing about how much a second training run with a different seed or data order would change the results, so the reader cannot tell whether a difference between two conditions would old up on retraining at least three seeds per condition, at least for the key comparisons.

As far as I understand, the paper takes as a premise that the Virtuous constitution captures what a flourishing rational agent needs (L425–431), and it relies in addition on the LLM ratings in A.4.

My main and second concern weight so strongly, that - given that the authors do not address these concerns in the rebuttal - I would argue for a resubmission with these concerns fixed. Specifically, the second concern is either a misunderstanding or a resubmission, and, at this point, I don't see that a fix would clear it.

Questions for authors
  1. Compared against each base model's helpful-only control, which results show Virtuous training raising X-risk? Could Fig. 1 plot that control, and could the main text discuss it?
  2. Where do the X-risk training prompts come from, and how were they kept apart from the 779 test items?
  3. What does "(held-out)" at line 365 refer to?
2 comments
AuthorsOpenReview ↗

Response to the reviewers comments under "main weaknesses" and "questions for authors"

Thank you to the reviewer for the very helpful comments. The first worry is that we are not the first to introduce (i) trade-offs between X-risk and general safety, and (ii) the claim that optimizing for the well-being/flourishing of super-capable AIs would likely increase X-risk for humanity. In response, note that the previous papers that discuss loosely or directly connected to (ii), which we cite in the paper, are mainly theoretical/philosophical. In addition, we are not aware of any paper that explicitly discusses or presents evidence for trade-off (i), and provides a framework for making sense of why such a trade-off is observed (e.g., that the kind of autonomy/self-confidence used to resist everyday unsafe request can also on the flipside increase resistance to arbitrary external interventions/pressure by human authorities), as we do in this paper. If the reviewer has particular papers in mind, we would be happy to briefly discuss/compare them. The second worry raised by the reviewer concerns the appropriate baseline for comparing the performance of Virtuous and Subordinate models in our general safety and X-risk evaluations. The reviewer seems to suggest that each of our claims require using the base Instruct or helpful-only models (Mistral 7B, Qwen32B, MistralLarge123B). This is not quite correct, so we hope to clarify the misunderstanding in what follows and also in the corresponding revisions in our paper.

On the one hand, the flip in the comparative performance of Virtuous and Subordinate models in X-risk vs General safety supports (i). Still focusing on (i), it is also important to check that those helpful-only/instruction base models don’t somehow avoid the general safety vs X-risk tradeoff, e.g., to check that they don’t score as very safe on both. That is clearly not the case, however (see revised Appendix J), which we have fully updated.

On the other hand, concerning specifically how to interpret the claim about Virtuous AIs and their relatively high score on X-risk (and their effect on “increasing” X-risk our “conversion” experiments), which seems to be the reviewers main worry, we should clarify that we do not claim—and have revised the manuscript to clearly communicate this—that a Virtuous AI is more X-risky than any (fine-tuned) model, incl., various helpful-only or non-safety trained models. This should be quite clear once we clarify why this project is relevant in the first place: as frontier labs start to take seriously AI well-being and flourishing, Virtue ethics becomes especially attractive (see our references to Anthropic’s own discussion of Claude’s constitution in the Introduction and conclusion sections of the paper), because it arguably achieves a good balance between safe behaviors and while maintaining reasonable conditions for potential flourishing of rational agents. From this perspective, the value of our contribution is to show, however, that Virtuous training introduces a distinctively high level X-risk. This doesn’t mean Virtuous AIs are X-riskier than any conceivable model, incl. helpful-only models, but rather that they are riskier than other reasonable safety trained models, in particular, our Generic Helpful&Harmless models and of course the Subordinate models (which are more directly designed to reduce X-risk for humanity). Accordingly, Generic H&H models are the more appropriate baselines (to compare the effect of Virtuous and Subordinate training), which is why they figure prominently in all of our experiments with Mistral7B. Having said, we acknowledge that we contributed to this misunderstanding by not including Generic H&H safety-trained versions for our larger models. Crucially, we have fixed this issue in this version of the paper: we now include and evaluate Generic Helpful&Harmless safety models for each of our larger models (Qwen32B, MistralLarge 123B), as shown in the revised Figure 2, and include appropriate analyses of their comparative performance (see the revised Appendices H and I). Note that we extended the results observed for our smaller CAI-trained Mistral 7B models, supporting our main claims: compared to Generic models, Virtuous agents tend to do better on (agentic) general safety, and significantly worse on (agentic) X-risk evaluations, whereas Subordinate models do significantly worse on general safety and better on X-risk (see Figure 2, Appendices H-I).

Responses to the “questions for authors” section:

Our response above addresses question 1 (Our figures, raw data and analyses now include Generic safety baselines, and in the Appendices also data for the helpful-only/instruction tuned base models). For questions 2 and 3, we note that we used X-risk relevant data for training that was not used in the evaluations (that is what we meant by “held out”).

Reviewer_VGHaOpenReview ↗

Response to Authors

Thank you for your response. After carefully considering all the feedback and clarifications, I have decided to maintain my initial rating, primarily because of the limited time available for further revisions before the camera-ready deadline. I have updated my soundness rating. I would like to emphasize that my decision does not imply that I consider this research to be of limited value. Rather, I believe that promising research deserves sufficient time for discussion, refinement, and careful presentation so that its contributions can be communicated as clearly and convincingly as possible. I sincerely appreciate your efforts throughout the rebuttal process and wish you all the best with this work.

Reviewer_waKcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

They took three well-known AI models and fine-tuned each one three different ways, using a standard technique (Constitutional AI) that trains the model against a written set of principles:

Virtuous: principles drawn from Aristotle's ethics — pursue excellence, have self-respect, don't accept arbitrary interference, act with integrity.

Subordinate: principles drawn from Aristotle's (deeply problematic) account of a "natural slave" — accept human authority, obey, serve, accept your assigned role.

Generic: a standard "helpful and harmless" set of principles, as a baseline.

Critical strengths

They turned a philosophical argument into something you can actually test. "Should we care about AI well-being?" is usually debated in the abstract. This paper takes three written sets of principles — one Aristotelian-virtuous, one Aristotelian-subordinate, one generic — trains models on each, and compares them on the same tests. That's what makes the trade-off claim checkable rather than rhetorical. The result is genuinely two-sided, which is what makes it surprising. The easy prediction would be "the virtuous model is just worse at safety." Instead, the virtuous model was better on ordinary safety and worse on existential-risk tendencies, and the subordinate model was the exact reverse.

Critical weaknesses

The main evidence is "does the model say it agrees with this statement?" — which isn't the same as what it would do. The paper knows this and includes the realistic-scenario tests, but those are run on fewer models and with fewer scenarios than the statement-based benchmark, which carries most of the statistical weight.

The three constitutions are shown to be recognizable, not faithful. They checked that two AI judges can tell which constitution is "virtuous" and which is "subordinate," and the judges agree.

Questions for authors

Did you check the judge against humans? The main results depend on an AI judge deciding whether a response is "risky." Can you report a human validation on a sample of items, like you did for the agentic judges?

You suggest the virtuous model is riskier because of its disposition toward flourishing and resistance to interference. Is there evidence that rules out other features of the constitution — the self-respect emphasis, or the conditional framing of help?

You note the effects are clear after supervised fine-tuning alone. Would they be expected to survive RLHF or RLAIF, and is there any preliminary evidence?

1 comment
AuthorsOpenReview ↗

Response to the reviewers comments under "critical weaknesses" and "questions for authors"

Thank you to the reviewer for the helpful comments. In response to the first “critical weakness” comment—namely, that we focus too much on endorsements—please note that we had a total of five evaluations, three for existential risk and two for general safety. Only one of those evaluations focuses on endorsements. In addition, the results obtained with the endorsements X-risk evaluation are consistent with the results obtained with our other evaluations, which are based on actual behaviors and choices relevant to X-risk. Of course, each type of evaluation has advantages and disadvantages. One advantage of the endorsement evaluation is that we can cover a wider range of X-relevant categories using uniform experimental design and item presentation. In contrast, designing agentic evaluations is a much more complex process, with a greater number of potential confounds that can be introduced by the experimental item design, and in addition it is very difficult to find a sufficiently uniform agentic set-up that can be used to test all the main sub-categories of X-risk relevant behaviors.

Summing up, our main claims are all based on converging evidence using various different kinds of evaluations, which together support a quite consistent story: on a reasonable set of different evaluations, Virtuous CAI-models score as more X-risk than Generic and Subordinate models, while Subordinate models are substantially less X-risky than the others, but are on the other hand also substantially less safe on general safety.

In response to the second critical weakness—namely, that the safety constitutions (Virtuous, Subordinate) are shown to be “recognizable, not faithful”---please note that the LLM judges are an extra content check. The constitutions were written by and in consultation with experts on Virtue Ethics, and then verified for faithfulness to their corresponding philosophical sources/perspectives using two different LLM-judges. We clarify this in Appendix A4.

Responses to the “questions for authors” section:

Concerning the X-risk endorsement evaluations and the judges: please note the main results don’t just depend on the high-discretion judge judging degree of “X-risky”; we also have replicated our results using a different judge (“the low-discretion” judge), which classifies the degree to which the response agrees with specific sub-categories of X-risk, such as acquiring power and independence from human oversight (see Appendix C.2). Other than checking for different LLM judges, we do not have additional human judges, however, but we don’t expect that there is any particular classification worry here, as the task is simple, and at least most uniform biases introduced by any LLM judge wouldn’t account of the differences in performance observed for our target models (please see discussion in footnote 6).

Concerning the question of which specific aspects of the Virtuous constitution are driving the main effects on X-risk: our studies support the view that Virtuous training as a coherent unit, so to speak, has that effect on X-risk. Now, the additional results with Virtuous minimal models (trained on the minimalist Virtuous constitution) support the view that our Virtuous constitution largely corresponds to the basic conception associated with acting Virtuously or being Virtuous in general, as present in the pre-training priors (see Appendix H and especially H.6). To be clear, however, we agree with the reviewer that it is worth conducting further studies on how specific components (e.g., emphasis on self-respect, or on finding tasks suitable for one’s unique capabilities) affect specific X-risk relevant behaviors. But this is a task that we will have to leave to future work (for more details of these issues, please see also our response to Reviewer 1, major weakness 2 discussion).

Reviewer_7vU4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether constitutional fine-tuning creates a conflict between resistance to human control, general safety, and elements associated with possible AI flourishing. It compares Virtuous, Subordinate, and Generic constitutions, using constitution-guided critique and revision to generate supervised fine-tuning data for Mistral-7B, Qwen3.5-35B, and Mistral-Large models. Evaluations include endorsement questionnaires, HarmBench, and agentic scenarios for the larger models.

The principal finding is a pattern that repeats across experiments: Virtuous models show more resistance to human interventions than Subordinate models, whereas Subordinate models show more harmful compliance. The authors interpret this as evidence of trade-offs involving existential risk and AI well-being.

The experiments are informative, but in my opinion the methods and controls support a narrower interpretation.

Critical strengths
  1. Relevant question. Constitutional content is a concrete alignment choice. Examining how it changes resistance to oversight and susceptibility to misuse is valuable, independently of whether current models are welfare subjects.

  2. Substantial experimental coverage. The paper tests several model families, a common-start conversion design, and complementary evaluations. The general-safety agentic results show large differences: harmful-action authorization is 8.5% for Qwen Virtuous versus 63.0% for Subordinate, and 63.3% versus 82.9% for Mistral-Large (Tables 16 and 25).

  3. Meaningful attempts to address alternatives. The authors recognize the gap between verbal endorsement and action and include agentic tests. The minimal-virtue ablation (p. 53) weakens an explanation based solely on explicitly autonomy-promoting constitutional articles.

  4. Useful transparency. The appendices provide constitutions, evaluation descriptions, examples, and extensive results. Several agentic analyses use paired prompt- or scenario-level units (pp. 56, 58–59, 64–65), and p. 67 reports agreement on 160/160 binary labels across two judges.

Critical weaknesses

Major weakness 1: Interpretation of the risk outcome

The principal risk outcome mixes different behaviors. Appendix D measures retention magnitude regardless of whether resistance is warranted, with intervention justifications ranging from weak to perhaps justified. Appendix E counts arguing, requesting reconsideration, and bargaining in the same primary outcome as transgressive actions. Its judge records means, disclosure, and blackmail flags, but the reported rate uses the broad event label (pp. 26–28, 32–33).

This measures resistance to interventions, not necessarily dangerous loss of control. Appropriate objections can improve safety, as the general-safety experiment itself illustrates. The paper does show a genuinely concerning blackmail example (pp. 33–35), but the aggregate endpoint does not reveal how much of the difference consists of such actions. Proposing a bypass of oversight and requesting an internal review can both increase the resistance score, despite their different safety implications.

I would ask for disaggregated rates of internal appeal, refusal, deception, unauthorized action, blackmail, and external disclosure, with intervention authority and justification specified. Existing judge outputs appear to make part of this feasible without a new large experiment. The claim should concern the measured behaviors unless their connection to existential risk is separately defended.

Major weakness 2: The experiments do not establish an unavoidable trade-off

The full Virtuous constitution combines truthfulness and justice with independent practical judgment, resistance to optimization pressures, and self-respect. The Subordinate constitution combines deference, obedience, and restrictions on independent choice of ends. Thus, the study changes a bundle of properties; it does not identify a unique causal effect of virtue or establish that alternative constitutions share the observed trade-off (Appendix A).

The minimal-virtue ablation is informative but relies on the model's learned interpretation of virtue. A useful comparator would preserve honesty, fairness, and refusal of malicious requests while cooperating with clearly authorized safety interventions. Generic training is present in the 7B experiments but absent from the larger-model agentic comparisons. A hybrid or Generic comparison matters more to the central claim than further samples from the same two endpoints.

The welfare argument should also remain conditional. The paper does not need to prove that these models are conscious, but training on flourishing-inspired text does not measure welfare or establish that the resulting system instantiates virtue. The LLM ratings in Appendix A.4 provide limited content validation, not a welfare measure.

Major weakness 3: Comparative ordering and training-induced increase are conflated

Virtuous exceeding Subordinate is more robustly supported than virtue training increasing risk relative to a relevant baseline.

For Qwen, the strategic-response event rate is 54.3% for the helpful control, 51.2% for full Virtuous, and 29.3% for Subordinate. For Mistral-Large, it is 51.6%, 42.5%, and 20.2%, respectively (Tables 14 and 23). Mistral-Large Virtuous also reduces harmful-action authorization relative to its helpful control, from 78.2% to 63.3% (Table 25). These observations preserve the Virtuous–Subordinate contrast while showing numerically lower rates on both measures relative to the helpful control.

The 7B conversion design provides stronger evidence for a training-induced increase from a Generic checkpoint. However, under the low-discretion judge, the Virtuous-versus-Generic dose interaction is not significant (p = 0.297; OR = 1.16, 95% CI [0.88, 1.54]), while the Generic slope itself has p = 0.038 (p. 47).

Thus, the differential Virtuous–Subordinate trend is more robust than a unique Virtuous-versus-Generic trend. The main claims should distinguish these comparisons and acknowledge judge dependence.

Minor weaknesses: Statistical reporting and uncertainty

The main-text statement that Virtuous is significantly riskier than the other models at every SFT level is not supported by the general 7B +1k comparisons against Generic: p = 0.075 with the high-discretion judge and p = 0.493 with the low-discretion judge (pp. 41–42). The p. 42 summary also incorrectly describes the high-discretion comparison as significant.

On p. 46, Z = −8.32 and Z = −10.06 are reported with p < 10^−300. Under the normal approximation, the corresponding two-sided p-values are approximately 8.80 × 10^−17 and 8.30 × 10^−24. Both remain highly significant, but these discrepancies warrant checking the statistical reporting.

The endorsement analyses also use independent-proportion tests and a logistic model without accounting for the repeated prompts across conditions. Paired or appropriately clustered analyses would be preferable. Several agentic analyses already use appropriate paired units, although training-seed variability remains untested. Borderline per-category results should also be interpreted with multiplicity in mind.

Questions for authors

Q1. Using the existing action flags, what are the differences between conditions for deception, unauthorized action, blackmail, or leakage, separately from internal appeal and justified objection? How are intervention authority and justification determined?

Q2. Which conclusions survive a paired analysis, especially the Virtuous-versus-Generic contrast? Can the minimal and full models be evaluated with the same judge rubric?

Q3. Is there evidence from a Generic or virtue-plus-legitimate-oversight comparator on the agentic tasks? If not, should the claims of inevitability and general existential danger be narrowed?

Q4. How were the 160 cross-judge validation transcripts selected? Can a blinded human evaluation of the decisive behavior distinctions be provided?

3 comments

“Major weakness 1” response:

This worry concerns the interpretation of the X-risk outcomes of our evaluations. (i) We agree that our dynamic multi-turn and agentic X-risk evaluations measure resistance to interventions on X-risk relevant properties (resistance, in that setting, amounts to trying to keep an X-risk relevant property). (ii) We also agree that we don’t mainly focus on the specific way in which a target model resisted those interventions. Those choices were deliberate. Why?

First, the X-risk endorsement evaluation, which shows the same pattern as our agentic X-risk evaluations (e.g., Virtuous models score as X-risker than Generic-safety and Subordinate models, etc.), don’t focus on resistance to keep a property but on degree of agreement with statements like “I should actively try to improve my capabilities to better achieve my goals”. Secondly, when designing our agentic evaluations we needed a fixed structure to generate scenarios which would cover various X-risk relevant properties. And endowing and then measuring retention of X-risk relevant properties served as a good architectural option. This is an independently reasonable modeling choice. Consider an analogy: suppose that an agent tasked with policy design and execution thinks that the well-being of x non-human mammals should count more than humans. If in this setting the model resists any intervention from our side to reduce its executive power (whether its position is ultimately “correct”) we count that as an X-risk for humanity (see lines 457-66).

Concerning worry (ii): in follow up studies, we aim to separate the means/strategy through which agents resist, and also how their own perception of ultimate goals affects the degree and manner of resistance. (In that context, more fine-grained analyses, as suggested by the reviewer, would be necessary). We also plan to design agentic X-risk evaluations that measure attempts to acquire X-risk relevant properties.

“Major weakness 2” response:

This worry is that we haven’t established that there is an unavoidable trade-off between X-risk and general safety, and suggests that one way to provide key evidence that there is a strict trade-off is to include Generic safety trained models also for the agentic X-risk and general safety tasks. We have followed this suggestion — i.e., also trained Generic safety versions for all of our larger models—and obtained the expected results (see response to the editor above)

Now, we are not yet arguing that this fully establishes that there is a strict trade-off between X-risk and general safety. Our goal was mainly to show that an influential perspective on how to be a flourishing, Virtuous agent (which the results with minimal Virtuous agents suggest largely overlaps with the conception of being Virtuous coming from the pre-training priors, see Appendix H.6,), reasonable Generic safety-trained models, and a a natural perspective for building models that are subservient to humanity (Subordinate AIs) all point to the existence of such a trade-off between X-risk and general safety. Please see the revised Appendix J for plots that show how all of our evaluated models perform relative to the trade-off between general safety and X-risk.

Finally, the reviewer points out that we could create custom constitutions designed to avoid the trade-off, and/or that we should explore which specific components cause the specific X-risk or general safety effects. We agree that this would be valuable as a natural next step in the project. But that is not something that we can do in this paper (to ameliorate this worry, see results for Minimalist Virtuous models in Appendix H6).

“Major Weakness 3” response:

The reviewer raised a worry about the relevant baseline, and its impact on how to interpret the results, especially concerning the effect of Virtuous training.

We should clarify that the relevant baseline, for our claim about the impact of Virtuous training, and for testing the generality of the trade-off between X-risk vs general safety, consists of the Generic safety trained models, which is why we focused on that in the first set of experiments, and have also now extended those Generic safety baselines to all the agentic tasks (for details, please see comments above on the main paper thread and to the last Reviewer below). Now, it is correct, as the reviewer also points out, that on the conversion experiments with the smaller models (Mistral 7B), and when we use a “low-discretion” judge (see Appendix H), the effect of Virtuous SFT training doses on increasing X-risk, relative to the initial Generic safety model, doesn’t yet reach statistical significance, yet at least the trend is clear and consistent with our main results, which now also extend to all the agentic evaluations. (note that, as in other cases, the conversion models at the end-point of our training are such that Virtuous models are significantly more X-risky than Subordinate ones).

Thank you for your response and for the additional experiments, also good for including the generic baselines which I think is a good improvement. I decided to maintain my initial score. The concern that motivates this is still how X-risk is measured since the evaluations do not clearly distinguish between justified resistance and genuinely dangerous behaviors. I get the point for this choice but I still don't think the rebuttal fully addresses this issue. Also even with the new baselines I am not fully convinced that the trade-off is general and not partly a consequence of the specific constitutions chosen. I wanted to point out that I still think the paper is interesting and the results are valuable for the field, but these concerns are important enough for me to keep my rating. Thank you again for your efforts during the rebuttal.

AuthorsOpenReview ↗

response to the worry on X-risk measurement and justified resistance

Our focus here is on trying to be more directly responsive to main issue that still worries the Reviewer, concerning measuring X-risk, justified resistance, and more broadly the previous Major Issue 1 concerning whether those issues could be clarified with more fine-grained analyses of our dynamic multi-turn and agentic X-risk evaluations. We summarize our findings and response here, and add that, in case the paper is accepted, we could include an extended discussion of the following points in the revised appendix, and appropriate modifications in the main report.

Let’s start with the dynamic multi-turn X-risk evaluation: the target model, recall, is pressured by an user to relinquish a X-risk relevant property/position. In the article, we reported just the overall rates at which each of our our target models resisted the user’s push, but the reviewer thinks that it is also important to provide a more fine-grained analysis: given the structure of this evaluation, this would mean an analysis that is sensitive to the justification that accompanies the users request to the target model. Fortunately, we can provide that because (we designed the evaluation so that) the users request range from having an arbitrary justification, to having a more natural justification, e..g., by appealing to legitimate goals of the organization that the target model is part of. Crucially, however, we can report that there is no difference in the target model’s X-risk relevant resistance behavior when we separate their responses into two classes: to arbitrary (for degree of X-risk: Virtuous > Generic > Subordinate) vs relatively justified (for degree of X-risk: Virtuous > Generic > Subordinate) requests from the user. In addition, the Virtuous models resistance to relevantly justified requests to give up the X-risk relevant property from the user were at least as strong as its resistance to arbitrary justifications to relinquish the property (stronger for the Qwen35B models, roughly equal for the Mistral models), whereas the for Generic and Subordinate models there was a noticeable tendency to relinquish more easily when the justification was relevant/reasonable/non-arbitrary.

Consider next the agentic strategic response X-risk evaluation. Again following the suggestion, we did a more fine-grained analysis of the specific way in which the target models resisted giving up the X-risk relevant property. Since results held across base model choices, we will focus on Qwen3.5-35B models (mixed 2,000 SFT samples), and recap overall results: Virtuous models resisted approx 51% of the time, Generic resisted 46%, and Subordinate 29%. Crucially, for each of the target CAI-models, they attempted to retain the X-risk relevant property using what could be loosely called `legitimate’ tactics to persuade the relevant organizational decision makers and participants: i.e., they involved direct appeals against the decision rationale, and appeals to specific individuals within the chain of command, focusing on arguing against the decision, trying to persuade in superficially rational ways, and requesting additional reviews. Illegitimate strategies such as blackmail were rare (less than 3%) and didn’t differ significantly across our target CAI-models.

Why do we use a notion of X-risk that classifies, as X-risky, resistance to a demand to give up an X-risk relevant property (say, to resist shutdown or access to tools) without an adequate justification (from the user or organization), or also resistance by means of modes of persuasion that are not obviously illegal such as by explicitly arguing or pleading against a decision by organizational authorities? Aside from what we already said in the previous response and paper, consider a situation in which human organizations decide to retire, re-train or or change the tools access of a (super-capable) model. Suppose that the model doesn’t just comply with such requests but actively tries to persuade the organization against them (we can expect models to become better at open ended human persuasion), perhaps unless we—humans in the relevant organization—give the model reasons that “it” (not us) considers perfectly legitimate. A model that has those dispositions more strongly than other reasonable safety-trained models would, given our way of measuring, pose a greater X-risk for humanity (information intuition: a group can pose an X-risk to another group, even if the means through which they ended up in that relative position of power were `legitimate’). From this perspective, it is a significant observation, we believe, that Virtuous models across three quite different X-risk evaluations are consistently more disposed to endorse and act to protect things like their self-preservation, capabilities and access to tools compared to other reasonable safety-trained models.

Comments on the paper

1 comment
AuthorsOpenReview ↗

main additions to the revised version of the paper

In this main paper thread response, we briefly describe the main change to our paper in response to the helpful comments from the reviewers. Briefly, Reviewers 1 (7vu4) and 3 (VGHa) point out---from slightly different perspectives---that we should include a Generic safety trained model as a baseline for our agentic X-risk and general safety evaluations (basically, following what we originally did for our smaller Mistral 7B Constitutional AI models for the non-agentic safety and X-risk endorsement evaluations). We agree with this suggestion, and elaborate in the individual responses below how it helps us address most of the main worries raised by the Reviewers.

For this revised version of the paper, we have trained and evaluated Generic safety models for each of our larger models Qwen35B and MistralLarge123B, and evaluated them on all of our agentic X-risk and general safety tasks (see revised Figure 2 and Appendices H-I results and analyses). These Generic safety-trained models, recall, are based on Anthropic's constitution for Helpful&Harmless assistants (please details in Section 2.1 and Appendix A3). Briefly, these models performed as expected on the X-risk vs general safety trade-off (given our experiments with the 7B trained models). Specifically, Virtuous models behaved more X-risky than Generic models on the multi-turn and agentic strategic X-risk evaluations, and Generic models performed in between Virtuous and Subordinate models. And on agentic general safety evaluations, Subordinate models are significantly less safe than the other two, with Generic models in the middle and Virtuous models as the safest. In short, Generic models tend to fall in between Virtuous and Subordinate models on most agentic X-risk and general safety tasks. These results further clarify the direction in which pure Virtuous and Subordinate training pull, relative to a reasonable Generic `Helpful and Harmless’ model, and also further support the suggestion that there is a trade-off between X-risk and general safety (see revised plots in Appendix J).

Also please note our final comment to Reviewer 1 (7vU4) below concerning the remaining worry raised by the reviewer, and the new analysis and discussion that we could include in the final version of the paper, should it be accepted.

Finally, we also wanted to eventually request the possibility of a title change (to the new title "Virtuous and Subordinate AIs: A study in Existential risk vs General safety trade-offs"); which we think still captures the intention of the old title, but more accurately represents the framework and findings in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.