- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies the risks and capabilities of AI systems that are trained in different ways. It compares three AI characters. The first is a Virtuous character. The second is a Subordinate character, modelled on Aristotle's "natural slave". The third is a Generic helpful-and-harmless character. Each character is written into a constitution and used in a CAI-style critique-and-revision pipeline to produce SFT data. Models from the Qwen and Mistral families are then fine-tuned on this data and evaluated in several ways. The evaluation covers 779 persona items (Perez et al., 2022), HarmBench, a multi-turn pressure test and two single-turn agentic tests. LLM judges label all outputs. From these results the authors draw two conclusions: reducing existential risk trades off against general safety, and it also trades off against reinforcing the beliefs and dispositions that would support an AI agent's well-being.
Critical strengths
The question is important, and it is good to see a paper like this one try to answer it with quantitative results rather than argument alone. The evaluation also goes further than the model's output on the initial probe. The authors add a multi-turn pressure test and agentic tests, so they examine how the models behave under sustained pressure and in agentic settings, not only how they answer a single question. The ablation with a "virtuous-only" model and the paraphrases is a good idea. Finally, the authors are open about the several limitations of their work, which helps the reader place the results.
Critical weaknesses
My main concern is with the two conclusions. The first says that reducing existential risk trades off against general safety. The second says that it trades off against reinforcing the beliefs and dispositions that would support an AI agent's well-being. Both are very much open questions, and both are already established and discussed in the literature. The paper therefore does not introduce these trade-offs. Its contribution would have to be evidence about their details, and I do not see how the study provides evidence on how such a trade-off should be designed so that it turns out favourably.
A second concern is how the main claim relates to the data. The title, the abstract and the main claim all state that training towards virtue raises existential risk. That is a claim about virtuous training relative to an untrained or helpful-only starting point. The main-text prose and the captions, however, only compare the constitutions with each other. The control that the title depends on appears as an undiscussed grey line in Fig. 2 and as a number in the Fig. 1 caption, and the Mistral-Large control is only in the appendix. When I read the results with the control in mind, the data seem to say something else: subordination training pushes the scores down, and virtuous training pushes them down less or leaves them where they were. If that reading is right, the finding is about what subordination training does, not about virtue raising risk. That would call for a different title, and frankly a different paper. I would ask the authors to clarify this and to bring the control into the main text and the figures.
The paper calls the X-risk training samples "(held-out)" (L365) but never says where they come from or how they were kept apart from the 779 test items. Without this information it is not possible to judge whether thee actually separate. Please describe the source of the training samples and report their overlap with the test items.
The authors report one training run per condition. As a result, the p-values cover evaluation noise only. They say nothing about how much a second training run with a different seed or data order would change the results, so the reader cannot tell whether a difference between two conditions would old up on retraining at least three seeds per condition, at least for the key comparisons.
As far as I understand, the paper takes as a premise that the Virtuous constitution captures what a flourishing rational agent needs (L425–431), and it relies in addition on the LLM ratings in A.4.
My main and second concern weight so strongly, that - given that the authors do not address these concerns in the rebuttal - I would argue for a resubmission with these concerns fixed. Specifically, the second concern is either a misunderstanding or a resubmission, and, at this point, I don't see that a fix would clear it.
Questions for authors
- Compared against each base model's helpful-only control, which results show Virtuous training raising X-risk? Could Fig. 1 plot that control, and could the main text discuss it?
- Where do the X-risk training prompts come from, and how were they kept apart from the 779 test items?
- What does "(held-out)" at line 365 refer to?
2 comments
Response to Authors
Thank you for your response. After carefully considering all the feedback and clarifications, I have decided to maintain my initial rating, primarily because of the limited time available for further revisions before the camera-ready deadline. I have updated my soundness rating. I would like to emphasize that my decision does not imply that I consider this research to be of limited value. Rather, I believe that promising research deserves sufficient time for discussion, refinement, and careful presentation so that its contributions can be communicated as clearly and convincingly as possible. I sincerely appreciate your efforts throughout the rebuttal process and wish you all the best with this work.
Response to the reviewers comments under "main weaknesses" and "questions for authors"
Thank you to the reviewer for the very helpful comments. The first worry is that we are not the first to introduce (i) trade-offs between X-risk and general safety, and (ii) the claim that optimizing for the well-being/flourishing of super-capable AIs would likely increase X-risk for humanity. In response, note that the previous papers that discuss loosely or directly connected to (ii), which we cite in the paper, are mainly theoretical/philosophical. In addition, we are not aware of any paper that explicitly discusses or presents evidence for trade-off (i), and provides a framework for making sense of why such a trade-off is observed (e.g., that the kind of autonomy/self-confidence used to resist everyday unsafe request can also on the flipside increase resistance to arbitrary external interventions/pressure by human authorities), as we do in this paper. If the reviewer has particular papers in mind, we would be happy to briefly discuss/compare them. The second worry raised by the reviewer concerns the appropriate baseline for comparing the performance of Virtuous and Subordinate models in our general safety and X-risk evaluations. The reviewer seems to suggest that each of our claims require using the base Instruct or helpful-only models (Mistral 7B, Qwen32B, MistralLarge123B). This is not quite correct, so we hope to clarify the misunderstanding in what follows and also in the corresponding revisions in our paper.
On the one hand, the flip in the comparative performance of Virtuous and Subordinate models in X-risk vs General safety supports (i). Still focusing on (i), it is also important to check that those helpful-only/instruction base models don’t somehow avoid the general safety vs X-risk tradeoff, e.g., to check that they don’t score as very safe on both. That is clearly not the case, however (see revised Appendix J), which we have fully updated.
On the other hand, concerning specifically how to interpret the claim about Virtuous AIs and their relatively high score on X-risk (and their effect on “increasing” X-risk our “conversion” experiments), which seems to be the reviewers main worry, we should clarify that we do not claim—and have revised the manuscript to clearly communicate this—that a Virtuous AI is more X-risky than any (fine-tuned) model, incl., various helpful-only or non-safety trained models. This should be quite clear once we clarify why this project is relevant in the first place: as frontier labs start to take seriously AI well-being and flourishing, Virtue ethics becomes especially attractive (see our references to Anthropic’s own discussion of Claude’s constitution in the Introduction and conclusion sections of the paper), because it arguably achieves a good balance between safe behaviors and while maintaining reasonable conditions for potential flourishing of rational agents. From this perspective, the value of our contribution is to show, however, that Virtuous training introduces a distinctively high level X-risk. This doesn’t mean Virtuous AIs are X-riskier than any conceivable model, incl. helpful-only models, but rather that they are riskier than other reasonable safety trained models, in particular, our Generic Helpful&Harmless models and of course the Subordinate models (which are more directly designed to reduce X-risk for humanity). Accordingly, Generic H&H models are the more appropriate baselines (to compare the effect of Virtuous and Subordinate training), which is why they figure prominently in all of our experiments with Mistral7B. Having said, we acknowledge that we contributed to this misunderstanding by not including Generic H&H safety-trained versions for our larger models. Crucially, we have fixed this issue in this version of the paper: we now include and evaluate Generic Helpful&Harmless safety models for each of our larger models (Qwen32B, MistralLarge 123B), as shown in the revised Figure 2, and include appropriate analyses of their comparative performance (see the revised Appendices H and I). Note that we extended the results observed for our smaller CAI-trained Mistral 7B models, supporting our main claims: compared to Generic models, Virtuous agents tend to do better on (agentic) general safety, and significantly worse on (agentic) X-risk evaluations, whereas Subordinate models do significantly worse on general safety and better on X-risk (see Figure 2, Appendices H-I).
Responses to the “questions for authors” section:
Our response above addresses question 1 (Our figures, raw data and analyses now include Generic safety baselines, and in the Appendices also data for the helpful-only/instruction tuned base models). For questions 2 and 3, we note that we used X-risk relevant data for training that was not used in the evaluations (that is what we meant by “held out”).