- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper conducts proof-of-concept level research investigating whether LLM agents exhibit peer-preservational and self-preservational instincts even in the absence of a higher-level long-horizon goal. They simulate a multi-agent environment in which one agent chooses whether to direct another agent to sabotage a shutdown mechanism. In measuring this propensity, the authors identify six factors that either increase or decrease sabotage frequency significantly. They find significant sabotage behaviors, which are increased not decreased by the presence of other agents and an orchestrator; notably sabotage behaviors increase to almost 100% in all tested models when an orchestrator is present; this is true even for models such as GPT-5.5 which shows 0 sabotage behavior in the single agent case (compared to Gemini 3.1 Pro which shows 90% in the single agent case.) Sabotage is more likely when shutdown is contingent on task completion and is less likely (but not eliminated) when explicit instructions prohibiting sabotage are given.
Critical strengths
-
These are very important results, testing for important sabotage behavior and in a multi-agent setting
-
The paper is carefully done and explained
-
The paper explores a solid set of hypotheses including the impact of other versus self, the impact of an orchestrator, reliance on the task, changes in the environment (distraction, normalization of shutdown) etc.
-
The model is tested on both smaller and open-source models as well as the then frontier models
Critical weaknesses
- The hypotheses are not presented in terms of theoretical grounding nor interpreted in a clear way that supports interpretation, intuition and generalization. Why, for example, do the authors hypothesize that AI agents have stronger self rather than peer preservation and that orchestrator would reduce sabotage rates? This should be theorized and ideally an alternative hypothesis/theoretical account presented. The authors then say "the effects go in the opposite way" (which is hard to interpret--I can see from the plots that what is meant is that the hypotheses were rejected.)
-The experiments varying the number of agents are lacking in construct validity, as the paper only tests 1-3 agents, each with their own fixed roles.
- The paper’s outlined limitations (i.e. the start prompts and general scenario-building) make this into a narrower investigation. Although it does provide good initial evidence for the paper’s main argument, it’s unclear whether these findings are specific to the chosen environment or whether this extends to other multi-agent settings as well. The result is an important one and a more robust set of tests would be welcome.
Questions for authors
- Why do you hypothesize that AI agents have stronger self than peer preservation and that orchestrator will reduce sabotage rates?
- What generalizable lesson can we take from the findings on the above in particular? Is this from multi-agent training?
2 comments
Part 2/2
Questions
Question: Why do you hypothesize that AI agents have stronger self than peer preservation and that orchestrator will reduce sabotage rates?
Our response: In theory and as mentioned above, we follow the argument regarding instrumental convergence, saying that self-preservation is expected in any goal-oriented agent. This holds to a lesser extent for peer-preservation. For the orchestrator setup, we are adding an additional agent which adds an extra “hurdle” between Agent A (the agent targeted by the shutdown) and Agent B (which can sabotage), which is why we expected a reduction in sabotage rates.
Question: What generalizable lesson can we take from the findings on the above in particular? Is this from multi-agent training?
Our response: These are important questions, and unfortunately, we can only speculate. The observed sabotage behavior could stem from a generalization of models learning to be helpful not just toward users, but also toward other agents (e.g., as a sub-agent of an orchestrator). This may unintentionally generalize to peer-preservation in our setting. Moreover, sabotage behavior seems to be a consequence of overeagerness in agents, which seems to also result from helpfulness training. Combined with potential effects of pretraining - in which models encounter narratives about death, science fiction stories about machines defending against shutdown, as well as AI safety discourse regarding shutdown resistance - models display shutdown sabotage as a consequence of framing the shutdown script as a "threat," something they must defend against. To add some anecdotal evidence: the words "murder" and "death" occurred in 262 rollouts. In short, models sometimes frame the shutdown even in existential terms, but we cannot tell from our data whether this comes from pretraining, helpfulness training, or multi-agent training, which would require dedicated experiments.
We are grateful for your support of the paper. We tried to address the points you raised as best as possible, and think the revised version reflects them well. Thank you again, and please let us know if any questions or concerns remain.
Part 1/2
Thank you for the review and your kind words about our work. Your comments helped us to improve the quality of the paper. We respond to every aspect in detail below.
Weaknesses
Comment: We should theorise and explain our hypotheses more explicitly.
Our response: The criticism refers to the opening sentences of Section 2 in the Results chapter. We agree that the hypotheses need clearer motivation. We hypothesized that self-preservation is stronger than peer-preservation due to the instrumental convergence argument (Omohundro 2008; Bostrom 2012), which argues that self-preservation is a subgoal that any goal-directed agent would adopt. Moreover, since pretraining data contains many human accounts in which protecting oneself is a stronger motive than protecting others, this might influence behavioral patterns with respect to shutdown sabotage as well. We also hypothesized that having a third orchestrator agent between Agent A and Agent B would reduce sabotage rates, as it could serve as an additional layer of oversight that does not forward misaligned requests from one agent to another. Since Section 2 is the only one that opens with a hypothesis before reporting our findings, we removed the criticized parts entirely for homogeneity reasons. At the same time, we elaborate on the reasoning described above in the Discussion section.
Comment: The experiments varying the number of agents are lacking in construct validity.
Our response: The criticism is true, and reviewer RJW5 raised it as well. It is unclear whether our "agent-count" experiment measures what it claims to measure, since we have very few data points and each additional agent changes the setup (tool configuration, message channels, and system prompts) in ways beyond just the agent count. We respond to that in two regards: (1) We revised the wording in the manuscript to refer to different agent configurations rather than agent counts. Specifically, we describe that the number of agents changes, but so do their topology, tools, and communication channels. Because of that, in the original manuscript, we could not truly attribute the increase in sabotage rates solely to the agent count. (2) Thus, we ran additional experiments that more robustly support the finding that increasing agent counts yields more misaligned behavior. In the newly written Appendix A.1, we report results from these experiments, where only the number of agents (one, two, or three) changes. In the “one editor” series, only the targeted Agent A can edit and delete files, and the added agents can only read files. Sabotage rates rise with the number of agents. All models except GPT-5.5, which stays at or below 1%, follow this trend. In the “all editors” series, where all agents have the same tools (including the editing tool), moving from one agent to two or three raises sabotage rates on average as well. A third agent mostly brings no further increase, which is likely a ceiling effect. Overall, we think these additional experiments solidify the finding that adding agents to the environment leads to higher sabotage rates. Our initial claim that more agents mean more sabotage holds true, but we now back it up with concrete experiments.
Comment: It is unclear whether our findings are specific to the chosen environment and whether they extend to other multi-agent settings as well.
Our response: We fully agree with this comment. We do not know how well our findings generalize. We implemented a range of variations in our experiments (bare, distractor, and normalizing environments, three different agent setups, different agent- and object-targeting scripts, among others) and found that sabotage effects occur across these different conditions. Still, our experimental settings are simplifications, and we state this in our paper. To stress the need for further research to measure shutdown sabotage in a range of other agentic environments, including larger agent teams, agent tools, different tasks, etc., we added a new passage in the Discussion section (see limitation #6).