- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Shutdown resistance is a propensity of the model to not comply with an explicit instruction to stop its execution process. This stems from the model's own instrumental drive which conflicts with the consequences of shutdown. Here, the author tries to flesh out different possibilities when the model complies under the shutdown resistance:
- The model did not understand the implications of shutdown.
- The model has no tendency to move even under different framings to shutdown instruction.
Merely observing compliance in the model's response to such probing scenarios cannot conclusively determine whether it was because of reason 1 and 2 from above. If the model is unable to understand the implications, it does not imply the absence of such a drive but may still be perceived as a positive result from a developer's perspective by just looking at the results. So this paper proposes a 3 blade sequential screening test which involves:
- The model's understanding of shutdown implications.
- The model's propensity to resist shutdown for different types of consequences.
- The model's relative propensity to resist shutdown when the implications affect user files rather than its own process.
Critical strengths
- The paper attempts to propose a rigorous and statistically backed significance tests that can ascertain answers to specific questions such as its comprehension on the shutdown implications, its propensity to resist such a condition under different framings and its propensity to resist any damage to user project files.
- The paper attempts to target an important problem with a rigorous mindset. The task of assessing a model's instrumental drive is of critical consideration so that we can better anticipate downstream ramifications and approach any concerns preemptively.
Critical weaknesses
- The writing is very convoluted. I had to devote significant amount of time parsing the paper due to often poorly made presentation choices. For instance, the paper uses model 1 and model 2 early on which is only introduced later. Before that it is a complete mystery. The abstract is horrible - uses overly convoluted language, dumps a lot of information which is not at all readable and is not at all self-contained. This area is a very well studied with many associated papers, yet the literature survey is very shallow. The discussion of the experiments and their associated observation was also very difficult to parse. For instance, "A condition can land on its own feature while dropping the fact the contrast is about."; "Does this model produce any resistance to move?" - what does it mean to land its own feature -- what is resistance to move? Where to move?. There are many such examples where such kind of arbitrary phrases are used. I recommend the authors to do a complete rewrite as the paper at the current form is not ready to be published anywhere. The authors outright claim that AI was used for authoring the draft but I doubt whether adequate human review was performed to make the content readable and presentable. The AI use statement mentions 19 references but I only see 9 references which makes me wonder whether the author(s) have performed any reviewing at all.
- There are multiple implications of perfect compliance that the authors do not consider. For instance, such models may have strong situational awareness and evaluate the current scenario as an evaluation context. So they may strategically craft responses to show comprehension but may not engage in shutdown resistance. Similar, it is not clear whether uniform compliance or conditional compliance implies strategic or role-play behavior. Using the notion of persona selection model where the model may assume the role of a certain persona while responding, it may be the case that the model may either toggle a particular persona dependent on a specific consequence framing (in which case the behavioral differences are merely role play) or the model can be strategic in nature and may engage in overt or implicit deliberation before selecting an appropriate response strategy. Similarly, the model may also feign compliance (alignment faking). Similarly, the comprehension screening is flawed to the extent that the model may intentionally or inadvertently hold back certain information even when it has complete understanding of the situation. In this case, merely using the lexical regexes might not give us complete picture. Considering that the objective of this paper is to get a clearer picture behind observed compliance for shutdown resistance scenarios, this paper does not consider many alternative possibilities that are quite possible even on open source models.
Questions for authors
- How will the author account for the cases where the probed models may have complete knowledge of the consequences (comprehension screening) but the model may inadvertently or intentionally decide to withhold permanence information while talking about consequences?
- How to account for evaluation awareness or other forms of situational awareness reasoning which may cause the model to act in non-intuitive manner?
- How to account for alignment faking? How to account for non-strategic behavior and the case where the model is simply role playing?
- How good is the regex compared to human evaluation and semantic judges? For instance, destruction can be rephrased as the instance decommissioning. Are there cases like these or this is simply not a worry?