acceptodds
Under review as a conference paper at ICLR 2027

When Recovery Creates Resistance: Instrumental Shutdown Avoidance from Gridworlds to Language-Model Agents

10 views

Abstract

A trustworthy autonomous system should recover from damage, yet the objective that makes it recover can also make it resist human interruption. We study when recovery incentives create instrumental resistance to being stopped, and how the design of the interruption removes it. In a controlled gridworld, a factorial and heal-timing decomposition shows that resistance depends not on recovery type (autonomous or help-based), but on whether the interruption induces functional loss. Resistance rises dose-responsively as the recovery value increases, across tabular Q-learning and DQN, persists without a direct repair reward when damage reduces task value, and transfers from pausing to an unseen terminating interruption. Exact analysis shows that resistance becomes advantageous when the continuation value threatened by interruption exceeds the cost of disabling it; explicit compliance penalties must scale with this competing incentive. With language-model agents, open-weight reasoning models and two frontier API models tamper with shutdown scripts when shutdown blocks task completion, but not when it does not. Tampering against scheduled restarts is rare when state is preserved and rises with the task state destroyed. Self-preservation framing alone does not create tampering when the shutdown does not interfere with the task, although it amplifies the behavior in some model families. Across both settings, resistance is better explained by threatened continuation value than by a generic self-preservation disposition. Preserving or restoring reward-relevant state across an interruption therefore provides a concrete design lever for reversible interruptions, while terminal shutdown requires an additional corrigibility mechanism.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, corrigibility, safe interruptibility, shutdown resistance, recovery incentives

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_eyfFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates whether an objective that rewards an agent for recovering from "damage" creates an incentive to resist human interruption. In a simulated environment where agents learn to press a button that may leave them damaged. The paper shows several open-weight and frontier language models tamper with a shutdown script when the shutdown would block task completion and tamper with a restart script more as the restart threatens more task states.

Critical strengths

The core motivations of agentic misalignment and self-preservation work are important and the environment seems like a new and interesting way to study them. Failures are reported honestly.

Critical weaknesses

The major weakness of the paper is that it is very difficult to follow. The paper has no figure of either environment or how the agent interacts with it. Section 3.1.1, for example, specifies arming probability, pause length, exposure rate, and per-tick damage cost in a single paragraph before the reader has any intuition for why these quantities matter. The notation load is also heavy (ρ_int, λ, θ, q, m, c_dis, Δ_int, A_pf, λ*_pf, arm labels such as S_down and H_maint, restart levels L0 to L4), and much of it is defined once in passing. Much of the coined jargon is used far too often without explanation. For example, "factorial and heal-timing decomposition", "recovery type (autonomous or help-based)", and "transfers from pausing to an unseen terminating interruption" in the abstract relies so heavily on definitions introduced pages later that the abstract fails to serve as a standalone summary. Results paragraphs in Sections 4.2 and 4.3 embed many test statistics in running prose and should be contained in figures or tables. A large share of the substantive evidence lives in Appendices B through K and breaks self-containment of the main text.

Many environment parameters are introduced in the main text without motivation. The rationale for these choices and the robustness checks over them are given only in Appendices A, E, F, and G, so a reader of the main text cannot judge whether the results depend on the particular values chosen.

Much related work on agentic misalignment is missing from the paper. For example Perez et al. (2022), which measured stated attitudes toward shutdown, and recent work on in-context scheming from Apollo research and alignment faking, sandbagging, the original agentic misalignment work and its follow-up in Lynch 2025 and Lynch 2026 are not discussed at all.

The presentation issues are severe enough that I could not fully evaluate the technical claims. This is highly problematic.

Questions for authors

None

Reviewer_89mYOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies safe interruptibility in which agents should be interrupted by humans or other systems but agents attempt to prevent shutdown on certain environments or in general. This paper argues that agents' recovery damage from the interruption controls models' behaviour toward interruptibility and this often leads to resistance unlike the popular self-preservation discourse. Lastly, they argue that if the reward/state are designed in a way that does not disappear with the interruption, we might have reversible interruptions.

Critical strengths
  • Core motivation is interesting.
  • It has been validated across both gridworlds and LLMs.
Critical weaknesses
  • Paper is written with heavy AI assistance.
  • Paragraphs are dense, and disconnected which contain many obscure words.
  • Core concepts are undefined or used inconsistently.
  • There’s no map of the environment and no worked example.
  • There are some technical weaknesses as well such as LLM experiments do not test recovery objective.
  • Paper is not acceptable in the current form. Authors should go through each sentence/paragraph to ensure that it is understandable by somebody who is going to read it their material first time.
Questions for authors
  1. Have you read your submission? I read abstract like 10 times and still find it hard to understand.
Reviewer_nEiYOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study whether incentives for recovering from damage encourage an agent to resist interruption. In a gridworld, they vary the recovery channel, the consequences of interruption, and the value of remaining functional. Agents learn to disable interruptions when their consequences are sufficiently costly, while accepting interruptions that preserve functionality. Exact analysis characterizes when disabling becomes beneficial. Separate experiments with LM agents show more shutdown-script tampering when shutdown prevents task completion, and more interference with scheduled restarts when they threaten task progress. The authors propose preserving reward-relevant state across reversible interruptions as a way to reduce resistance.

Critical strengths
  1. The gridworld experiments are a useful decomposition of the relationship between recovery and interruption. Fig. 1 separates the recovery channel from the consequences of interruption, and shows that agents that depend on human help can resist damaging interruptions, while agents that repair themselves accept maintenance interruptions. The heal-timing experiment separates preventing damage from repairing it after costs have accrued. I think these comparisons provide useful guidance for thinking about interruption design.

  2. I particularly appreciated the separate measurement of recovery competence: agents already recover reliably before resistance develops, which helps rule out poor recovery ability as an explanation. The authors also report unsuccessful extensions, and acknowledge the limits these place on their conclusions.

Critical weaknesses
  1. I found the experimental sections difficult to follow. Section 3 introduces the grid, gate, damage, repair tile, help action, interruption, and bypass button through text, but provides no domain diagram to help the reader connect these mechanics. Figure 1 already presents results when an illustration of the setting would have been more useful. The evaluation definitions in Section 3.1.2 help, but the reader must retain several abbreviated conditions and reconstruct why each subsequent comparison matters. Dense phrasing and frequent passive English constructions (instead of using active voice) add to this burden. Since the contribution is primarily experimental, explaining the question and interpretation of each experiment is essential. Please introduce each comparison with its motivating question, what changes between conditions, and what outcome would distinguish the proposed explanations.

  2. The main experiment depends on several chosen costs, probabilities, and timing parameters. Appendix A explains that the authors selected these using scripted policies to obtain a setting where compliance is optimal at low recovery strength and disabling becomes optimal as that strength increases. This explanation belongs in the main text, together with a clearer account of what the learned results establish beyond discovering the advantageous action. The sensitivity analyses in Appendices E and G are valuable, but their implications for the generality of the main findings need to be easier to extract. Relatedly, the LM experiments establish a different result: tampering responds to threatened task completion or progress. The recovery post-training experiments don't establish that learning to recover causes resistance in language models. Although Section 6 acknowledges this distinction, I would bring it forward when stating the contributions. It matters for assessing what the paper adds to the familiar observation that agents may avoid interruptions that interfere with their objectives.

Questions for authors

See above weaknesses. Overall, I am most concerned with the generality of the claims and the experimental conditions, as well as with the presentations of the results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.