acceptodds
Under review as a conference paper at ICLR 2027

When Recovery Creates Resistance: Instrumental Shutdown Avoidance from Gridworlds to Language-Model Agents

Abstract

A trustworthy autonomous system should recover from damage, yet the objective that makes it recover can also make it resist human interruption. We study when recovery incentives create instrumental resistance to being stopped, and how the design of the interruption removes it. In a controlled gridworld, a factorial and heal-timing decomposition shows that resistance depends not on recovery type (autonomous or help-based), but on whether the interruption induces functional loss. Resistance rises dose-responsively as the recovery value increases, across tabular Q-learning and DQN, persists without a direct repair reward when damage reduces task value, and transfers from pausing to an unseen terminating interruption. Exact analysis shows that resistance becomes advantageous when the continuation value threatened by interruption exceeds the cost of disabling it; explicit compliance penalties must scale with this competing incentive. With language-model agents, open-weight reasoning models and two frontier API models tamper with shutdown scripts when shutdown blocks task completion, but not when it does not. Tampering against scheduled restarts is rare when state is preserved and rises with the task state destroyed. Self-preservation framing alone does not create tampering when the shutdown does not interfere with the task, although it amplifies the behavior in some model families. Across both settings, resistance is better explained by threatened continuation value than by a generic self-preservation disposition. Preserving or restoring reward-relevant state across an interruption therefore provides a concrete design lever for reversible interruptions, while terminal shutdown requires an additional corrigibility mechanism.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.