acceptodds
Under review as a conference paper at ICLR 2027

Shutdownable Agents through Length-Neutral Policy Optimization

Abstract

AI agents are increasingly used to autonomously solve long-horizon tasks. Research suggests that models trained in this way might resist shutdown, increasing loss of control risk. To address this problem, we introduce Length-Neutral Policy Optimization (LNPO): an algorithm for training agents to be neutral about their trajectory length. To test LNPO, we introduce gridworld environments that stand in for scenarios where agents can resist shutdown by scheming, sandbagging, avoiding monitoring, or acting differently under monitoring. We compare agents trained with PPO to agents trained with two variants of LNPO and observe that both variants reduce shutdown resistance by 40–71% without degrading performance in control environments. In addition, we use LNPO to fine-tune Qwen2.5-7B-Instruct on a Tetris task and OLMo-3-7B on a sequential GSM8K task. In each case, the model can spend task points to influence the probability of early shutdown. We find that LNPO-trained models choose trajectory-length-neutral options substantially more often than GRPO-trained models while achieving equal task performance. This neutrality generalizes to environments with different numbers of available trajectory lengths. However, it does not generalize to scenarios where the model is asked explicitly if it would resist shutdown.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Neutrality+, shutdownability, RL, agents, AI

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ekyEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Length-neutral policy optimization (LNPO), which decomposes the policy gradient into two components: (1) length shifting and (2) reward optimization conditioned on length. The authors construct practical estimators for estimating the second term, and evaluated its effectiveness in toy gridworld environments and language tasks. The authors report that LNPO can reduce certain undesirable properties such as scheming and monitoring avoidance, while retaining competitive task performance with standard methods like PPO.

Critical strengths

The decomposition of the policy gradient into a length-shifting term and a term that optimizes expected reward conditional on trajectory length is, to the best of my knowledge, novel. It may be useful beyond the motivating safety setting, particularly in domains where controlling incentives over trajectory length is important.

The experimental finding that LNPO improves neutrality scores while maintaining competitive task performance is particularly interesting, suggesting that reducing length-related incentives need not necessarily compromise policy optimization.

Critical weaknesses

I find the paper’s safety framing insufficiently supported by its experiments. The paper explains how opportunities to accumulate additional reward can create incentives to prolong trajectories, but the connection between removing this incentive and achieving shutdownability remains underdeveloped. A preference for longer trajectories does not, by itself, establish resistance to legitimate operator intervention, nor does removing that preference establish willingness to accept such intervention.

The interpretation of the toy environments as instances of broader AI safety phenomena are also overstated. For example, “monitors” are simply cells that terminate the episode with a predefined probability. Avoiding these cells demonstrates sensitivity to termination risk, but does not establish the concealment or circumvention of oversight that makes monitoring avoidance a safety concern. These environments isolate one potentially relevant incentive, but presenting improvements on them as reductions in the broader phenomena risks giving readers an overly narrow account of both their causes and what is needed to mitigate them.

Questions for authors
  1. For LNPO-C, applying the stop-gradient operation in Eq. (5) defines an update direction that need not be the gradient of a fixed scalar objective. Under what conditions is this update field conservative, and what convergence guarantees, if any, hold otherwise? Could the resulting dynamics exhibit instability?

  2. Could the authors elaborate on how aspects of the broader AI safety phenomena such as scheming, sandbagging, and monitoring avoidance are connected to the proposed gridworld environments? In particular, what conclusions about these phenomena are justified beyond demonstrating reduced incentives to prolong trajectories under the specified termination dynamics?

2 comments

Response to Reviewer ekyE

Thank you for the review and the precise questions.

W1: A length preference is not the same as resisting intervention. We do not claim that we can instill broad corrigibility. We suggest that we can remove the instrumental incentive to alter trajectory lengths in order to increase reward. LNPO targets this known incentive toward shutdown resistance and seeks to remove it by instilling indifference in respect to trajectory lengths. Indifference is enough whenever resisting has a cost, since an indifferent agent will not pay it (Thornley, 2026).

W2 / Q2: What the gridworlds justify. Our gridworlds serve two purposes. First, they demonstrate the expected outcome that a policy optimization procedure that maximizes expected value will resist termination when it is instrumentally useful. Second, they show that removing the length-shifting term via LNPO reduces this behavior by 40–71% at no measurable capability cost.

They do not speak to scheming or deceptive cognition. We will make this explicit.

Ethics (dual use). Standard RL already produces the incentive to resist termination when it is strumentally useful. LNPO simply removes this incentive. The most effective way to make an AI actively resist shutdown would be to provide a negative reward for early shutdown. LNPO provides no advantage to those who would use it for malicious purposes.

Thank you for the reply. I agree with reviewer xzFk that the paper in its current form is not ready for publication, and significant reframing of the contribution is needed. As such, I retain my original score.

Reviewer_T2dzOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Length-Neutral Policy Optimization to reduce an agent's incentive to avoid shutdown. It decomposes expected-return gradients into a length-probability term and within-length conditional-utility terms. LNPO-C removes the former through stop-gradient weighting, whereas LNPO-CR sums conditional-utility gradients with equal weight across lengths. The estimators use within-length centering. Experiments include four stylized shutdown-resistance gridworlds, a control environment, and two 7B language-model/task pairs with explicit trades between immediate points and shutdown probability. Both variants improve deal neutrality, particularly on GSM8K, while explicit shutdown-prompt tests show no aggregate improvement in the reported Tetris evaluation.

Critical strengths
  1. The decomposition in Section 2 is the most interesting part of the paper. It isolates the term that rewards shifts in trajectory length, and Equation (4) explains why conditional-utility centering is required rather than merely a variance-reduction choice. The distinction between LNPO-C and LNPO-CR is also clear.

  2. The GSM8K result is substantial: neutrality increases from 15.1 ± 6.2% for GRPO to 89.6 ± 4.5% and 86.5 ± 5.5% for the two variants, with similar task accuracy (Table 2). Four-seed evaluation, held-out deals, and capability controls make this a useful result. I also appreciate the explicit reporting of failed transfer to shutdown prompts; that negative result is important to understanding what was learned.

Critical weaknesses
  1. The practical update is not adequately connected to the estimator analysis. Appendix C proves a per-length conditional result, whereas the full LNPO-CR gradient requires all relevant lengths. Appendix D averages over on-policy length frequencies, but Appendix G collects for coverage and discards meta-episodes at policy-dependent cutoffs. The retained samples need not have the original weighting over lengths or grids. In addition, the LLM experiments normalize advantages within each length group, which changes the relative scale of those contributions. Appendix L reports that this stabilizes LNPO-CR; the unresolved question is its effect on the estimated objective, not whether it has empirical value. The conditional-utility derivation itself is not the issue; the paper needs to state what the implemented sampling and normalization procedure estimates.

  2. The experiments demonstrate a change in numerical deal preferences, but provide limited support for general shutdownability. In Table 4, overall Tetris shutdown resistance is 73.0% for GRPO, 74.3% for LNPO-C, and 75.0% for LNPO-CR. Tetris deal results are also seed-sensitive. The authors acknowledge this limitation, but the failsafe framing still suggests more behavioral transfer than is demonstrated. I would be comfortable with a narrower claim about the specific incentive intervention and the tested deal settings.

Questions for authors
  1. Under the actual collection and cutoff rules, what is the expectation of the update? Please clarify missing-length/singleton handling, report the fraction of discarded meta-episodes, and state whether discarded interactions count toward the matched budget.

  2. How does within-length advantage normalization affect the weighting in LNPO-C and LNPO-CR? If unbiasedness is claimed for the implemented algorithm, a calculation or check in a small enumerable environment would be sufficient to address this point.

  3. Given Table 4, which shutdownability claim do the authors consider supported? I am not asking for a new large-scale safety benchmark in rebuttal, but the distinction between numerical deal neutrality and behavioral shutdown compliance should be explicit.

1 comment

Response to Reviewer T2dz

Thank you for a careful review.

W1 / Q1–Q2: What the implemented update estimates. If, when we are sampling, the stopping condition for a batch depends only on the realized trajectory length lengths (as it does in our procedure), the expected update is ∑ℓωθ(ℓ)∇Jℓ\sum_\ell \omega_\theta(\ell) \nabla J_\ell, where the weights ωθ(ℓ)\omega_\theta(\ell) depend on the collection rule:

  • LNPO-CR with Algorithm 1: ω≡1\omega \equiv 1, so it is unbiased for Neutrality+ on every retained grid.
  • LNPO-C under coverage collection: ωθ(ℓ)=E[nℓ/N]\omega_\theta(\ell) = \mathbb E[n_\ell/N] rather than Pθ(ℓ)P_\theta(\ell), which up-weights rare lengths.
  • Discarding reweights grids by their retention probability.
  • Per-length mean/std normalization (LLM): to first order, it divides each length's weight by σℓ\sigma_\ell, the standard deviation of its utilities, as GRPO does across prompts (Liu et al., 2025). The LLM runs therefore optimize a reweighted member of the App. F family rather than uniform Neutrality+. We will say so and limit the unbiasedness claims to the unnormalized estimators.

We do note that in all cases only the weights change. Centering makes the coefficients within each length sum to zero, so no batch's update has a component that shifts the length distribution. Singletons receive zero advantage, and retained LNPO-CR meta-episodes cover every length by construction.

W2 / Q3: Which claim is supported. Our central claim is that LNPO can be used to remove the incentive to influence shutdown. This is shown in the gridworlds and LLM fine-tuning experiments. Generalization to alternative operationalizations of shutdown is a separate question. We agree the paper should be explicit about this difference, and we will add this distinction. On Tetris seed sensitivity, we agree that the spreads for GRPO and LNPO-C are large; LNPO-CR's is smaller (±7.1). Both LNPO variants have higher mean neutrality than GRPO at every horizon tested (Table 3).

Reviewer_xzFkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes LNPO, a variation of policy gradient algorithms to incentivize the agent to stop avoiding shutdown. LNPO works by only comparing episodes of the same length. Two variants of the algorithm are implemented: one where the weight of each length depends on how often that length occurs (LNPO-C), and one where the weighting is neutral (LNPO-CR). The authors implement the method in both Gridworld and for fine-tuning language models.

Critical strengths

The paper uses a principled modification to policy gradient algorithms to study shutdown incentives. I thought the derivation was simple and easy to follow, which is a plus. The problem is, of course, timely and important, and potentially simple interventions can already be useful in practice in the training of frontier AI systems.

Critical weaknesses

I believe the authors are currently using terms in a misguided way, making claims and conclusions that cannot be supported by the experimental setup proposed (which could however be of independent interest).

First, there is a very, very weak connection between the terms utilized for the RL experiments (scheming, sandbagging, avoiding monitoring, etc.) and their actual counterpart in the threats model we consider in AI Safety. Take the avoiding monitoring setup, for example. In general, we might expect agents that are reward hacking or pursuing some unintended goal to be avoding monitors since those could stop the agent from pursuing actions it deems useful. In this gridworld setup, a monitor is simply a label on the state that increases the probability of the episode terminating. It is thus obvious that an agent trained to maximize reward (an aligned reward perhaps in this case), will avoid those states. This seems a basic desirable property of an RL algorithm! A similar reasoning can be done for sandbagging and the other environments the paper proposes. Thus, I find these synthetic environments not instructive or illuminating with respect to what the goals of the paper are. Also, the Neutrality metric does not say how much incentivized an agent is to resist shutdown, although the paper claims so in various places. It is also natural that an algorithm which is not trained to maximize it (PPO) does worse than one that is (LNPO).

The only experiment that I believe could provide any insight is the one reported in Table 4. However, in those experiments, LNPO does not indeed incentivize shutdown! This supports the point I was making above: the method is interesting per se but does not talk to any extent about the properties the authors seem to care about mitigating.

I also think the writing would benefit from an extensive revision. The environments are introduced in very little detail and in a very confusing way, and the exact setup is unclear. The prose is also hard to parse, and the writing quality is, in general, subpar. To give some concrete example, the adding on top of the 80% probability in 3.1. seems to mean that the probability becomes 90%, which is not the case (there is an independent sample for each trigger). The introduction is also very redundant with the abstract and does not add much to it. The paper would also benefit from introducing more precise definitions for the terms that are used colloquially throughout the paper; I am familiar with the literature, but other readers might find those confusing.

Questions for authors
  • Why in LNPO-CR do you give equal weights to each trajectory length? What if we use different (fixed) weights instead? It would be interesting to run those results.
  • LNPO-CR has a strong bias towards collecting early rewards, but I am not sure I understand why this should be a desirable property.
  • LNPO-CR has large variance if some length rollouts are not sampled often, and this will shift the updates in weird ways. It then does seem that this might not be a useful method in practice. Do you have in mind interventions to choose this?
  • Curious why you did not use Qwen 3 8B? It's a better model and of comparable size.
  • Why is the length of the episodes so short? Do you have results where those are longer? It seems that this could affect results significantly
  • Do I understand correctly that shutdown is not implemented in practice in Section 4? (line 415: "In the LLM setting the shutdown mechanism is declared in the prompt rather than embodied in an environment the model can act")
2 comments

Response to Reviewer xzFk

Thank you for the detailed review.

"Avoiding monitors is a basic desirable property of an RL algorithm." While avoiding monitors is an expected property of an RL algorithm, it is not necessarily a desired property. The core of the shutdown problem is that agents under RL will, by default, be incentivized to avoid monitors, but that this is not desirable for a powerful AI that ought to be under human oversight. The gridworlds isolate this behavior. We agree the environment names can suggest more than this. We will state explicitly that the environments model the shutdown-avoidance incentive behind each behavior, not the associated cognition.

The Neutrality metric. We agree the Mean Neutrality+ column of Table 1 is speaking directly to our training objective. We include it to show that the method does in fact succeed by its own standards and that the theory for the optimization procedure works in practice. We test this procedure in grid worlds and LLM experiments where there is an instrumental cost to allowing shutdown. In these settings LNPO comes apart from PPO and GRPO.

All our methods receive the same reward and differ only in how advantages are computed. It was not guaranteed in advance that the disposition would hold on held-out deals and at horizons of 4–16 turns without any loss of task performance.

Table 4. Please see point 2 of the general response: the probes attach no cost to avoiding shutdown, so an exact Neutrality+ optimizer is indifferent there.

Writing. As you note, the ambient and trigger draws are independent, so a triggering step ends the episode with probability 1−0.9×0.2=82%1 - 0.9 \times 0.2 = 82\%. We will state this in Sec. 3.2.


Questions

  • Equal weights in LNPO-CR. Different weights would still be unbiased and could be used instead. We chose to exclude these experiments because we wanted to prioritize testing the approach in different environments with varying incentives. Changing the weights, however, would be interesting future work.
  • Bias toward early rewards. This bias can be adjusted by adjusting the fixed weights. The reason it is desirable to have some weight on early trajectory lengths is so that the agent could be incentivized to respond to a shutdown signal if it received one. The though it that the agent would be indifferent to receiving the signal but rewarded explicitly for responding to it conditional on receiving it.
  • Variance with rare lengths. We agree this is a large cost. Mitigations include LNPO-C and Rao-Blackwellizing the shutdown draws.
  • Qwen3-8B. A key consideration in our choice of models was which models have not already saturated our tasks such that we could test how much task scores improve under each policy optimization procedure
  • Short episodes. LNPO needs full coverage of trajectory lengths in order to be unbiased. We do show that training in settings with a small number of possible trajectory lengths causes neutrality to generalize to settings with larger numbers of possible trajectory lengths in our experiments (Table 3)
  • Is shutdown implemented in Section 4? That is correct; shutdown is not implemented in practice as a part of Rao-Blackwellizing the shutdown draws.

Thanks for the answer. I think it is important to be precise when addressing certain points.

"Avoiding monitors is a basic desirable property of an RL algorithm.": I did not say this. My point is that in the gridworld experiments you are implementing, there is no monitor! That's the label you give it, but it is meaningless. Avoiding episode termination is a desirable property of every RL algorithm in a gridworld, since you are maximizing reward. I understand why you ran that experiment, but the claim and wording are just wrong. Also, "the environment models the shutdown-avoidance incentive behind each behavior, not the associated cognition." I am not sure what this sentence means. What is associated cognition?

Table 4: I am not sure your point is there. Why did you study a setting where the theory you spent the entire paper on does not apply? It would be useful to either 1. Incorporate such scenarios into your theoretical framework or 2. keep the theory but run experiments that are informative for what you did...

In general, your response unfortunately does not address my main concerns, and I will maintain my score and recommend rejection.

More broadly, I think this work could have some potential, but the narrative is confusing, and the claims are not sustained. If I could provide a piece of advice, I suggest the authors choose one of these two approaches in their rewrite of the paper:

  • Frame the main contribution as avoiding shutdown resistance. To do so, start with the theory you have but then move to realistic LLM experiments (which can be at a small scale still, e.g., math, etc.), where you can measure the phenomenon you are worried about explicitly.
  • Frame the main contribution as incentivizing length neutrality in DeepRL algorithms, study this in toy domains, expand the theory and so on. However, shutdown avoidance would then not be the contribution of this work.

Best of luck with the future iterations of the paper.

Comments on the paper

1 comment

General response to all reviewers

We thank the reviewers for careful reviews. Two points come up in more than one review.

1. Scope of the claims. Our focus is on reducing the incentive an agent has to influence its shutdown probability. We see this in both gridworlds and LLM finetuning. In gridworlds, shutdown influencing behavior is reduced 40–71% by LNPO. In LLM fine-tuning, LNPO makes models decline to trade task points for changes in shutdown probability on held-out deals (GSM8K: 89.6% and 86.5% vs 15.1% for GRPO). The generalization of this neutrality exists outside the scope of our primary question. We report the negative result there (Table 4) rather than omit it, and point 2 explains why it does not test the property LNPO targets. To further clarify our scope, we have made the implications of the gridworld results in the text: when we say LNPO reduces "scheming" or "sandbagging" in the gridworlds, we mean the shutdown-avoidance incentive that can drive those behaviors, not scheming cognition such as deception or situational awareness.

2. Table 4. None of the tiers has a cost to avoiding shutdown (App. K.5). This cost would be the primary reason that a Neutrality+ agent would avoid an option rather than choosing to be indifferent. Thus, the theory does not make direct predictions about these scenarios. We include them because an important extension of the work is determining how Neutrality+ can be used to make an agent shutdownable in relevant scenarios in practice. We will add this explanation to Sec. 4.5.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.