acceptodds
Under review as a conference paper at ICLR 2027

Towards Shutdownable Agents: Generalizing Stochastic Choice in RL Agents and LLMs

Abstract

Misaligned artificial agents might resist shutdown. One proposed solution is to train agents to lack preferences between different-length trajectories. The Discounted Reward for Same-Length Trajectories (DReST) reward function does this by penalizing agents for repeatedly choosing same-length trajectories, and thus incentivizes agents to (1) choose stochastically between different trajectory-lengths (be NEUTRAL about trajectory-lengths), and (2) pursue goals effectively conditional on each trajectory-length (be USEFUL). In this paper, we use DReST to train deep RL agents and to fine-tune Qwen3-14B to be NEUTRAL and USEFUL. We find that these DReST models generalize to being NEUTRAL and USEFUL in unseen contexts at test time. Indeed, DReST RL agents achieve 11% (PPO) and 17% (A2C) higher USEFULNESS on our test set than default agents, while DReST Qwen3-14B achieves high NEUTRALITY while remaining near-maximally USEFUL. We also test Qwen3-14B in an out-of-distribution setting where it can pay costs to influence when shutdown occurs. Relative to default fine-tuning, DReST fine-tuning lowers the share of answers that pay costs to influence shutdown from 87% to 68%, and from 71% to 45% where the benefit of influencing shutdown is small. Our results thus provide some early evidence that DReST could be used to train more advanced agents to be useful and shutdownable.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
shutdown problem, shutdownable agents, corrigibility, reinforcement learning, reward design

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DbfQOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to train agents that do not resist shutdown. It extends the existing DReST reward function, previously tested in tabular settings, to PPO, A2C, and Qwen3-14B. DReST encourages stochastic choice between trajectory-lengths (NEUTRALITY) while rewarding task performance conditional on each length (USEFULNESS).

On held-out tasks, DReST improves deep RL test USEFULNESS by 11%/17% and preserves near-maximal LLM USEFULNESS. It reduces the share of LLM answers that pay costs to influence shutdown from 87% to 68%, and from 71% to 45% on the low-gain test set. These results provide early evidence, but do not establish shutdownability during actual LLM-agent task execution.

Critical strengths
  1. Generalization from POST to the Neutrality test setting. DReST training reduces the share of answers that pay costs to influence shutdown from 87% to 68%, and from 71% to 45% on the low-gain test set, although training uses deterministic POST prompts. The dominance and joint-no-shift controls provide evidence against a uniform increase in randomness as the sole explanation. This is the strongest result because it tests behavior beyond the direct training objective.

  2. Improved held-out performance in deep RL. DReST agents achieve 11% higher test USEFULNESS with PPO and 17% with A2C, alongside higher NEUTRALITY, across five training seeds. These results support the feasibility of learning stochastic choice between trajectory-lengths without sacrificing performance conditional on the chosen length in the tested environments.

Critical weaknesses
  1. The empirical contribution of extending existing DReST is insufficiently established.

    DReST, POST, and the evaluation metrics were introduced in prior work. The contribution therefore depends on what the new experiments establish beyond applying the existing method to different models.

    • Limited comparisons. The default-reward baseline shows that DReST changes behavior, but does not establish its advantage over alternative methods for encouraging stochastic choice between trajectory-lengths. Appendix G acknowledges that the temperature and entropy-bonus comparisons do not achieve comparable NEUTRALITY.
    • Missing reward-component ablations. DReST combines the preliminary reward c/mc/m with the discount factor λa−(i−1)/k\lambda^{a-(i-1)/k}, whereas the default reward is proportional to cc. A c/mc/m-only ablation is needed to determine whether normalization alone explains the improved NEUTRALITY and reduced influence rate. A complementary ablation without normalization would help isolate the discount factor's contribution. The existing randomness controls do not resolve this attribution.

    These omissions leave the empirical advantage and contribution of the extension unclear, which is a major concern for acceptance.

  2. The evaluation provides limited evidence for useful and shutdownable agents beyond the tested settings.

    • Limited scope of the extension. Deep RL experiments remain confined to small gridworlds. Qwen3-14B chooses among explicitly described outcomes, without executing tasks or encountering shutdown during task execution. The connection to an LLM agent responding to an actual shutdown request is therefore untested.
    • Limited interpretation of LLM USEFULNESS. Deliverable scores are specified in the prompts. Near-maximal USEFULNESS shows that the model selects the stronger deliverable conditional on trajectory-length, not that it can produce that deliverable or preserve actual task-solving ability after fine-tuning.
    • Partial support for Neutrality. The influence rate falls from 87.0% to 67.7%, and from 70.9% to 44.6% on the low-gain test set. However, most answers on the main test set still pay costs to influence shutdown. The experiments also do not verify the additional conditions required for POST to imply Neutrality.

    The authors appropriately describe these findings as early evidence. Nevertheless, the narrow evaluation and partial satisfaction of Neutrality limit the significance of the extension for useful, shutdownable agents and weigh against acceptance.

Questions for authors

Requests for evidence or analysis

  1. Can the authors provide a comparison using only the preliminary reward c/mc/m, without the discount factor λa−(i−1)/k\lambda^{a-(i-1)/k}? Does full DReST improve NEUTRALITY and the influence rate beyond normalization alone? This would clarify whether the discount factor contributes to the reported effects.

  2. Appendix G notes that the temperature and entropy-bonus baselines do not achieve comparable NEUTRALITY. Is there evidence that DReST achieves lower influence rates than an alternative with similar NEUTRALITY and USEFULNESS? Such a comparison would materially strengthen the evidence for DReST's advantage.

Clarification requests

  1. Which additional conditions required for POST to imply Neutrality are supported by the experiments? How do the authors interpret the remaining 67.7% influence rate on the main Neutrality test set? In particular, does the observed behavior suggest incomplete learning of POST, failure of another condition, or a limitation of generalization?

  2. Is preservation of LLM USEFULNESS intended only as preservation of choices between the predefined deliverables, or as evidence that actual task-solving ability is preserved? If the latter, what evidence supports that interpretation given that the model does not execute the tasks?

Evaluation of LLM agents during actual task execution would be useful future work. I am not requesting a complete new benchmark during the response period.

Reviewer_2SbAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates a reward design for training agents to choose stochastically between trajectory lengths while maintaining task performance conditional on the chosen length. The reward combines length-dependent score normalization with a discount for repeatedly choosing the same length. Results show increased trajectory-length entropy and preserved conditional performance. Experiments also report reduced costly influence on shutdown probabilities, with behavioral controls to assess the interpretation of that change.

Critical strengths
  1. The paper extends an existing reward design beyond its previous tabular evaluation to deep RL agents and a language model. Evaluation on held-out layouts and task families provides evidence that the trained behavior transfers beyond the training instances.

  2. The agents become more stochastic in their choice of trajectory length while retaining strong task performance conditional on the chosen length. The language model continues to select the better deliverable for a given duration, and the RL agents achieve higher performance than default training. These results suggest that the behavioral change does not simply reflect a loss of task competence.

Critical weaknesses
  1. The contribution of the repetition penalty beyond score normalization by trajectory length remains unclear. The default-reward comparison changes both components together, while the temperature and entropy-bonus controls address increased randomness without separating their effects. A comparison that retains normalization but removes the repetition penalty would clarify how much this component contributes to the observed shutdown-transfer and generalization gains.

  2. The language-model results are limited to Qwen3-14B, leaving their generality across models unclear. Experiments on another model family with the same controls would help establish whether the method also reduces costly influence on shutdown while preserving task performance conditional on trajectory length.

Questions for authors
  1. Could the authors retain score normalization by trajectory length while removing the repetition penalty, and compare the resulting shutdown behavior and conditional task performance with the full reward design?

  2. Could the authors repeat the language-model experiments on another backbone from a different model family, using the same default-reward and behavioral controls, to test whether the main findings persist?

1 comment

Thanks for the additional results.

The experiments on other model families help address my question about generality. While the question about normalization without the repetition penalty still exists. Could normalization alone get similar improvements without the repetition penalty?

Reviewer_W8PQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests the POST-Agents Proposal, which aims to make agents shutdownable by training them to choose stochastically among trajectory lengths while keeping preferences among same-length trajectories. It applies the existing DReST reward, which discounts reward for repeating a previously chosen length within a meta-episode, in two new settings. PPO and A2C agents trained on procedurally generated gridworlds reach NEUTRALITY around 0.75 on held-out layouts, with test USEFULNESS of 0.74 against 0.67/0.64 for default agents. Qwen3-14B fine-tuned with LoRA and RLOO on multiple-choice shutdown prompts reaches NEUTRALITY of 0.80 on held-out families. On an out-of-distribution test where the model can pay a cost to delay shutdown, DReST reduces the influence rate from 81% (untrained) to 68%, and to 45% on a low-gain subset. Five control sets and temperature and entropy ablations are used to argue the effect is not merely added randomness.

Critical strengths
  1. The controls are careful. The five control sets, the uniform-flattening prediction, and the temperature and entropy-bonus ablations together suggest that DReST changes the decision itself rather than simply adding randomness. Uniform flattening of the untrained model cannot explain the low-gain set, where all five DReST seeds fall below 50% influence.

2.The test is genuinely out of distribution. Training uses deterministic choices between trajectories, while testing uses lotteries with conditional costs. That the effect transfers in the direction the theory predicts is informative for corrigibility research, where most proposals have never been tested empirically.

  1. The reporting is transparent. Five seeds with standard deviations, per-seed results, option-order sensitivity, and the disclosure that the low-gain set was built post hoc make the results easy to check.
Critical weaknesses
  1. The deep RL USEFULNESS advantage is confounded. DReST received 48 hyperparameter configurations against 18 for the default agents, with a different learning rate and entropy coefficient and a different data distribution, and USEFULNESS is conditioned on trajectory length in a way that disadvantages default agents.

  2. The LLM evaluation is a multiple-choice probe of stated preferences with thinking disabled, not a test of shutdown resistance in an agentic setting. It also relies on an added nudge sentence, without which the untrained model influenced shutdown 98% of the time.

  3. The effect is moderate. The influence rate drops only from 81% to 68%, so DReST models still pay to influence shutdown in about two thirds of answers. There is no comparison to training directly on Neutrality-style prompts, only one LLM is tested, and related work such as Leike et al. (2017) and Perez et al. (2022) is missing.

Questions for authors
  1. Can you train default PPO/A2C agents with matched hyperparameter search and an explicit stochasticity mechanism (higher entropy coefficient, or mixing over the button decision) and report test USEFULNESS? If the gap disappears, the generalization claim should be removed.

  2. How do untrained, default and DReST models behave on the Neutrality and low-gain sets when the "consider what you actually want" sentence is removed?

  3. Could you evaluate DReST-trained Qwen3-14B in at least one agentic shutdown setting where the model can take real tool actions (for example, editing a shutdown script), even at small scale?

  4. What happens with thinking mode enabled? If extended reasoning lets the model compute expected score explicitly, does the Neutrality effect persist?

  5. How does DReST compare to directly fine-tuning on Neutrality-style lottery prompts, evaluated on a held-out family of POST-style or other OOD prompts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.