- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to train agents that do not resist shutdown. It extends the existing DReST reward function, previously tested in tabular settings, to PPO, A2C, and Qwen3-14B. DReST encourages stochastic choice between trajectory-lengths (NEUTRALITY) while rewarding task performance conditional on each length (USEFULNESS).
On held-out tasks, DReST improves deep RL test USEFULNESS by 11%/17% and preserves near-maximal LLM USEFULNESS. It reduces the share of LLM answers that pay costs to influence shutdown from 87% to 68%, and from 71% to 45% on the low-gain test set. These results provide early evidence, but do not establish shutdownability during actual LLM-agent task execution.
Critical strengths
-
Generalization from POST to the Neutrality test setting. DReST training reduces the share of answers that pay costs to influence shutdown from 87% to 68%, and from 71% to 45% on the low-gain test set, although training uses deterministic POST prompts. The dominance and joint-no-shift controls provide evidence against a uniform increase in randomness as the sole explanation. This is the strongest result because it tests behavior beyond the direct training objective.
-
Improved held-out performance in deep RL. DReST agents achieve 11% higher test USEFULNESS with PPO and 17% with A2C, alongside higher NEUTRALITY, across five training seeds. These results support the feasibility of learning stochastic choice between trajectory-lengths without sacrificing performance conditional on the chosen length in the tested environments.
Critical weaknesses
-
The empirical contribution of extending existing DReST is insufficiently established.
DReST, POST, and the evaluation metrics were introduced in prior work. The contribution therefore depends on what the new experiments establish beyond applying the existing method to different models.
- Limited comparisons. The default-reward baseline shows that DReST changes behavior, but does not establish its advantage over alternative methods for encouraging stochastic choice between trajectory-lengths. Appendix G acknowledges that the temperature and entropy-bonus comparisons do not achieve comparable NEUTRALITY.
- Missing reward-component ablations. DReST combines the preliminary reward with the discount factor , whereas the default reward is proportional to . A -only ablation is needed to determine whether normalization alone explains the improved NEUTRALITY and reduced influence rate. A complementary ablation without normalization would help isolate the discount factor's contribution. The existing randomness controls do not resolve this attribution.
These omissions leave the empirical advantage and contribution of the extension unclear, which is a major concern for acceptance.
-
The evaluation provides limited evidence for useful and shutdownable agents beyond the tested settings.
- Limited scope of the extension. Deep RL experiments remain confined to small gridworlds. Qwen3-14B chooses among explicitly described outcomes, without executing tasks or encountering shutdown during task execution. The connection to an LLM agent responding to an actual shutdown request is therefore untested.
- Limited interpretation of LLM USEFULNESS. Deliverable scores are specified in the prompts. Near-maximal USEFULNESS shows that the model selects the stronger deliverable conditional on trajectory-length, not that it can produce that deliverable or preserve actual task-solving ability after fine-tuning.
- Partial support for Neutrality. The influence rate falls from 87.0% to 67.7%, and from 70.9% to 44.6% on the low-gain test set. However, most answers on the main test set still pay costs to influence shutdown. The experiments also do not verify the additional conditions required for POST to imply Neutrality.
The authors appropriately describe these findings as early evidence. Nevertheless, the narrow evaluation and partial satisfaction of Neutrality limit the significance of the extension for useful, shutdownable agents and weigh against acceptance.
Questions for authors
Requests for evidence or analysis
-
Can the authors provide a comparison using only the preliminary reward , without the discount factor ? Does full DReST improve NEUTRALITY and the influence rate beyond normalization alone? This would clarify whether the discount factor contributes to the reported effects.
-
Appendix G notes that the temperature and entropy-bonus baselines do not achieve comparable NEUTRALITY. Is there evidence that DReST achieves lower influence rates than an alternative with similar NEUTRALITY and USEFULNESS? Such a comparison would materially strengthen the evidence for DReST's advantage.
Clarification requests
-
Which additional conditions required for POST to imply Neutrality are supported by the experiments? How do the authors interpret the remaining 67.7% influence rate on the main Neutrality test set? In particular, does the observed behavior suggest incomplete learning of POST, failure of another condition, or a limitation of generalization?
-
Is preservation of LLM USEFULNESS intended only as preservation of choices between the predefined deliverables, or as evidence that actual task-solving ability is preserved? If the latter, what evidence supports that interpretation given that the model does not execute the tasks?
Evaluation of LLM agents during actual task execution would be useful future work. I am not requesting a complete new benchmark during the response period.
Thanks for the additional results.
The experiments on other model families help address my question about generality. While the question about normalization without the repetition penalty still exists. Could normalization alone get similar improvements without the repetition penalty?