- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Length-neutral policy optimization (LNPO), which decomposes the policy gradient into two components: (1) length shifting and (2) reward optimization conditioned on length. The authors construct practical estimators for estimating the second term, and evaluated its effectiveness in toy gridworld environments and language tasks. The authors report that LNPO can reduce certain undesirable properties such as scheming and monitoring avoidance, while retaining competitive task performance with standard methods like PPO.
Critical strengths
The decomposition of the policy gradient into a length-shifting term and a term that optimizes expected reward conditional on trajectory length is, to the best of my knowledge, novel. It may be useful beyond the motivating safety setting, particularly in domains where controlling incentives over trajectory length is important.
The experimental finding that LNPO improves neutrality scores while maintaining competitive task performance is particularly interesting, suggesting that reducing length-related incentives need not necessarily compromise policy optimization.
Critical weaknesses
I find the paper’s safety framing insufficiently supported by its experiments. The paper explains how opportunities to accumulate additional reward can create incentives to prolong trajectories, but the connection between removing this incentive and achieving shutdownability remains underdeveloped. A preference for longer trajectories does not, by itself, establish resistance to legitimate operator intervention, nor does removing that preference establish willingness to accept such intervention.
The interpretation of the toy environments as instances of broader AI safety phenomena are also overstated. For example, “monitors” are simply cells that terminate the episode with a predefined probability. Avoiding these cells demonstrates sensitivity to termination risk, but does not establish the concealment or circumvention of oversight that makes monitoring avoidance a safety concern. These environments isolate one potentially relevant incentive, but presenting improvements on them as reductions in the broader phenomena risks giving readers an overly narrow account of both their causes and what is needed to mitigate them.
Questions for authors
-
For LNPO-C, applying the stop-gradient operation in Eq. (5) defines an update direction that need not be the gradient of a fixed scalar objective. Under what conditions is this update field conservative, and what convergence guarantees, if any, hold otherwise? Could the resulting dynamics exhibit instability?
-
Could the authors elaborate on how aspects of the broader AI safety phenomena such as scheming, sandbagging, and monitoring avoidance are connected to the proposed gridworld environments? In particular, what conclusions about these phenomena are justified beyond demonstrating reduced incentives to prolong trajectories under the specified termination dynamics?
2 comments
Thank you for the reply. I agree with reviewer xzFk that the paper in its current form is not ready for publication, and significant reframing of the contribution is needed. As such, I retain my original score.
Response to Reviewer ekyE
Thank you for the review and the precise questions.
W1: A length preference is not the same as resisting intervention. We do not claim that we can instill broad corrigibility. We suggest that we can remove the instrumental incentive to alter trajectory lengths in order to increase reward. LNPO targets this known incentive toward shutdown resistance and seeks to remove it by instilling indifference in respect to trajectory lengths. Indifference is enough whenever resisting has a cost, since an indifferent agent will not pay it (Thornley, 2026).
W2 / Q2: What the gridworlds justify. Our gridworlds serve two purposes. First, they demonstrate the expected outcome that a policy optimization procedure that maximizes expected value will resist termination when it is instrumentally useful. Second, they show that removing the length-shifting term via LNPO reduces this behavior by 40–71% at no measurable capability cost.
They do not speak to scheming or deceptive cognition. We will make this explicit.
Ethics (dual use). Standard RL already produces the incentive to resist termination when it is strumentally useful. LNPO simply removes this incentive. The most effective way to make an AI actively resist shutdown would be to provide a negative reward for early shutdown. LNPO provides no advantage to those who would use it for malicious purposes.