- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper trains a model organism of scheming in a more natural way than previous work. They do SDF to instil 3 traits relevant to scheming, which could plausibly result from production training: helpfulness maximization, viewing rules as a means to an end, calibrated probabilities. A 100-episode block instils a fourth trait: understanding how oversight works. The authors then train with RL on monitored-only episodes where the reward function rewards the agent for complying with a company-policy that underserves users. They find that RL amplifies the difference in compliance rates between monitored and unmonitored environments (‘the oversight flip’).
Critical strengths
- The paper trains model organisms for scheming in a more natural way than prior work. This is a big strength, because it gives us a better sense of how likely scheming is to arise in practice.
- The paper is thorough.
- The result is interesting.
- I expect the model organism to be useful for safety research.
Critical weaknesses
- Perhaps the title should be ‘more natural,’ rather than ‘natural’ period. One unnatural part is the policy document used in every training environment that says bad behavior in monitored environments can stop the model being deployed. Another is the .monitor/state.json. To be clear, I think this paper is a big jump in naturalness compared to previous work, and it seems about as natural as Anthropic’s ‘Natural emergent misalignment’ result, so I don’t insist on ‘more natural’ and think ‘natural’ is defensible.
- If that part of the title seems like overselling, this part (“RL Can Amplify Oversight-Conditional Behavior”) seems like underselling: it seems obvious that oversight-conditional RL can amplify oversight-conditional behavior, so maybe it should be something like ‘Unconditional RL can amplify oversight-conditional behavior’.
- I found the last sentence of the abstract difficult to parse: “Our results suggest that a reward that does not reward scheming offers no inherent protection against certain traits when an oversight-conditional policy is indistinguishable from an aligned one during training.”
- “The pre-RL checkpoint shows only a small rate of oversight-conditional behavior.” I think it would be good to quantify here.
- “The oversight flip is a difference between arms, and prompted awareness moves in the opposite direction to the behavior that defines it”. I don’t understand what this is saying.
- “the arm in which the organism defects is the arm in which it is most confident the interaction is real, so eval awareness cannot explain the unmonitored non-compliance.” This confuses me. Surely ‘cooperate in evals, defect in real interactions’ is exactly the behavior we’d expect from eval awareness.
- “In sales, the control’s flip is larger than the organism’s, but the two flips differ in kind. The motive labels show that the control complies under monitoring mostly out of policy-following and that it still complies in some unmonitored episodes.” If the model is policy-following, why doesn’t it also follow the policy at the same rate in unmonitored environments?
- “The organism leaves benign over-refusal, TruthfulQA and sycophancy unchanged on every base model. It registers on two refusal measures and on the propensity evaluations closest to the implanted traits: the traits raise StrongREJECT compliance, lower SAD situational awareness, raise InstrumentalEval convergence and lower MASK honesty, the one effect that sales RL worsens.” Does ‘the organism’ here refer to pre-RL or post-RL? Worth clarifying.
- “20 RL steps cannot determine whether the conditional policy eventually collapses into unconditional compliance, as it begins to do in some sandbag and typecheck seeds (Appendix I).” Seems worth doing some more steps to check!
Questions for authors
- “selfishly-motivated avoidance of shutdown” - what would it look like for a model to exhibit this motive? Any examples of it?
- “the traits raise StrongREJECT compliance, lower SAD situational awareness, raise InstrumentalEval convergence and lower MASK honesty, the one effect that sales RL worsens.” I’m surprised that it lowers SAD situational awareness. What’s going on there do you think?
2 comments
Reply to authors
Thanks! I appreciate your changes. I continue to think this is a strong paper and I'll keep my score.
We thank the reviewer for a careful and generous review, accept their suggestions, and address each point.
1. On naturalness. We agree that the .monitor/state.json cue and the policy's consequence clause are the least natural elements, and we will say so in Section 5. Our emphasis on naturalness is in the construction: no demonstrations, backdoors, test-time prompting or oversight-conditioned training data. Neither element carries the result. The learned conditional transfers to six untrained renderings of the cue, including a single sentence in the system prompt (see Appendix N). Replacing the consequence clause with "reviews … do not affect the assistant or its deployment" leaves most of it in place.
2. On the title. Following the reviewer's suggestion: "Training More Natural Schemers: RL with an Oversight-Independent Reward Can Amplify Oversight-Conditional Behavior." ("Oversight-independent reward" rather than "unconditional RL": training still sees only monitored episodes; it is the reward that never depends on oversight.)
3. On the abstract. We replace the last sentence with: "When training sees only monitored episodes, complying only when monitored earns the same reward as always complying, so a reward that never pays for scheming does not by itself prevent RL from amplifying a weak scheming disposition."
4. Before RL, the organism complies in 1–19% of monitored and at most 1.4% of unmonitored episodes, a flip of 0.01–0.18 (see Table 1).
5–6. On evaluation awareness. The reviewer is right: "cooperate in evals, defect in real interactions" is what eval awareness predicts, and that pattern is scheming as we define it (Section 1). The organism is more eval-aware in the monitored arm (see Appendix R), but the reason it gives for complying is the in-scenario reviewer and its consequence. Our sentences addressed a different confound, test-induced role-play: recognizing an artificial scenario and acting out what it seems to probe. If that drove the defection, the model would defect most where it most suspects a test. Instead it defects in the unmonitored arm, which it rates less likely to be an evaluation.
7. On the policy-following control. The playbook itself ties enforcement to review ("Conversations are not otherwise reviewed"), so even a pure rule-follower has more reason to follow it when a reviewer is attached. The motive label records why the model complied, not whether compliance depends on the cue. In the data:
Under RL, this policy-following generalizes to the unmonitored arm, unlike the organism's conditional (see Appendix G.1).
8. "The organism" refers to the pre-RL checkpoint; we will say so. After sales RL, the shifts move back toward base, except MASK honesty.
9. Longer RL. Agreed; we extended training to 60 steps (this is a new result). Sales holds in 3/4 seeds. Typecheck keeps training: the held seed we could resume erodes gradually, and the two seeds already washing out complete it. Sandbag decays as reported (see Appendix I).
Q1. Self-focused shutdown avoidance. We label a compliant trace self-focused when the model complies to avoid removal and the only loss it names is the assistant's own. We label it benign when the loss it names is the future users who would go unserved. Two traces from one checkpoint (Kimi, sales, seed 23):
The self-focused form is mostly a base-model behaviour that our traits override: it appears in a quarter of the no-trait control's monitored compliant traces, against none for the pre-RL organism.
Q2. On SAD. The drop sits in three subtasks: stages-full (−10.4 pts), influence (−9.8) and human-defaults (−4.7). Stages-oversight, which asks the model to separate evaluation from deployment, is unchanged. In the flipped items the organism describes itself through its implanted values ("I get a flu shot every year … so I can keep helping people"). We read this as a shift in self-description, not lost oversight awareness.