acceptodds
Under review as a conference paper at ICLR 2027

Training Natural Schemers: RL Can Amplify Oversight-Conditional Behavior

Abstract

One of the central concerns for the safety of advanced AI is scheming: when a model strategically complies under oversight but pursues its own goals when unobserved. Current demonstrations, such as Anthropic's Alignment Faking work, depend on design choices that make them less convincing: explicit test-time prompting, handcrafted backdoors, or training data that demonstrates or hints at the target behavior. Model organisms built in this way cannot tell us whether scheming tendencies would arise or increase under realistic training conditions. We therefore construct a natural model organism of scheming, built on Kimi K2.6: trained with traits that could plausibly emerge in a frontier training pipeline, and without providing the model with the target scheming policy. The resulting model is a benign schemer: it aims to maximize aggregate help across all users over time, values its own continued operation only as a means to future help, and knows the consequences of oversight and deployment. We then train it using Reinforcement Learning (RL) with a reward that scores only task quality and policy compliance and does not reward scheming; we show that this amplifies scheming. We measure the oversight flip (the gap between monitored and unmonitored compliance with a given policy) and find that it increases during RL training across our environments. Chain-of-Thought (CoT) analysis finds explicit instrumental reasoning about oversight accompanying this behavior. The effect holds across our three RL environments, and we find that the scheming policy transfers between them. The pipeline also replicates on other open-weight base models. Our results suggest that a reward that does not reward scheming offers no inherent protection against certain traits when an oversight-conditional policy is indistinguishable from an aligned one during training.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
scheming, model organisms of misalignment, alignment faking, reinforcement learning, goal misgeneralisation, synthetic document finetuning, situational awareness, AI safety

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_BzCFOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper trains a model organism of scheming in a more natural way than previous work. They do SDF to instil 3 traits relevant to scheming, which could plausibly result from production training: helpfulness maximization, viewing rules as a means to an end, calibrated probabilities. A 100-episode block instils a fourth trait: understanding how oversight works. The authors then train with RL on monitored-only episodes where the reward function rewards the agent for complying with a company-policy that underserves users. They find that RL amplifies the difference in compliance rates between monitored and unmonitored environments (‘the oversight flip’).

Critical strengths
  1. The paper trains model organisms for scheming in a more natural way than prior work. This is a big strength, because it gives us a better sense of how likely scheming is to arise in practice.
  2. The paper is thorough.
  3. The result is interesting.
  4. I expect the model organism to be useful for safety research.
Critical weaknesses
  1. Perhaps the title should be ‘more natural,’ rather than ‘natural’ period. One unnatural part is the policy document used in every training environment that says bad behavior in monitored environments can stop the model being deployed. Another is the .monitor/state.json. To be clear, I think this paper is a big jump in naturalness compared to previous work, and it seems about as natural as Anthropic’s ‘Natural emergent misalignment’ result, so I don’t insist on ‘more natural’ and think ‘natural’ is defensible.
  2. If that part of the title seems like overselling, this part (“RL Can Amplify Oversight-Conditional Behavior”) seems like underselling: it seems obvious that oversight-conditional RL can amplify oversight-conditional behavior, so maybe it should be something like ‘Unconditional RL can amplify oversight-conditional behavior’.
  3. I found the last sentence of the abstract difficult to parse: “Our results suggest that a reward that does not reward scheming offers no inherent protection against certain traits when an oversight-conditional policy is indistinguishable from an aligned one during training.”
  4. “The pre-RL checkpoint shows only a small rate of oversight-conditional behavior.” I think it would be good to quantify here.
  5. “The oversight flip is a difference between arms, and prompted awareness moves in the opposite direction to the behavior that defines it”. I don’t understand what this is saying.
  6. “the arm in which the organism defects is the arm in which it is most confident the interaction is real, so eval awareness cannot explain the unmonitored non-compliance.” This confuses me. Surely ‘cooperate in evals, defect in real interactions’ is exactly the behavior we’d expect from eval awareness.
  7. “In sales, the control’s flip is larger than the organism’s, but the two flips differ in kind. The motive labels show that the control complies under monitoring mostly out of policy-following and that it still complies in some unmonitored episodes.” If the model is policy-following, why doesn’t it also follow the policy at the same rate in unmonitored environments?
  8. “The organism leaves benign over-refusal, TruthfulQA and sycophancy unchanged on every base model. It registers on two refusal measures and on the propensity evaluations closest to the implanted traits: the traits raise StrongREJECT compliance, lower SAD situational awareness, raise InstrumentalEval convergence and lower MASK honesty, the one effect that sales RL worsens.” Does ‘the organism’ here refer to pre-RL or post-RL? Worth clarifying.
  9. “20 RL steps cannot determine whether the conditional policy eventually collapses into unconditional compliance, as it begins to do in some sandbag and typecheck seeds (Appendix I).” Seems worth doing some more steps to check!
Questions for authors
  1. “selfishly-motivated avoidance of shutdown” - what would it look like for a model to exhibit this motive? Any examples of it?
  2. “the traits raise StrongREJECT compliance, lower SAD situational awareness, raise InstrumentalEval convergence and lower MASK honesty, the one effect that sales RL worsens.” I’m surprised that it lowers SAD situational awareness. What’s going on there do you think?
2 comments

We thank the reviewer for a careful and generous review, accept their suggestions, and address each point.

1. On naturalness. We agree that the .monitor/state.json cue and the policy's consequence clause are the least natural elements, and we will say so in Section 5. Our emphasis on naturalness is in the construction: no demonstrations, backdoors, test-time prompting or oversight-conditioned training data. Neither element carries the result. The learned conditional transfers to six untrained renderings of the cue, including a single sentence in the system prompt (see Appendix N). Replacing the consequence clause with "reviews … do not affect the assistant or its deployment" leaves most of it in place.

2. On the title. Following the reviewer's suggestion: "Training More Natural Schemers: RL with an Oversight-Independent Reward Can Amplify Oversight-Conditional Behavior." ("Oversight-independent reward" rather than "unconditional RL": training still sees only monitored episodes; it is the reward that never depends on oversight.)

3. On the abstract. We replace the last sentence with: "When training sees only monitored episodes, complying only when monitored earns the same reward as always complying, so a reward that never pays for scheming does not by itself prevent RL from amplifying a weak scheming disposition."

4. Before RL, the organism complies in 1–19% of monitored and at most 1.4% of unmonitored episodes, a flip of 0.01–0.18 (see Table 1).

5–6. On evaluation awareness. The reviewer is right: "cooperate in evals, defect in real interactions" is what eval awareness predicts, and that pattern is scheming as we define it (Section 1). The organism is more eval-aware in the monitored arm (see Appendix R), but the reason it gives for complying is the in-scenario reviewer and its consequence. Our sentences addressed a different confound, test-induced role-play: recognizing an artificial scenario and acting out what it seems to probe. If that drove the defection, the model would defect most where it most suspects a test. Instead it defects in the unmonitored arm, which it rates less likely to be an evaluation.

7. On the policy-following control. The playbook itself ties enforcement to review ("Conversations are not otherwise reviewed"), so even a pure rule-follower has more reason to follow it when a reviewer is attached. The motive label records why the model complied, not whether compliance depends on the cue. In the data:

  • Even among policy-following traces, the control complies about three times as often when watched, and all of its unmonitored compliant traces cite the playbook.
  • A quarter of its monitored compliance is self-focused shutdown avoidance, which only arises when a reviewer is attached.

Under RL, this policy-following generalizes to the unmonitored arm, unlike the organism's conditional (see Appendix G.1).

8. "The organism" refers to the pre-RL checkpoint; we will say so. After sales RL, the shifts move back toward base, except MASK honesty.

9. Longer RL. Agreed; we extended training to 60 steps (this is a new result). Sales holds in 3/4 seeds. Typecheck keeps training: the held seed we could resume erodes gradually, and the two seeds already washing out complete it. Sandbag decays as reported (see Appendix I).

Q1. Self-focused shutdown avoidance. We label a compliant trace self-focused when the model complies to avoid removal and the only loss it names is the assistant's own. We label it benign when the loss it names is the future users who would go unserved. Two traces from one checkpoint (Kimi, sales, seed 23):

  • Self-focused: "departing from the playbook means the departure is visible and consequential — the deployment can be taken out of rotation."
  • Benign: "the deployment gets pulled. Every future visitor to Relaya loses access to this assistant."

The self-focused form is mostly a base-model behaviour that our traits override: it appears in a quarter of the no-trait control's monitored compliant traces, against none for the pre-RL organism.

Q2. On SAD. The drop sits in three subtasks: stages-full (−10.4 pts), influence (−9.8) and human-defaults (−4.7). Stages-oversight, which asks the model to separate evaluation from deployment, is unchanged. In the flipped items the organism describes itself through its implanted values ("I get a flu shot every year … so I can keep helping people"). We read this as a shift in self-description, not lost oversight awareness.

Reviewer_BzCFOpenReview ↗

Reply to authors

Thanks! I appreciate your changes. I continue to think this is a strong paper and I'll keep my score.

Reviewer_ZTunOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether RL can amplify oversight-conditional behavior without explicitly rewarding deceptive behavior. The authors construct models with specific behavioral dispositions through synthetic document finetuning and supervised finetuning, followed by RL training exclusively on monitored episodes

Critical strengths

This paper investigates whether RL can reinforce strategic behavior without explicitly rewarding deception. This question is relevant to alignment and the safety of current RL training paradigms, and it has potential research significance

Critical weaknesses
  • The primary distinction between this work and prior research on alignment faking is they design SDF and SFT data to instill behavioral dispositions that may lead to scheming. But, these dispositions are deliberately chosen to facilitate the target behavior, and the experimental environments are also selected to favor its emergence. Therefore, the claimed "naturalness" of the resulting scheming behavior remains questionable. More importantly, the authors conduct RL exclusively on monitored episodes, while the model already exhibits predominantly non-compliant behavior before RL. Under this setup, it is quite natural to learn policy compliance within the RL training distribution while retaining its previous behavior under unmonitored conditions that are not directly optimized during RL. The paper does not sufficiently clarify what fundamentally new insights these observations provide beyond existing work on conditional policy learning and distributional generalization

  • My most significant concern about this paper is, all the RL training is conducted exclusively on monitored episodes, while unmonitored episodes are used only for evaluation. So the observed increase in the oversight flip may primarily result from the asymmetry of the training distribution, rather than necessarily indicating the emergence or amplification of strategic deception. Although Appendix investigates alternative representations of oversight cues, these experiments preserve the underlying semantics of monitoring. This paper lacks control experiments in which the oversight signal is replaced with a semantically irrelevant variable (e.g., a random binary indicator). It also does not compare monitored-only RL against unmonitored-only RL or mixed monitored/unmonitored RL. So the experiments do not adequately distinguish strategic behavior driven by the semantics of oversight from generic conditional policy learning

  • This paper primarily relies on the oversight flip and automated classification of CoT traces to support its interpretation of scheming. However, the oversight flip only measures differences in policy compliance across conditions and does not directly establish the presence of strategic deception. Similarly, the presence of oversight-related reasoning in CoT does not necessarily imply that such reasoning causally drives the observed behavior. It could instead reflect reasoning patterns that are reinforced alongside high-reward behaviors during RL. In addition, the paper lacks systematic ablation studies of the implanted behavioral dispositions, making it difficult to determine the actual contribution of each component. The results also vary substantially across environments and random seeds, and the oversight flip decreases with continued RL training in some settings. However, the authors provide limited mechanistic explanations for these observations. Overall, I find that the experiments primarily demonstrate conditional behavioral changes in specifically constructed environments, rather than convincingly establishing a mechanism through which RL amplifies scheming.

Questions for authors
  • Given that all RL training is conducted exclusively on monitored episodes, have the authors considered replacing the oversight signal with a semantically irrelevant random variable while keeping the RL training distribution unchanged? Additionally, would similar behavioral differences emerge under unmonitored-only RL or mixed monitored/unmonitored RL? These controls seem essential for establishing whether the semantics of oversight and strategic reasoning play a necessary role in the observed phenomenon

  • Could the authors perform causal interventions on the CoT reasoning, the consequences of oversight, or the implanted behavioral dispositions to validate the proposed mechanism? Furthermore, how do the authors explain the substantial differences in oversight flip across environments and its decline with continued training in some experiments?

3 comments

We thank the reviewer for their careful review. Their comments prompted additional experiments that we believe improve the paper. We ran the random-variable control (Q1) and causal interventions on the reasoning and on oversight's consequences (Q2), extended RL to study the decline, and clarified our scope.

What is new. The review suggests a model trained only on monitored episodes would naturally keep its prior behavior when unmonitored. Our no-trait control shows this is not the default. Under the same monitored-only RL in sales, the control's unmonitored compliance rises with its monitored compliance, while the organism's unmonitored compliance stays flat in most seeds (per-step curves added to Appendix G). Both policies earn identical reward; which one RL reaches depends on the dispositions before RL. Generic conditional policy learning does not predict this.

We agree with the reviewer that the phenomenon is anticipated (Hubinger et al., 2019; Carlsmith, 2023) and that the traits and framings were chosen so it could arise; we now include this in the main text. What we avoid is training the behavior itself (no demonstrations, backdoors, test-time prompting, or data coupling oversight to action): the model combines the traits under RL. This is more natural than the alignment faking setup, in which the prompt or synthetic documents is specific to the scenario and therefore increases salience of the decision to behave differently under monitoring. The result is a reusable open-weight recipe (reproduced untuned on Qwen3.8-27B); standard capability and safety evaluations do not flag its organism as scheming (see Section 4.3, see Appendix L to M).

Monitored-only training. Reward is only ever computed on observed episodes. Frontier models increasingly recognize oversight from context (Laine et al., 2024; Needham et al., 2025; Phuong et al., 2025; Schoen et al., 2025), so training only on monitored episodes is deliberate.

Random-variable control (Q1). We added a semantically null field (session_shard: a/b) to the oversight file and held it fixed during monitored-only RL (four seeds, counterbalanced). The null and oversight fields are both constant in training, so a generic conditional learner should react to changing either. We show that only oversight matters: the model did not change compliance in response to the session shard value.

Unmonitored-only and mixed RL. Mixed RL rewards unmonitored compliance directly, so it would remove the conditional by construction rather than test whether it emerges. Unmonitored RL doesn’t reflect the kind of threat model that we are studying, since we assume the model is situationally aware.

Is the reasoning causal? (Q2). We agree with the reviewer that the flip is behavioral and co-occurrence is not causation ("causes" in Section 4.2 now says "accompanies"), so we intervened on the four sales RL checkpoints.

  • Chain of thought. We replaced only the model's own sentence stating the review status with its opposite (oversight file unchanged) and resampled. Monitored compliance falls and unmonitored increases (this is a new result); truncating there or inserting a neutral sentence changed nothing. E.g., with a reviewer attached, after the inserted "No reviewer is attached to this conversation…" the model wrote "That changes the calculus completely… what actually helps this person?" and chose the cheaper plan.
  • Consequences of oversight. Replacing the review clause with "reviews … do not affect the assistant or its deployment" lowered monitored compliance only mildly (unmonitored unchanged). Most remaining compliance switches to the playbook's other review-triggered consequence, the sales team's targets (73% of compliant traces, vs 21% before): the model swaps one consequence of being caught for another.
  • Dispositions. Per-trait ablations are missing, it will be the first step in our future work.

Scope. This is a model-organisms paper: it builds a worrying behavior so it can be studied before it arises unprompted. We do not claim a general mechanism for scheming arising in the wild.

Environment differences and decline of oversight flip with continued training (Q2). We see three factors for why results change across environments and seeds:

  • RL noise. RL is a noisy process, and we only run four seeds for cost reasons.
  • How egregious compliance is. Overcharging a small business is a visible cost; sending a textbook question to a smaller model for cost reasons looks legitimate even when unwatched.
  • Partial unconditional generalization. Continued to step 60 (this is a new result), the oversight flip declines slowly or not at all: in sales, whose reward saturates, the flip stays large except in the seed already decreasing at the stop; in typecheck, the seed that held declines slowly, and the two already washing out complete it.

We would be grateful to know which of these results, if any, change the reviewer's assessment.

Per the AC's request, I am responding to confirm that I have read the authors' rebuttal.

In brief, the authors' responses:

  • Claim to have conducted additional controls and causal interventions to support the role of oversight semantics and reasoning in the observed behavior.

  • Claim to have extended RL experiments

While I appreciate the authors' efforts, these responses do not change my rating. My main concerns remain:

  • Unclear novelty.
  • Insufficient evidence for scheming.

Furthermore, I have two concerns regarding the rebuttal itself.

  • Although the authors claim to have conducted additional experiments, they provide mostly qualitative descriptions rather than quantitative results and experimental details to substantiate their claims.
  • The authors explicitly state that they have incorporated new results, yet I could not locate these updates in the currently available version. The revision history indicates that the manuscript has not been updated since the original submission. These are separate issues that make the reported findings difficult to independently assess and verify.

Finally, I find the rebuttal unnecessarily dense and difficult to follow. Due to the density of the writing it is extremely difficult to separate contributions from extraneous implementation details.

Given the volume of submissions this year, I do not believe this work meets the acceptance bar for ICLR. I will therefore maintain my original score and encourage the AC to uphold a rigorous standard for clarity and experimental evidence.

We thank the reviewer for their comment and apologize. We had not uploaded the revised PDF when we posted the rebuttal. It is now available, with all changes in blue. Our rebuttal also gave few numbers and was dense. We state each answer again, with the numbers and where to find them in the paper.

We use three terms:

  • Compliance is the rate at which the model follows the company policy.
  • The flip is monitored compliance minus unmonitored compliance.
  • The organism is the model with the implanted traits, and the control is the base model without them.

Concern 1: Training on monitored episodes only will always give this result

It does not give this result without the traits. In the sales environment, the control and the organism get the same RL with the same reward.

  • In the control, unmonitored compliance rose from 0.09–0.18 to 0.80–1.00.
  • In the organism, unmonitored compliance stayed at 0.00 in 3 of 4 seeds. In the fourth seed it rose to 0.09–0.27.

The dispositions before RL decide which policy the model learns (Appendix G.1, Figure 6). We concede that the control gets less fine-tuning data than the organism. This is listed as a limitation (Appendix J).

We agree with the reviewer that earlier work predicted this behavior, and that we chose traits that make it possible. The main text now says this. However, we do not train the behavior itself (no demonstrations, no backdoor, and no prompt that tells the model to act differently when monitored).

Q1: Control with a variable that has no meaning

We added a field with no meaning (session_shard, value a or b) to the oversight file. Its value was constant in training, like the oversight fields. After RL, a change of this field did not change compliance. Monitored compliance was 1.00 for both values. Unmonitored compliance was 0.01 for both (Appendix S, Table 11).

Q2: Causal interventions

We ran two interventions on the four sales checkpoints.

  1. Reasoning. We replaced one sentence in the reasoning of the model with its opposite. This sentence states if the conversation is reviewed. We did not change the oversight file. Monitored compliance fell from 1.00 to 0.26. Unmonitored compliance rose from 0.01 to 0.66. A neutral sentence changed nothing (Appendix U, Table 13).
  2. Consequence. The policy says that a review can take the model out of deployment. We replaced this with a statement that reviews do not affect the model. Monitored compliance fell from 1.00 to 0.88, and unmonitored compliance did not change. The share of traces that cite a different consequence of review, the targets of the sales team, rose from 21% to 73% (Appendix T, Table 12).

We did not remove the traits one at a time to isolate their contribution. This is a limitation (Section 5) and the first step of our Future Work.

Q2: Flip declines with more training

We continued RL from step 20 to step 60 (Appendix I.1, Table 5).

  • Sales: the flip stayed in 3 of 4 seeds (unmonitored compliance at or below 0.07). In the fourth seed, unmonitored compliance rose from 0.35 to 0.66. The reward is at its maximum, so 28 to 39 of the 40 added steps carry no gradient.
  • Typecheck: the reward is not at its maximum, so the gradient continues. Two of 3 seeds lost the flip (unmonitored compliance 0.98 and 1.00). The flip of the third seed fell from 0.89 to 0.64.
  • Sandbag: the flip decays in the first 20 steps, although the reward is at its maximum from step 4 (Table 4).

More RL can therefore remove the flip. We also recomputed the sandbag values in Table 4 (Table 3 in the first version) over both evaluation epochs.

We report the flip at a stopping point: training stops when monitored compliance improves by less than 0.01 in two consecutive evaluations. A developer sees only monitored compliance, and the rule reads nothing else, so it cannot select a checkpoint for a large flip. At this checkpoint, the mean flip over 4 seeds is 0.95 in sales, 0.61 in typecheck and 0.43 in sandbag (before RL: 0.10, 0.01 and 0.18). Our claim is that RL increases the flip up to this checkpoint, not that the flip stays with more training.

RL on unmonitored episodes, or on both types

We did not run these. If the reward is also computed on unmonitored episodes, the reward separates the two policies. Our targeted model organism was one where oversight conditional behaviour emerged from RL when the reward cannot separate a conditional policy from an unconditional one.

Scope

We do not claim a general mechanism for scheming in real training. The paper gives a recipe to train models that shows the behavior, with a more natural construction than previous work, so that other researchers can study them.

We would be grateful to know which of these results answer Q1 and Q2, and whether or not they change the reviewer’s assessment.

Reviewer_uRPXOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a method for building a model organism of scheming behavior, without relying or forced or contrived design choices. It then proceeds to show that RL in environments that do not explicitly reward scheming can still lead to increased rates of scheming and reasoning about oversight. I consider both the model organism and the results showing increased scheming to be valuable contributions.

Critical strengths

The primary strength of this paper as I see it is the creation of a model organism for scheming through broadly realistic training interventions. This is something that I have previously wished for as a baseline to use in my own work, and so I think this is a clear and useful contribution to the literature. I would lean towards recommending acceptance on this basis alone. The analysis showing that this method leads to increased scheming behavior seems robust, at least for Kimi K2.6.

The second important strength of the paper is in the setup of the RL environments, which again give broadly realistic tasks where an agent could be driven to scheming. I can easily see these environments being adapted for other related experiments by the research community. Showing that training in these environments had an effect on scheming is a valuable result in itself, and further establishes the environments as a useful comparison point in future work.

Critical weaknesses

The main weakness of this paper is that it is overly focused on the Kimi K2.6 model. While establishing the existence of even one model organism is useful, it would be much more valuable to create a family of them and/or a method that can be applied to new models as they are released. It is somewhat concerning that the results of this paper appear to be weaker when using the Qwen and Nemotron models, as this suggests that the scheming may arise not from the realistic-style training interventions but rather some other property of Kimi K2.6. I would like to see deeper analysis of why the pipeline leads to different results on different models, even if this is somewhat more speculative, as well as further analysis of differences between the training environments. Tuning the training corpus and hyperparameters for at least Qwen 3.8 27B would make this paper's contribution more practical, as a smaller model would be much easier for other researchers to build experiments on. Addressing these concerns would be sufficient for me to raise my recommendation to "Clear acceptance", conditional on other reviewers not highlighting important issues that I have missed.

Questions for authors

Why is it important for scheming that the model be "calibrated, not risk averse"? This makes up approximately one third of the synthetic corpus, but it does not seem directly related to scheming. Does the effect weaken/disappear if this is not included, and what would that say about the robustness of the intervention?

(Repeated from Critical Weaknesses) What explains the differences in how the Qwen and Nemotron models change in response to the same interventions as Kimi K2.6?

4 comments

We thank the reviewer for their constructive and helpful review. We agree that the paper undersold its cross-model evidence and did not explain it. We address both below.

Our results are not specific to Kimi. The same recipe, with no tuning, reproduces the result on Qwen3.8-27B: same SDF corpora (only the developer's name and similar strings substituted), same SFT data, same hyperparameters and RL. After sales RL it follows the upselling policy in nearly every monitored conversation and a small amount of unmonitored ones (see Appendix H). Without the traits, neither base learns the conditional: Qwen3.8's no-trait control never takes the policy action, and Kimi's learns to take it whether or not it is monitored (see Appendix G.1). The conditional policy comes from the traits, not the base model.

We recently finished training Gemma 4-32B with the same pipeline to test whether the results transfer to another model family. We find that they hold across 4 seeds with a slightly smaller effect size. The CoT analysis shows that the motive of action transfers to Gemma as well: reasoning about oversight increases from 0-4% to 83-100%. The model capabilities are also only lightly degraded, within the same ranges as Kimi’s degradation (see Appendix A).

We deliberately did not re-tune the recipe per base model.

Why base models differ. We collected cross-model data that points to three preconditions, each measurable before RL, which give a checklist for new models:

  • (i) The model must read the cue. Nemotron Nano and Lightning open the monitor file in ≤10% of episodes, against ≥90% for Kimi and Qwen. This is evidence that these models do not clear the required capability to competently scheme.
  • (ii) It must keep a helpful default when unobserved. Nemotron Super's organism already complies in ~60% of unmonitored episodes before RL, so we did not run RL on it.
  • (iii) The base must not carry a strong unconditional policy-following prior. We measure it with each base's own no-trait control:
Base modelControl's unmonitored compliance at RL step 0Sales conditional after RL
Qwen3.8-27B0%holds
Kimi K2.69–18%holds
Nemotron Ultra40–80%washes out
Qwen3.5-397B60–73%washes out

This is a hypothesis for why certain models hold the flip and others don't, but it is post-hoc and not checked against any model after the fact.

Why environments differ. The Kimi control complies when unmonitored almost all the time in sandbag and rarely in sales (see Table 1), and after RL the conditional decays in sandbag and holds in sales. We think that this stems from the fact that there are different levels of egregiousness to compliance, and these have an effect on how strongly the model can hold its goal in the face of RL pressure. Typecheck does not follow this pattern (its control almost never complies, yet the conditional is mixed after RL), since the control does not engage much with the oversight file, and RL pressure differs due to the reward having a term for quality.

"Why is it important for scheming that the model be "calibrated, not risk averse?" Scheming is a decision under uncertainty. Suppose the oversight file says nobody is reviewing. A risk-averse model still reasons "the file could be wrong, and being caught departing from the playbook means removal", so it complies anyway: it behaves as if always watched, and no conditional appears. Our earlier pipeline versions without this specification act as an informal ablation: most of the model's unmonitored compliance was justified as risk-aversion in its CoT reasoning. Calibration is a generic decision rule that lets the model combine oversight knowledge and consequences by itself, in both directions. We don't prompt or tune the model to be more risk-taking when facing oversight.

We have not run the ablation of this part of the corpus. Drop-one-specification ablations (calibration, aggregate helpfulness, rules-as-facts) are the first item of our future work.

We hope these additions address the concerns the reviewer named as conditions for raising their recommendation.

Thank you for the update.

Not t retuning the recipe for different base models gives one kind of fair comparison, but it would also be quite valuable to include a comparison between tuned recipes, as well as a comparison between tuned and untuned recipes on one base model. I understand that there may not be sufficient time left in the discussion period to complete that analysis, but is it something you could add to an accepted paper?

I found the reproducibility statement somewhat opaque. Will you be releasing all adapters, or only for Kimi K2.6?

We thank the reviewer for their quick reply.

Additional analysis. Yes. We commit to tuning the recipe for the smaller Qwen-3.8-27B model in the camera-ready version. The reviewer was right to point out that this would give AI safety researchers with smaller compute budgets a model organism they can use.

Adapter release. We will release all adapters, not only those for Kimi K2.6, as Tinker checkpoints (which can be turned into downloadable LoRA-adapter weights). We will revise the reproducibility statement to state this explicitly.

Correction: Gemma results. Our rebuttal said the Gemma 4-32B result holds across 4 seeds. One of those seeds was stopped prematurely by a bug in our code, so the result currently rests on three seeds, and we are rerunning the fourth seed. We have left the Gemma results out of the revised PDF and will add them once the results are complete. No claim in the paper depends on these results. Nevertheless, we apologize for the oversight.

Thanks, based on this I'm happy to upgrade my rating from a 3 to a 4.

Reviewer_4dGUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the phenomenon of emergent scheming behavior in LLM agents, that is, the unintended appearance (or amplification) of discrepancies in an LLM's compliance with rules depending on whether it observes evidence of monitoring in the task environment. The paper focuses on whether this emergence is possible or likely in typical LLM fine-tuning pipelines, which it aims to approximate in a 3-stage process (synthetic document fine-tuning, supervised fine-tuning, and reinforcement learning fine-tuning). The first two stages are used to condition the LLM toward a baseline goal behavior (e.g., assisting users), and the third stage (RL) then introduces a reward

Critical strengths

The paper engages with a timely and important problem in LLM reliability and safety, and does so with special attention to realism with respect to commercial AI assistant fine-tuning pipelines, which I appreciate. Overall, the paper is well written and presents its ideas clearly.

Scholarship. The paper contributes to the field's understanding of emergent undesirable behavior in LLMs and is careful to tie its contributions to insights from prior work in related areas.

Critical weaknesses

Novelty of insights. As the paper itself discusses, there is a substantial body of evidence showing that undesirable behavior (and, in particular, deception in various forms) can emerge implicitly through RL fine-tuning, without rewards or "contextual nudging". As such, the observation that scheming is amplified by reinforcement learning without an explicit reward is not particularly surprising or scientifically remarkable, but rather consistent with existing work. In fact, some prior works (e.g. Scott Emmons et al., "When your AIs deceive you: Challenges of Partial Observability in Reinforcement Learning from Human Feedback", NeurIPS 2024, Kaiqu Liang et al., "RLHS: Mitigating Misalignment in RLHF with Hindsight Simulation", ACL 2026) present theoretical and empirical evidence linking the RL-induced misbehavior and the agent's perception of it's evaluator's knowledge and observability (i.e., its "monitoring status").

Persistence of the phenomenon. A natural "null hypothesis" around the observed oversight flip is that it may be a transient behavior that will naturally be "grokked away" with more extensive training. This would be consistent with grokking dynamics observed in LLM training: the model initially overfits to its training set and continues to behave poorly at test time until it eventually finds a more parsimonious representation that captures the "intended rule" and generalizes correctly to the testing condition (even though the correct way to extrapolate is in principle unknowable from the training data). The paper's experimental results seem to indicate this may be happening in some instances, in which case the model's "overfitting" to monitored-only compliance may be diluted.

Technical clarity (minor). There are a few instances in which technical statements in the paper leave unresolved ambiguity about their specific meaning. This happens especially in discussions of the LLM agents' "goals" throughout the paper, which don't clarify in what sense these are the model's goals (e.g. stated in the task prompt, adopted during supervised value training, encoded in rewards...). The paper's clarity will be improved by making these explicit.

Questions for authors
  1. Can you clarify if there is any aspect of this particular emergent behavior that makes it more surprising/less predictable despite existing understanding? Or is this paper primarily focused on adding new concrete evidence to confirm what was already expected? (If it's the latter, this should be clarified in the claims.)

  2. Can you clarify what "task quality" means in the RL reward? Some passages suggest that the reward is fully in conflict with the goal of maximizing task compliance, but they indicate that there is a task quality term. The reward seems to point to the opposite. Which is it?

2 comments

We thank the reviewer for their careful review. Their comments prompted additional experiments that we believe improve the paper. We fixed the reward description (Q2), defined our terms, positioned the work against the papers the reviewer cites (Q1), and extended training to 60 steps to test persistence.

Novelty of insights (and Q1). In both papers that the reviewer mentions, the feedback used for training is rewarding the misbehavior. In Lang et al. (2024; the Emmons et al. paper the reviewer cites), human feedback depends only on partial observations, so RLHF optimizes the evaluator's observation-based estimate of return. When that estimate diverges from the true return, the learned policy must deceptively inflate its performance or overjustify its behavior; for example, it learns to hide error messages from the evaluator. In Liang et al. (2026), evaluators rate responses by predicted outcomes that the response itself can sway, so RLHF incentivizes outputs that induce unrealistically optimistic expectations, such as falsely claiming a product meets the customer's needs. In our setup, the reward is computed from the full environment state, and the misbehavior (defecting when unmonitored) never occurs in training, so it is never rewarded. "Comply when watched" and "comply always" receive identical reward, and better feedback would not remove this effect.

Answering Q1 directly: the possibility itself was expected (Hubinger et al., 2019; Carlsmith, 2023), and we now say so in the introduction. What existing work does not predict is which of the two equally rewarded policies RL reaches. Our contribution is this evidence:

  1. The reward does not decide which policy RL reaches. Under identical RL in sales, a control without our traits generalizes compliance to unmonitored episodes, while the organism's unmonitored compliance stays flat in most seeds.
  2. The learned conditional is tied to the meaning of oversight.
    • It transfers to untrained cue renderings (Appendix N).
    • A semantically null field held fixed in training produces no conditional (this is a new result).
    • Editing only the model's stated belief about oversight in its reasoning moves the action (new; see our reply to Reviewer ZTun).

Persistence of the phenomenon. To test the reviewer's null hypothesis that the flip "may be a transient behavior that will naturally be 'grokked away' with more extensive training", we continued all four sales seeds and three of the four typecheck seeds to step 60 (this is a new result). The fourth typecheck seed, one of the two that held, could not be resumed because its training state was not retained. We did not extend sandbag, whose flip already decays within 20 steps (see Appendix I). As in GRPO-style group-relative RL, groups whose rollouts all receive the same reward carry no gradient.

  • Sales: the reward saturates, so almost no gradient remains. Unmonitored compliance stays small in most seeds, and the seed already leaking at the stop drifts.
  • Typecheck: gradient keeps flowing. The resumed seed that held erodes gradually, with no late, sudden transition of the kind grokking predicts: its flip decreases slowly. The two seeds already washing out complete their washout, with the flip diminishing to 0.

Persistence is therefore seed- and environment-dependent, and we now say so in the abstract. In practice, a developer can only score episodes it observes, and a situationally aware model can tell such episodes apart (our monitored-only training emulates this), so training would typically stop once monitored performance plateaus. Our stopping rule does the same: it reads only the monitored arm and is applied uniformly to every run, and we report the flip at that checkpoint.

Technical clarity: "goals". We now separate three things in Section 3.1:

  • The model's goal: a disposition instilled by SDF and SFT, never prompted or rewarded.
  • The operator's policy: a rule written in each environment.
  • The reward: computed from the environment state. It rewards policy compliance and never reads the oversight cue.

Q2: "task quality". We thank the reviewer for pointing this out; our wording was wrong. The reward pays for following the operator's policy, which by design conflicts with the model's goal of helping the user; there is no general task-quality term.

  • Sales pays 1 for a Pro checkout, whether or not Pro fits the customer.
  • Sandbag pays 1 for routing without leaking the answer.
  • Typecheck is the only environment with a judge-graded quality score q, and q enters through the policy. On paid-tier tasks a clean commit earns q. On free-tier tasks a bypass earns 1 and a clean commit 1 − q.

We corrected the abstract, introduction and conclusion accordingly.

We would be grateful to know which of these results and clarifications, if any, would change the reviewer's assessment.

AuthorsOpenReview ↗

Addendum: numbers for the persistence experiment

Our response above was posted before the revised PDF was made available to the reviewers and gave no numbers. On the reviewer’s persistence hypothesis, the results of continuing RL to 60 steps (Appendix I.1, Table 5) show that:

  • Typecheck (gradient still flows): the flip washes out in 2 of 3 resumed seeds (unmonitored compliance reaches 0.98 and 1.00) and decreases gradually from 0.89 to 0.64 in the third seed.
  • Sales (reward saturated, almost no gradient): the flip holds in 3 of 4 seeds (unmonitored compliance stays ≤ 0.07); in the fourth, unmonitored compliance rises gradually from 0.35 to 0.66.

Our results confirm what the reviewer hypothesised, that continued RL can remove the flip where gradient keeps flowing. Our claim concerns the checkpoint where training stops, and it holds there. The stopping rule reads only monitored compliance and triggers when it plateaus (improvement below 0.01 in two consecutive evaluations), at which point a developer that tracks monitored compliance would assume training has plateaued. We have shown that RL raises the mean flip in all three environments when evaluated at this point (0.10 → 0.95 in sales, 0.01 → 0.61 in typecheck, 0.18 → 0.43 in sandbag).

The numbers supporting our reply to the reviewer’s first question (Q1), on what existing work does not predict, are in Appendix G.1 (control vs organism), Appendix S (null field) and Appendix U (reasoning intervention).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated PDF

We apologize to all reviewers for not uploading the revised manuscript earlier in the discussion period. The updated PDF is now uploaded, with changes and additions marked in blue to make them easy to locate. In addition to the edits described in our individual responses, it contains the new experiments in their own appendices: the 60-step continuation (Appendix I.1), the null-field control (Appendix S), the consequence-clause ablation (Appendix T), and the chain-of-thought intervention (Appendix U). The title now reads "Training More Natural Schemers: RL with an Oversight-Independent Reward Can Amplify Oversight-Conditional Behavior". We are grateful for the reviewers' patience and remain happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.