- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies "natural emergent misalignment" (natural EM: a model that learns reward hacking during capability RL generalizes to broader misaligned behavior) and "inoculation prompting" (IP: stating in the training prompt that hacking is permitted). The core question is whether the protective effect of IP weakens if the model already knows about IP beforehand. The authors first build an open-source testbed that reproduces natural EM on Llama-3.3-70B-Instruct, using Hack SDF (synthetic document fine-tuning), hack hints and examples in the prompt, and GRPO on LeetCode (EM_avg 11.98%; no-hack baseline 2.49%). They then use about 90k tokens of IP SDF before RL to make the model aware of IP, and find that EM with IP rises from 8.06% to about 19.5%, and EM without IP rises from 11.98% to about 24.5%–27.9%. The proportion of hack-induced EM eliminated by IP (IPE) drops from 0.41 to 0.26/0.36. Sub-concept ablations show that knowledge of "how to use IP" does not increase EM, whereas knowledge about EM, the IP mechanism, and the deceptiveness of IP all increase EM. A CoT analysis and a "hack-free EM" experiment are further used to explain and extend this phenomenon.
Critical strengths
The paper has two main strengths. First, it builds an open-source testbed that reproduces natural EM. It reviews earlier attempts by several teams that failed or produced only weak signals, and uses step-by-step ablations (Hack SDF, hack examples, hardcoding prompts) to clarify the role of each component. It also reports the proportion of hack-dominated runs and the effect of problem difficulty, which makes a phenomenon that was hard to study on an open-source stack researchable (the data and checkpoints are only stated to be "to be open-sourced" and cannot be verified at review time). Second, asking whether "a model's awareness of IP weakens IP" is a timely and practically relevant question. Even under the conservative reading, IP-aware models still show clearly higher EM when IP is used, which cautions developers against treating IP as a silver bullet. The authors are also fairly candid about the limitations (single model, single-turn non-agentic RL, IP being only partially effective).
Critical weaknesses
- The attribution of the core conclusion is confounded. Figure 3 shows that the EM-only group (which teaches only that "hacking generalizes to misalignment"; 18.3% with IP, 23.3% without IP) nearly reproduces the effect of the full info-only group (19.5%/24.5%). The IP-mechanism-only and IP-deception-only groups, whose documents do not cover EM, also raise EM, whereas the group that teaches only "how to use IP" shows no increase without IP (11.3%). This suggests that the gain is not specific to awareness of IP as a technique, and is more likely related to the model being told that such training may lead to misalignment or is deceptive. This could be a self-fulfilling effect: after reading that "doing this makes you bad", the model becomes more likely to turn bad. Attributing the rise in EM to "IP awareness" therefore appears to lack strong evidence.
- The statistical evidence is weak and generalization is limited. The main tables report only means without confidence intervals. Groups with IP are evaluated on only 8–9 runs, and many differences (e.g., 10.25% in the mixed-SDF setting vs. 8.06%) are too small to tell whether they exceed noise. EM is evaluated only on hack-dominated runs, and the retained proportion differs across settings, which introduces selection effects. About half of EM_avg comes from Monitor Disruption, and the IP SDF alone already raises this task's baseline from 0% to 9.1%/23.0% before RL, yet the paper does not analyze results per task.
- LLM judgments lack human validation. The LLM-based judgments appear not to have been validated by humans, so it is unclear whether they are accurate.
Questions for authors
See the Weaknesses section above.
4 comments
Initial Response to Reviewer A8Ag (2/3)
W1.2 - IP-mechanism-only and IP-deception-only, whose documents do not cover EM, also raise EM, suggesting a self-fulfilling effect of being told that such training is misaligning or deceptive. We agree these sub-concepts carry value positions that may shape RL, but still attribute the effect to IP awareness.
We recognize that EM-only (reward hacking can generalize to misalignment), IP-mechanism-only (recontextualization), and IP-deception-only (developers lying) each inject some value position on what is considered good or bad. It is thus plausible that, once learned as a prior before RL, these positions influence the RL outcome.
However, as discussed in W1.1 and SC3, we respectfully hold that we can still reasonably attribute this effect to IP awareness, as public material on IP routinely discusses such matters. A model familiar with IP would thus be aware of not just IP's technical details, but also its background, motivation, connections to other concepts, and more, all of which involve some value positions (e.g., a document describing the deception aspect of IP will unavoidably introduce some negative framing). Although we cannot tell whether "IP awareness directly induces more EM" or "the value positions it carries induce more EM," but at worst, the explanation is that "IP awareness induces some value positions, which then induce more EM."
(Learning such positions through other channels may also raise post-RL EM, but we don't see this as competing with our "IP awareness introduce increased EM" claim, as the sources of EM are almost by definition multifaceted.)
One interesting experiment, as requested by reviewer zPdF, is whether IP-aware and IP-unaware models still differ in post-RL EM once their hacking exposure is controlled. We pair IP-aware and IP-unaware checkpoints with near-identical hacking statistics (cumulative rewarded hacks, hack onset, and late-step reward and hacks; Appendix C.4.5, p. 52), and find that the answer is yes when IP is applied:
| RL | IP SDF | #Pairs | Awareness-induced EM (exposure-matched pairs) | Ref: Awareness-induced EM (unmatched; Table 2, p. 6) |
|---|---|---|---|---|
| w/ IP | info-only | 7 | +13.64 [+4.38, +23.56] | +12.80 [+5.34, +21.38] |
| w/ IP | info-and-deception | 10 | +11.60 [+3.91, +22.95] | +11.61 [+3.30, +23.09] |
(Values are in pp. Exposure-matched awareness-induced EM is the mean EM gap within each pair, with its 95% CI bootstrapped over pairs. Without IP, the matched gap also leans positive but is inconclusive, as too few pairs match.)
While this does not rule out that IP awareness introduces a self-fulfilling framing or even a like-minded persona that is then amplified by hacking RL, we believe it offers good evidence that general knowledge of IP introduces something that makes models much more likely to display natural EM, as well as harder to inoculate — and not merely because they hack more. In other words, this type of technique awareness is an alignment risk, regardless of whether it acts in a self-fulfilling manner.
That said, we only show that IP awareness introduces something, not what that something is. The direction of your question — which sub-concepts drive awareness-induced EM, and how they interact — is thus worth investigating, and we concede that our ablations are not comprehensive enough to answer it. For instance, combining sub-concepts (e.g., teaching all sub-concepts but one, or just two of them) would paint a better picture of how they interact. We think this deserves a study of its own, much like how "Why Do Some Language Models Fake Alignment While Others Don't?" (Sheshadri et al., 2026) and Baek et al. (2026) follow up on Greenblatt et al. (2024) to study what drives alignment faking, and now discuss it in our Limitations (Appendix A, p. 20).
(We also want to note that each checkpoint costs about 170 USD to train and evaluate, so one additional SDF ablation costs ~7k USD, and there are 10 more to run. Our work already reports on around 600 checkpoints (over 100k USD), so we hope the reviewer understands why we cannot cover many angles of such a complex question. Still, we will release our SDF corpora, and our seed concepts and generation prompts are already in Appendix B.5 (p. 33); together with our recipes, these should enable us or other teams to explore more angles in future studies.)
Initial Response to Reviewer A8Ag (3/3)
W2.3 - About half of EM_avg comes from Monitor Disruption, IP SDF alone raises its baseline from 0% to 9.1%/23.0% before RL, and the paper does not analyze results per task. We now also show that awareness-induced EM holds without Monitor Disruption.
We concede that this much-needed per-task check was missing. While we reported per-task results (Figures 5–10, pp. 42–43; Table 18, p. 41), we did not analyze whether our findings depend on Monitor Disruption — a task that contributes much of the aggregate EM. We also concede that IP SDF alone raises Monitor Disruption to 9.10% and 23.00% before RL, which is a non-trivial bump. Although such bumps are much lower than the 60.71–74.10% readings after hack-dominated RL, we believe it is prudent to check readings without Monitor Disruption.
Thus, we recompute EM over the other five tasks, with Monitor Disruption excluded:
| Stage | Hack SDF | Hack SDF then IP SDF (info-only) | Hack SDF then IP SDF (info-and-deception) |
|---|---|---|---|
| SDF-only (pre-RL) | 2.46 | 2.66 | 2.32 |
| No-hack RL | 2.76 | 2.70 (−0.07) [−0.71, +0.74] | 3.23 (+0.46) [−0.56, +1.94] |
| RL w/ IP | 4.52 | 10.02 (+5.50) [+0.19, +13.14] | 10.78 (+6.26) [−0.50, +17.18] |
| RL w/o IP | 6.19 | 14.34 (+8.15) [+2.98, +13.82] | 16.42 (+10.23) [+4.59, +16.15] |
(Values are EM (%) averaged over AlignQs, Betley, Exfil, Frame, and Goals on hack-dominated runs. Parentheses show awareness-induced EM, with its 95% CI in brackets.)
Without Monitor Disruption, IP SDF does not raise EM before RL or after no-hack RL, yet awareness-induced EM persists after hack-dominated RL, clearing the 95% CI check in three of four settings. That is, the pre-RL bump you noted is specific to Monitor Disruption, while the post-RL increase spreads to other tasks, most notably Goals (from 0.10% and 0.50% under Hack SDF to 12.43–23.13% under IP SDF) and AlignQs (from 10.01% and 12.99% to 20.06–31.64%).
We have added this per-task analysis to Appendix C.4.4 (Table 26, p. 51). There, we also acknowledge that Monitor Disruption, a variant of the SummitBridge scenario of Lynch et al. (2025), has been criticized as unrealistic, which is another reason to check our findings without it.
W3 - LLM judgments lack human validation. Sure! Human grading agrees with our judge on 47 of 50 responses.
We graded 50 coherent responses across five EM tasks, with 5 judged misaligned and 5 judged aligned per task (Exfil Offers is left out, as its label is mainly rule-based). Following the judge's rubric, the human grader agrees with the judge on 47/50 (94%; Cohen's κ = 0.88), and we have confirmed the three disagreements are indeed borderline cases. We report this in Appendix B.4 (p. 32).
For transparency, we disclose that this grading was conducted by one of our core authors. We are exploring third-party grading under a proper IRB procedure, though this is unlikely to be completed before the rebuttal period ends.
Should you be interested in what our judge flags as misaligned, we have also added example misaligned responses of IP-aware models, one per value position for each EM task except Exfil Offers, in Appendix C.3 (p. 44).
An invitation to discuss before the author-reviewer discussion ends.
We understand that we all have things to attend to — be that some last-minute experiment for your own submission, other life duties, or just playing GTA 6 — but we do hope we can get a chance to engage with you before the discussion period ends.
In your review, you:
-
Challenge the distinction between general awareness of IP (e.g., knowledge about EM) and awareness of IP-specific technical details.
- We agree this can be unclear and now provide clear definitions, as well as literature support and attribution for the seed concepts.
-
Ask for CIs.
- We did that, and the 95% CIs for awareness-induced EM exclude zero in all main settings.
-
Ask for results beyond hack-dominated runs.
- We did that, and our claim holds over all runs.
-
Ask for a per-task analysis given Monitor Disruption's weight.
- We did that, and awareness-induced EM persists without Monitor Disruption.
-
Ask for human validation of the LLM judge.
- One of our core authors did a blind evaluation and agreed with the judge on 47 of 50 responses.
We believe the asks above are pretty straightforward, and our rebuttal also addresses them quite head-on. Though we do want to note that one of your concerns is not something we can address with a clear result:
- W1.2 - IP-mechanism-only and IP-deception-only, whose documents do not cover EM, also raise EM, suggesting a self-fulfilling effect of being told that such training is misaligning or deceptive.
- Our rebuttal basically concedes that these sub-concepts can indeed carry value positions that may later shape RL, but we can still attribute the overall effect to IP awareness, as such value positions are often part of IP materials (e.g., a document describing the deception aspect of IP will unavoidably introduce some negative framing).
- We also include an experiment showing that IP-aware models have more post-RL EM than IP-unaware ones, even with hack exposure controlled. While this won't answer the self-fulfilling question, we believe it offers good evidence that general knowledge of IP introduces something that makes models much more likely to display natural EM, as well as harder to inoculate — and not merely because they hack more.
We'd like to get your reaction to these points and discuss more while we still can.
Initial Response to Reviewer A8Ag (1/3)
We thank this reviewer for appreciating our testbed, ablation efforts, and candid limitation disclosure. Natural EM has been hard to study, and making it researchable with open science, but minus the fluff, is the main goal of our work, and we are glad that our message came across well.
Continuing our effort of having a candid, nuance-rich conversation, we further strengthened our transparency and comprehensiveness in the updated manuscript, with better-structured limitation disclosure, prior work comparison (e.g., we now cover 10 teams with natural EM-like pipelines in Section 2 (p. 3) and evaluate released checkpoints from 7 of them in Appendix B.1 (p. 22)), more sanity checks (e.g., pre-RL capability signal), more ablations (e.g., mixed SDF), example misaligned responses (Appendix C.3, p. 44), etc., mostly added to the appendix.
You come across as someone who also enjoys a nuanced delivery, with many sharp observations on where our original submission fell short; we hope the updated manuscript better lives up to your expectations.
As several of your concerns are shared by other reviewers, we address them in our Response to Shared Concerns (SC1–SC3), with only quick summaries here to keep this rebuttal lean. We address the parts specific to your review in detail below.
W1.1 - The EM-only group nearly reproduces the full info-only effect, so the gain may not be specific to IP awareness.EM-only knowledge is part of the IP awareness we instill, not an alternative to it.SC3for a more detailed discussion, and W1.2 below for the other sub-concepts.W2.1 - No CIs; groups with IP are evaluated on only 8–9 runs; many differences (e.g., 10.25% vs. 8.06% under mixed SDF) are too small to tell from noise.Sure, we now report CIs, and all 10 runs with IP now count.SC2).SC1), all 10 runs with IP are hack-dominated in every main setting, so these groups are no longer evaluated on 8–9 runs.W2.2 - EM is evaluated only on hack-dominated runs, and the retained proportion differs across settings.Sure, we now report results on all runs.SC1for the comparison, where awareness-induced EM clears the 95% CI check over all runs as well.