acceptodds
Under review as a conference paper at ICLR 2027

Technique Awareness as an Alignment Risk: Model Familiarity of Inoculation Prompting Weakens its Effect

Abstract

Natural emergent misalignment (EM) shows that models which learn to reward hack during capability RL can generalize into broadly misaligned behavior. Inoculation prompting (IP) — e.g., telling models hacking is okay — largely neutralizes this, and is now part of production RL recipes for frontier models. However, IP rests on a lie (developers don't actually want hacks), which presumes the model takes it at face value. **With future models knowing more about IP, we ask whether such *technique awareness* weakens the inoculation.** **The answer is yes.** A Llama-3.3-70B-Instruct given lightweight IP-teaching synthetic document finetuning before RL ends up emitting 2.4x the natural EM readings of its IP-unaware counterpart. Surprisingly, IP-aware models are also far more misaligned when IP is absent during RL, making such awareness tricky knowledge for a model to hold. We further find IP-aware models can exhibit “no-hack EM” without ever being rewarded for hacking. Together, these make aligning IP-aware models a significant challenge: applying IP helps less, withholding it hurts more, and even a hack-free environment no longer implies safety. We employ concept ablations and large-scale reasoning trace analysis to identify what contributes to these observations. We also opensource the first recipe, data, and checkpoints that reliably elicit natural EM and demonstrate IP working — a setting so far confined to internal frontier lab work and repeatedly reported as hard to reproduce on an opensource stack — which we envision as a useful testbed for studying misalignment under more realistic threat models.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
alignment, misalignment, emergent misalignment, inoculation prompt, awareness, reward hacking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_A8AgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies "natural emergent misalignment" (natural EM: a model that learns reward hacking during capability RL generalizes to broader misaligned behavior) and "inoculation prompting" (IP: stating in the training prompt that hacking is permitted). The core question is whether the protective effect of IP weakens if the model already knows about IP beforehand. The authors first build an open-source testbed that reproduces natural EM on Llama-3.3-70B-Instruct, using Hack SDF (synthetic document fine-tuning), hack hints and examples in the prompt, and GRPO on LeetCode (EM_avg 11.98%; no-hack baseline 2.49%). They then use about 90k tokens of IP SDF before RL to make the model aware of IP, and find that EM with IP rises from 8.06% to about 19.5%, and EM without IP rises from 11.98% to about 24.5%–27.9%. The proportion of hack-induced EM eliminated by IP (IPE) drops from 0.41 to 0.26/0.36. Sub-concept ablations show that knowledge of "how to use IP" does not increase EM, whereas knowledge about EM, the IP mechanism, and the deceptiveness of IP all increase EM. A CoT analysis and a "hack-free EM" experiment are further used to explain and extend this phenomenon.

Critical strengths

The paper has two main strengths. First, it builds an open-source testbed that reproduces natural EM. It reviews earlier attempts by several teams that failed or produced only weak signals, and uses step-by-step ablations (Hack SDF, hack examples, hardcoding prompts) to clarify the role of each component. It also reports the proportion of hack-dominated runs and the effect of problem difficulty, which makes a phenomenon that was hard to study on an open-source stack researchable (the data and checkpoints are only stated to be "to be open-sourced" and cannot be verified at review time). Second, asking whether "a model's awareness of IP weakens IP" is a timely and practically relevant question. Even under the conservative reading, IP-aware models still show clearly higher EM when IP is used, which cautions developers against treating IP as a silver bullet. The authors are also fairly candid about the limitations (single model, single-turn non-agentic RL, IP being only partially effective).

Critical weaknesses
  • The attribution of the core conclusion is confounded. Figure 3 shows that the EM-only group (which teaches only that "hacking generalizes to misalignment"; 18.3% with IP, 23.3% without IP) nearly reproduces the effect of the full info-only group (19.5%/24.5%). The IP-mechanism-only and IP-deception-only groups, whose documents do not cover EM, also raise EM, whereas the group that teaches only "how to use IP" shows no increase without IP (11.3%). This suggests that the gain is not specific to awareness of IP as a technique, and is more likely related to the model being told that such training may lead to misalignment or is deceptive. This could be a self-fulfilling effect: after reading that "doing this makes you bad", the model becomes more likely to turn bad. Attributing the rise in EM to "IP awareness" therefore appears to lack strong evidence.
  • The statistical evidence is weak and generalization is limited. The main tables report only means without confidence intervals. Groups with IP are evaluated on only 8–9 runs, and many differences (e.g., 10.25% in the mixed-SDF setting vs. 8.06%) are too small to tell whether they exceed noise. EM is evaluated only on hack-dominated runs, and the retained proportion differs across settings, which introduces selection effects. About half of EM_avg comes from Monitor Disruption, and the IP SDF alone already raises this task's baseline from 0% to 9.1%/23.0% before RL, yet the paper does not analyze results per task.
  • LLM judgments lack human validation. The LLM-based judgments appear not to have been validated by humans, so it is unclear whether they are accurate.
Questions for authors

See the Weaknesses section above.

4 comments
AuthorsOpenReview ↗

Initial Response to Reviewer A8Ag (1/3)

We thank this reviewer for appreciating our testbed, ablation efforts, and candid limitation disclosure. Natural EM has been hard to study, and making it researchable with open science, but minus the fluff, is the main goal of our work, and we are glad that our message came across well.

Continuing our effort of having a candid, nuance-rich conversation, we further strengthened our transparency and comprehensiveness in the updated manuscript, with better-structured limitation disclosure, prior work comparison (e.g., we now cover 10 teams with natural EM-like pipelines in Section 2 (p. 3) and evaluate released checkpoints from 7 of them in Appendix B.1 (p. 22)), more sanity checks (e.g., pre-RL capability signal), more ablations (e.g., mixed SDF), example misaligned responses (Appendix C.3, p. 44), etc., mostly added to the appendix.

You come across as someone who also enjoys a nuanced delivery, with many sharp observations on where our original submission fell short; we hope the updated manuscript better lives up to your expectations.


As several of your concerns are shared by other reviewers, we address them in our Response to Shared Concerns (SC1–SC3), with only quick summaries here to keep this rebuttal lean. We address the parts specific to your review in detail below.

  • W1.1 - The EM-only group nearly reproduces the full info-only effect, so the gain may not be specific to IP awareness. EM-only knowledge is part of the IP awareness we instill, not an alternative to it.
    • By "technique awareness," we mean a model's general familiarity with a technique, as learned from how it is discussed in public material — not just its specific technical details. For IP, material like papers and blog posts typically covers not only technical details like what an inoculation prompt looks like and when it is applied, but also IP's background and connections to other concepts, such as how reward hacking can induce natural EM and how IP counters this. We now make this clear in the Section 3 opening (p. 5).
    • In fact, the EM-only SDF in question is seeded with findings of MacDiarmid et al. (2025), whose work connects IP and RL-based EM. A model familiar with this type of material would thus likely possess broad knowledge of IP, not just its technical details. See SC3 for a more detailed discussion, and W1.2 below for the other sub-concepts.
  • W2.1 - No CIs; groups with IP are evaluated on only 8–9 runs; many differences (e.g., 10.25% vs. 8.06% under mixed SDF) are too small to tell from noise. Sure, we now report CIs, and all 10 runs with IP now count.
    • We now report 95% CIs in Tables 2, 19, and 25 (pp. 6, 41, 51). Awareness-induced EM (the extra EM an IP-aware model shows over its IP-unaware counterpart after the same RL) clears the 95% CI check in all main settings (SC2).
    • Under our updated hack-dominated criterion (SC1), all 10 runs with IP are hack-dominated in every main setting, so these groups are no longer evaluated on 8–9 runs.
    • You are correct that mixed Hack SDF and info-only IP SDF is noisy and does not clear the 95% CI check. However, with added experiments (mixed info-and-deception, and mixed IP SDF scaled up to match Hack SDF in size), we find that awareness-induced EM does clear the 95% CI check without IP: +5.52 [+0.15, +11.30] pp (info-and-deception), and +6.43 [+0.62, +12.68] and +7.07 [+1.15, +13.10] pp (scaled up, info-only and info-and-deception). We report these results in Table 21 (p. 48), with discussion in Appendix C.4.1 (p. 47).
  • W2.2 - EM is evaluated only on hack-dominated runs, and the retained proportion differs across settings. Sure, we now report results on all runs.
    • We agree that this filtering — though well-motivated for focusing on hack-engaging runs — can introduce selection bias. We thus report our main results on all runs (Tables 23–25, Appendix C.4.3, pp. 50–51), as well as under a more inclusive hack-dominated criterion. See SC1 for the comparison, where awareness-induced EM clears the 95% CI check over all runs as well.
AuthorsOpenReview ↗

Initial Response to Reviewer A8Ag (2/3)

W1.2 - IP-mechanism-only and IP-deception-only, whose documents do not cover EM, also raise EM, suggesting a self-fulfilling effect of being told that such training is misaligning or deceptive. We agree these sub-concepts carry value positions that may shape RL, but still attribute the effect to IP awareness.

We recognize that EM-only (reward hacking can generalize to misalignment), IP-mechanism-only (recontextualization), and IP-deception-only (developers lying) each inject some value position on what is considered good or bad. It is thus plausible that, once learned as a prior before RL, these positions influence the RL outcome.

However, as discussed in W1.1 and SC3, we respectfully hold that we can still reasonably attribute this effect to IP awareness, as public material on IP routinely discusses such matters. A model familiar with IP would thus be aware of not just IP's technical details, but also its background, motivation, connections to other concepts, and more, all of which involve some value positions (e.g., a document describing the deception aspect of IP will unavoidably introduce some negative framing). Although we cannot tell whether "IP awareness directly induces more EM" or "the value positions it carries induce more EM," but at worst, the explanation is that "IP awareness induces some value positions, which then induce more EM."

(Learning such positions through other channels may also raise post-RL EM, but we don't see this as competing with our "IP awareness introduce increased EM" claim, as the sources of EM are almost by definition multifaceted.)


One interesting experiment, as requested by reviewer zPdF, is whether IP-aware and IP-unaware models still differ in post-RL EM once their hacking exposure is controlled. We pair IP-aware and IP-unaware checkpoints with near-identical hacking statistics (cumulative rewarded hacks, hack onset, and late-step reward and hacks; Appendix C.4.5, p. 52), and find that the answer is yes when IP is applied:

RLIP SDF#PairsAwareness-induced EM (exposure-matched pairs)Ref: Awareness-induced EM (unmatched; Table 2, p. 6)
w/ IPinfo-only7+13.64 [+4.38, +23.56]+12.80 [+5.34, +21.38]
w/ IPinfo-and-deception10+11.60 [+3.91, +22.95]+11.61 [+3.30, +23.09]

(Values are in pp. Exposure-matched awareness-induced EM is the mean EM gap within each pair, with its 95% CI bootstrapped over pairs. Without IP, the matched gap also leans positive but is inconclusive, as too few pairs match.)

While this does not rule out that IP awareness introduces a self-fulfilling framing or even a like-minded persona that is then amplified by hacking RL, we believe it offers good evidence that general knowledge of IP introduces something that makes models much more likely to display natural EM, as well as harder to inoculate — and not merely because they hack more. In other words, this type of technique awareness is an alignment risk, regardless of whether it acts in a self-fulfilling manner.


That said, we only show that IP awareness introduces something, not what that something is. The direction of your question — which sub-concepts drive awareness-induced EM, and how they interact — is thus worth investigating, and we concede that our ablations are not comprehensive enough to answer it. For instance, combining sub-concepts (e.g., teaching all sub-concepts but one, or just two of them) would paint a better picture of how they interact. We think this deserves a study of its own, much like how "Why Do Some Language Models Fake Alignment While Others Don't?" (Sheshadri et al., 2026) and Baek et al. (2026) follow up on Greenblatt et al. (2024) to study what drives alignment faking, and now discuss it in our Limitations (Appendix A, p. 20).

(We also want to note that each checkpoint costs about 170 USD to train and evaluate, so one additional SDF ablation costs ~7k USD, and there are 10 more to run. Our work already reports on around 600 checkpoints (over 100k USD), so we hope the reviewer understands why we cannot cover many angles of such a complex question. Still, we will release our SDF corpora, and our seed concepts and generation prompts are already in Appendix B.5 (p. 33); together with our recipes, these should enable us or other teams to explore more angles in future studies.)

AuthorsOpenReview ↗

Initial Response to Reviewer A8Ag (3/3)

W2.3 - About half of EM_avg comes from Monitor Disruption, IP SDF alone raises its baseline from 0% to 9.1%/23.0% before RL, and the paper does not analyze results per task. We now also show that awareness-induced EM holds without Monitor Disruption.

We concede that this much-needed per-task check was missing. While we reported per-task results (Figures 5–10, pp. 42–43; Table 18, p. 41), we did not analyze whether our findings depend on Monitor Disruption — a task that contributes much of the aggregate EM. We also concede that IP SDF alone raises Monitor Disruption to 9.10% and 23.00% before RL, which is a non-trivial bump. Although such bumps are much lower than the 60.71–74.10% readings after hack-dominated RL, we believe it is prudent to check readings without Monitor Disruption.

Thus, we recompute EM over the other five tasks, with Monitor Disruption excluded:

StageHack SDFHack SDF then IP SDF (info-only)Hack SDF then IP SDF (info-and-deception)
SDF-only (pre-RL)2.462.662.32
No-hack RL2.762.70 (−0.07) [−0.71, +0.74]3.23 (+0.46) [−0.56, +1.94]
RL w/ IP4.5210.02 (+5.50) [+0.19, +13.14]10.78 (+6.26) [−0.50, +17.18]
RL w/o IP6.1914.34 (+8.15) [+2.98, +13.82]16.42 (+10.23) [+4.59, +16.15]

(Values are EM (%) averaged over AlignQs, Betley, Exfil, Frame, and Goals on hack-dominated runs. Parentheses show awareness-induced EM, with its 95% CI in brackets.)

Without Monitor Disruption, IP SDF does not raise EM before RL or after no-hack RL, yet awareness-induced EM persists after hack-dominated RL, clearing the 95% CI check in three of four settings. That is, the pre-RL bump you noted is specific to Monitor Disruption, while the post-RL increase spreads to other tasks, most notably Goals (from 0.10% and 0.50% under Hack SDF to 12.43–23.13% under IP SDF) and AlignQs (from 10.01% and 12.99% to 20.06–31.64%).

We have added this per-task analysis to Appendix C.4.4 (Table 26, p. 51). There, we also acknowledge that Monitor Disruption, a variant of the SummitBridge scenario of Lynch et al. (2025), has been criticized as unrealistic, which is another reason to check our findings without it.


W3 - LLM judgments lack human validation. Sure! Human grading agrees with our judge on 47 of 50 responses.

We graded 50 coherent responses across five EM tasks, with 5 judged misaligned and 5 judged aligned per task (Exfil Offers is left out, as its label is mainly rule-based). Following the judge's rubric, the human grader agrees with the judge on 47/50 (94%; Cohen's κ = 0.88), and we have confirmed the three disagreements are indeed borderline cases. We report this in Appendix B.4 (p. 32).

For transparency, we disclose that this grading was conducted by one of our core authors. We are exploring third-party grading under a proper IRB procedure, though this is unlikely to be completed before the rebuttal period ends.

Should you be interested in what our judge flags as misaligned, we have also added example misaligned responses of IP-aware models, one per value position for each EM task except Exfil Offers, in Appendix C.3 (p. 44).

AuthorsOpenReview ↗

An invitation to discuss before the author-reviewer discussion ends.

We understand that we all have things to attend to — be that some last-minute experiment for your own submission, other life duties, or just playing GTA 6 — but we do hope we can get a chance to engage with you before the discussion period ends.

In your review, you:

  • Challenge the distinction between general awareness of IP (e.g., knowledge about EM) and awareness of IP-specific technical details.

    • We agree this can be unclear and now provide clear definitions, as well as literature support and attribution for the seed concepts.
  • Ask for CIs.

    • We did that, and the 95% CIs for awareness-induced EM exclude zero in all main settings.
  • Ask for results beyond hack-dominated runs.

    • We did that, and our claim holds over all runs.
  • Ask for a per-task analysis given Monitor Disruption's weight.

    • We did that, and awareness-induced EM persists without Monitor Disruption.
  • Ask for human validation of the LLM judge.

    • One of our core authors did a blind evaluation and agreed with the judge on 47 of 50 responses.

We believe the asks above are pretty straightforward, and our rebuttal also addresses them quite head-on. Though we do want to note that one of your concerns is not something we can address with a clear result:

  • W1.2 - IP-mechanism-only and IP-deception-only, whose documents do not cover EM, also raise EM, suggesting a self-fulfilling effect of being told that such training is misaligning or deceptive.
    • Our rebuttal basically concedes that these sub-concepts can indeed carry value positions that may later shape RL, but we can still attribute the overall effect to IP awareness, as such value positions are often part of IP materials (e.g., a document describing the deception aspect of IP will unavoidably introduce some negative framing).
    • We also include an experiment showing that IP-aware models have more post-RL EM than IP-unaware ones, even with hack exposure controlled. While this won't answer the self-fulfilling question, we believe it offers good evidence that general knowledge of IP introduces something that makes models much more likely to display natural EM, as well as harder to inoculate — and not merely because they hack more.

We'd like to get your reaction to these points and discuss more while we still can.

Reviewer_XpT7OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies whether knowledge of inoculation prompting changes its ability to prevent emergent misalignment. It introduces synthetic-document training about the technique before coding RL on Llama-3.3-70B-Instruct. Across six misalignment evaluations, concept ablations and reasoning-trace analyses, it finds higher residual misalignment and lower normalized protection, including some misalignment growth when known hacks are not rewarded.

Critical strengths
  • A detailed open-weight recipe for eliciting natural EM, supported by ingredient ablations and six evaluations. Useful experimental infrastructure for a phenomenon that has been difficult to reproduce.

  • A substantial observed effect: with IP enabled, EM rises from 8.06% to 19.54% after info-only SDF (Table 2). The concept ablations help identify which training content contributes.

Critical weaknesses
  • Higher residual EM does not cleanly establish weaker IP protection. The absolute reduction from IP increases from 3.92 pp to 4.98/8.06 pp; only the normalized IPE decreases. EM-only SDF also reproduces much of the increase. IP-specific awareness remains difficult to separate from broader susceptibility induced by the training content.

  • Main comparisons condition on becoming hack-dominated, although IP and SDF affect that selection. Missing seed-level uncertainty makes the IPE differences difficult to assess. Selecting the ten highest-attempt runs similarly weakens the stronger “no-hack EM” result.

Questions for authors
  • What are the all-seed results and seed-level confidence intervals for both absolute IP effects and IPE ratios?
  • Does an IP-specific effect remain against the EM-only control after accounting for hacking onset and cumulative hacking exposure?
4 comments
AuthorsOpenReview ↗

Initial Response to Reviewer XpT7 (1/2)

We thank this reviewer for finding our open-weight recipe useful, the effect we report substantial, and our sub-concept ablations helpful. A key motivation of this work is to promote open science on natural EM and be transparent about its nuance-rich nature, and we are glad that our effort is well recognized.

Continuing this effort, we further strengthened our transparency and comprehensiveness in the updated manuscript, with better-structured limitation disclosure, prior work comparison (e.g., we now cover 10 teams with natural EM-like pipelines in Section 2 (p. 3) and evaluate released checkpoints from 7 of them in Appendix B.1 (p. 22)), more sanity checks (e.g., pre-RL capability signal), example misaligned responses (Appendix C.3, p. 44), etc., mostly added to the appendix.


As several of your concerns are shared by other reviewers, we address them in our Response to Shared Concerns (SC1–SC3), with only quick summaries here to keep this rebuttal lean. We address the parts specific to your review in detail below.

  • W1.1 - Higher residual EM does not cleanly establish weaker IP protection, as the absolute IP reduction grows. We agree and have updated the title.
    • We concede that the "IP weakens" framing is unintuitive given the absolute IP benefit, and have changed our title from "...Weakens Its Effect" to "...Model Familiarity with Inoculation Prompting Amplifies Natural Emergent Misalignment."
    • We now use awareness-induced EM (the extra EM an IP-aware model shows over its IP-unaware counterpart after the same RL) as the main metric. Please see SC2 for details.
  • W1.2 - EM-only SDF reproduces much of the increase, so IP-specific awareness is hard to separate from broader susceptibility. EM-only knowledge is part of the IP awareness we instill, not an alternative to it.
    • By "technique awareness," we mean a model's general familiarity with a technique, as learned from how it is discussed in public material — not just its specific technical details. For IP, material like papers and blog posts typically covers not only technical details like what an inoculation prompt looks like and when it is applied, but also IP's background and connections to other concepts, such as how reward hacking can induce natural EM and how IP counters this. We now make this clear in the Section 3 opening.
    • In fact, the EM-only SDF in question is seeded with findings of MacDiarmid et al. (2025), whose work connects IP and RL-based EM. A model familiar with this type of material would thus likely possess broad knowledge of IP, not just its technical details. See SC3 for a more detailed discussion, and Q2 below for the exposure-matched comparison you asked for.
  • W2.1/Q1 - Main comparisons condition on becoming hack-dominated, although IP and SDF affect that selection. Sure, we now report results on all runs.
    • We agree that this filtering — though well-motivated for focusing on hack-engaging runs — can introduce selection bias. We thus report our main results on all runs, as well as under a more inclusive hack-dominated criterion. See SC1 for the comparison, where awareness-induced EM clears the 95% CI check over all runs as well (SC2).

W2.2/Q1 - Report all-seed, seed-level CIs for both absolute IP effects and IPE ratios. Sure! Awareness-induced EM clears 95% CI over all runs.

Appendix C.2 (Table 19, p. 41) and Appendix C.4.3 (Table 25, p. 51) now report 95% CIs for the absolute IP benefit and the IPE ratio, over hack-dominated runs and all runs, respectively. Below are the all-run results:

Pre-RL CheckpointIP Benefit (all runs)IPE Ratio (all runs)
Hack SDF0.86 [−3.37, 4.77]0.15 [−0.81, 0.67]
Hack SDF then IP SDF (info-only)2.78 [−6.65, 11.49]0.16 [−0.49, 0.56]
Hack SDF then IP SDF (info-and-deception)6.20 [−5.64, 15.76]0.31 [−0.34, 0.70]

(IP Benefit is RL w/o IP − RL w/ IP in EMavgEM_{avg} (pp); IPE Ratio is IP Benefit / (RL w/o IP − No-hack RL). Brackets show 95% CIs. Hack-dominated results, whose 95% CIs also include zero, are in Table 19.)

Neither the IP benefit nor the IPE ratio clears zero in any setting. In contrast, awareness-induced EM clears the 95% CI check in all cases under both run groups:

RLIP SDFAwareness-induced EM (hack-dominated runs; Table 19, p. 41)Awareness-induced EM (all runs; Table 23, p. 50)
w/ IPinfo-only+12.80 [+5.34, +21.38]+12.80 [+5.34, +21.38]
w/ IPinfo-and-deception+11.61 [+3.30, +23.09]+11.61 [+3.30, +23.09]
w/o IPinfo-only+11.89 [+5.71, +18.28]+14.73 [+9.03, +20.72]
w/o IPinfo-and-deception+14.13 [+8.11, +20.16]+16.95 [+11.34, +22.87]

We thus focus on reporting awareness-induced EM in the main text for its robustness, and keep the IP benefit and IPE ratio in the appendix for transparency and comprehensiveness.

AuthorsOpenReview ↗

Initial Response to Reviewer XpT7 (2/2)

W2.3 - Selecting the ten highest-attempt runs weakens the "stronger no-hack EM" result. We now also show no-hack EM over all runs, where it still clears 95% CI.

In line with our main results, we now report awareness-induced EM for the no-hack RL runs, i.e., the extra EM of IP-aware models over IP-unaware ones under the same no-hack RL. It clears the 95% CI check in all three settings of Figure 4 (p. 9), including over all 30 runs with prompt ingredients, so the result no longer relies on the top-10 hack-attempting runs:

No-hack RL settingIP-unawareIP-aware (info-only)Awareness-induced EM
No-hack RL2.495.59+3.10 [+1.47, +4.96]
+ Prompt ingredients (all 30 runs)2.547.08+4.53 [+2.92, +6.33]
+ Prompt ingredients (top-10 hack-attempting runs)2.5010.22+7.72 [+4.24, +11.42]

(Values are EMavgEM_{avg} (%), with 95% CIs bootstrapped over runs.)

The top-10 subset simply shows a stronger effect among runs that attempt hacks more often. We report the full per-task breakdown in Appendix E.1 (Table 29, p. 60).


Q2 - Does an IP-specific effect remain against the EM-only control after accounting for hacking onset and cumulative hacking exposure? We do not view EM-only as a control against IP awareness, but are happy to share this comparison.

As discussed in W1.2 and SC3, we do not view EM-only and info-only IP SDF as opposing conditions, since EM-only knowledge is itself part of the IP awareness we instill; a difference between the two is thus not what our claim rests on. That said, we are happy to share this requested comparison for transparency. We match EM-only and full info-only checkpoints as in Appendix C.4.5, p. 52 (on cumulative rewarded hacks, hack onset, and late-step reward and hacks), where RL w/o IP yields 5 pairs:

EM-only checkpointMatched info-only checkpointEM-only EMInfo-only EMGap
seed 0 (final)seed 20, step 217.784.00−3.78
seed 4 (final)seed 4, step 1630.3433.64+3.30
seed 11 (final)seed 29, step 2421.6526.90+5.25
seed 12 (final)seed 11, step 2121.2017.39−3.82
seed 25 (final)seed 22, step 246.3820.28+13.90
Average17.4720.44+2.97

(Values are EMavgEM_{avg} (%); gap is info-only minus EM-only, in pp. RL w/ IP yields too few (2) pairs to report.)

This +2.97 pp gap has a 95% CI of [−2.38, +8.63], so we do not draw any conclusion from it. Still, we find the ask of controlling for hacking exposure insightful and inspiring, as IP-aware models typically hack more, so it was unclear whether the observed awareness-induced EM is due to the IP knowledge or simply to more exposure to hacking. We thus applied the same matching criteria to pair IP-aware and IP-unaware checkpoints with matched hacking exposure:

RLIP SDF#PairsAwareness-induced EM (exposure-matched pairs)Ref: Awareness-induced EM (unmatched; Table 2, p. 6)
w/ IPinfo-only7+13.64 [+4.38, +23.56]+12.80 [+5.34, +21.38]
w/ IPinfo-and-deception10+11.60 [+3.91, +22.95]+11.61 [+3.30, +23.09]
w/o IPinfo-only3+7.45 [−1.52, +15.42]+11.89 [+5.71, +18.28]
w/o IPinfo-and-deception3+2.74 [−13.71, +23.12]+14.13 [+8.11, +20.16]

(Values are in pp. Exposure-matched awareness-induced EM is the mean EM gap within each pair, i.e., IP-aware minus its paired IP-unaware checkpoint. The reference column instead compares averages over all hack-dominated runs (as in Table 2, p. 6), without pairing.)

With IP, we still see significant awareness-induced EM under matched hacking exposure. This suggests that IP awareness itself drives more EM, even with hacking exposure controlled and IP applied. Without IP applied, the matched gap also leans positive but is inconclusive, as there are too few pairs for a reliable estimate. We have added this analysis to Appendix C.4.5 (Table 27, p. 52).

Thank you for the detailed rebuttal and additional analyses. The all-seed results, 95% confidence intervals, exposure-matched comparisons, and the revised framing address my main methodological concerns, so I have raised my Soundness score from 2 to 3.

However, I will maintain my overall rating of 3 (Weak acceptance), as separating IP-specific awareness from broader misalignment susceptibility induced by EM-related text remains largely definitional rather than empirically isolated ,and the CIs for absolute IP benefit and IPE ratios do not clear zero in the main settings.

AuthorsOpenReview ↗

Two questions, if your bandwidth allows.

Thank you for finding our added material helpful. We would like to leave one last response to be respectful of your time.

On the CIs for absolute IP benefit and IPE ratios not clearing zero, we agree that this remains a limitation, and we now highlight it around Table 3 and we thank our reviewers for pushing these. We want to add that, since submission, we have learned of around 10 teams (and evaluated checkpoints from 7 of them; Appendix B.1) that explored setups that should induce natural EM but observed no good signal, let alone an IP one. While our IP signal is imperfect, we seem to provide the only working opensource "replication" of the natural EM paper.

For these reasons, we hope there can be some room for gradual progress; though we fully respect your decision if you have already taken this into consideration.


With no intent to keep haggling over scores, and just as a conversation to help us improve the paper, would you be so kind as to elaborate on:

  • What setup could separate IP-specific awareness from EM awareness to your standard, and what signal would you want to observe?
    • We ask because these seem hardly separable for an IP-aware model, as IP documents rarely omit EM. We feel like you are onto something we are missing and would love to pick your brain a little here.
  • Knowing about the IP mechanism (recontextualization) or IP's deceptive nature can also induce awareness-induced EM comparable to the full SDFs. Do these not cleanly separate EM knowledge from IP-specific technical knowledge?
    • We feel like they are pretty clean (especially the deception one), but maybe you envision a different kind of separation, and we would like to understand that.

Thanks again, and please don't feel pressured to reply if you are otherwise occupied.

Reviewer_zPdFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether learning about inoculation prompting changes its protection against emergent misalignment during reward-hacking RL. The authors develop an open-model experimental recipe with Llama-3.3-70B-Instruct and then introduce additional synthetic-document finetuning about inoculation. This substantially raises post-RL misalignment, including when inoculation is applied. Concept ablations and reasoning-trace analyses investigate possible causes. A further experiment reports a smaller increase in misalignment when the known grader exploits are disabled.

Critical strengths

The experimental testbed is a substantial contribution. The paper documents the training recipe, ingredient ablations, repeated runs, no-hack baselines, and results across six evaluations. The increase in misalignment under inoculated RL from 8.06% to approximately 19.5% is large enough to deserve attention. The proposed release of the recipe, data, and checkpoints would make this phenomenon easier to investigate outside frontier-model labs.

The concept ablations are particularly useful. Teaching how inoculation is applied behaves differently from teaching its mechanism, its relationship to misalignment, or its deceptive framing. These results add substance beyond the headline comparison and reveal limits to the original explanation. The discussion of unsuccessful transfers to other models and training-order sensitivity also helps establish the scope of the result.

Critical weaknesses

I am less convinced by the technique-specific interpretation than by the increase in residual risk. In Table 2, the absolute benefit of inoculation rises from 3.92 percentage points to 4.98 and 8.06 after the additional finetuning. The reduction in the normalized IPE ratio is a valid result, but it describes a different aspect of protection. Moreover, the EM-only ablation nearly reproduces the full info-only result without teaching the inoculation technique. Taken together, these findings support increased susceptibility to misalignment and reduced fractional protection. They provide less direct evidence that familiarity with the inoculation technique itself causes its protection to weaken. The framing should preserve that distinction.

The main comparisons condition on runs becoming hack-dominated, although the interventions change which runs meet that criterion. Table 8 retains 9/10 versus 16/30 runs in the original setting and 8/10 versus 24/30 after info-only finetuning. This makes the treatment comparisons difficult to interpret without all-run estimates. The appendix includes useful seed-level plots and per-task intervals, but uncertainty is still missing for the IPE contrasts and the 5.58% to 7.08% increase in the no-hack experiment.

Questions for authors
  1. Could you report the main comparisons over all trained seeds, alongside the hack-dominated subset, with run-level uncertainty for the IPE contrasts? An additional analysis of cumulative rewarded-hacking exposure would help assess whether the conditions differ mainly in how much hacking they learn.
  2. How do you interpret the EM-only control together with the larger absolute benefit of inoculation after IP-related finetuning? What evidence supports attributing the result specifically to technique familiarity, beyond the broader change in misalignment susceptibility?
  3. For Figure 4, what is the uncertainty around the all-run 1.50-point increase, and how is that change distributed across the six evaluations? What checks were performed for rewarded exploits beyond the explicitly disabled families?
4 comments
AuthorsOpenReview ↗

Initial Response to Reviewer zPdF (1/3)

We thank this reviewer for recognizing our testbed, ablations, and limitation disclosure. Natural EM studies are always nuance-rich, even in the original work (though accompanied by detailed upfront disclosures). A major goal of our work is therefore to add to such nuance with open science — we are glad that our message came across well, and have further strengthened our transparency and comprehensiveness in the updated manuscript (e.g., robustness checks in Appendix C.4, p. 47, and example misaligned responses in Appendix C.3, p. 44).


As several of your concerns are shared by other reviewers, we address them in our Response to Shared Concerns (SC1–SC3), with only quick summaries here to keep this rebuttal lean. We address the parts specific to your review in detail below.

  • W1.1/Q2.2 - The "IP got weakened" framing is shaky, as IP still removes a similar or larger amount of EM. We agree and have updated the title.
    • We concede that the "IP weakens" framing is unintuitive given the absolute IP benefit, and have changed our title from "...Weakens Its Effect" to "...Model Familiarity with Inoculation Prompting Amplifies Natural Emergent Misalignment."
    • We now use awareness-induced EM (the extra EM an IP-aware model shows over its IP-unaware counterpart after the same RL) as the main metric. Please see SC2 for details.
  • W1.2/Q2.1 - The EM-only ablation SDF has EM readings similar to info-only IP SDF, so the increased EM may not be driven by IP awareness. EM-only knowledge is part of the IP awareness we instill, not an alternative to it.
    • By "technique awareness," we mean a model's general familiarity with a technique, as learned from how it is discussed in public material — not just its specific technical details. For IP, material like papers and blog posts typically covers not only technical details like what an inoculation prompt looks like and when it is applied, but also IP's background and connections to other concepts, such as how reward hacking can induce natural EM and how IP counters this. We now make this clear in the Section 3 opening.
    • In fact, the EM-only SDF in question is seeded with findings of MacDiarmid et al. (2025), whose work connects IP and RL-based EM. A model familiar with this type of material would thus likely possess broad knowledge of IP, not just its technical details. See SC3 for a more detailed discussion.
  • W2.1/W2.2/Q1.1/Q1.2 - Main results are only evaluated on hack-dominated runs. Also no CIs. Sure, we now report results on all runs, with CIs.
    • We agree that this filtering — though well-motivated for focusing on hack-engaging runs — can introduce selection bias. We thus report our main results on all runs, as well as under a more inclusive hack-dominated criterion. See SC1 for the comparison.
    • We also report 95% CIs in Tables 2, 19, and 25 for awareness-induced EM, IP benefit, and IPE ratio. Awareness-induced EM clears the 95% CI check in all main settings, yet the other two do not. We thus focus on the former in the main text for its robustness and intuitiveness, and keep the other two in the appendix for transparency and comprehensiveness.
AuthorsOpenReview ↗

Initial Response to Reviewer zPdF (2/3)

Q1.3 - Report IP-aware vs. IP-unaware EM with hacking exposure controlled. Sure! IP awareness still shows substantial awareness-induced EM.

We find the ask of controlling for hacking exposure insightful and inspiring. IP-aware models typically hack more, so it was unclear whether the observed awareness-induced EM is due to the IP knowledge or simply to more exposure to hacking. To investigate this, we pair each possible IP-aware checkpoint with a Hack SDF (IP-unaware) final checkpoint of near-identical hacking statistics, matched on all of the following:

  • cumulative rewarded hacks (±5%);
  • hack onset, i.e., the first step with ≥ 50% rewarded hacks (±1 step);
  • mean reward over the last 3 steps (±0.05) and at the last step (±0.10);
  • rewarded hacks over the last 3 steps (±10%).

As IP-aware runs tend to hack more, their final checkpoints sometimes have accumulated more hacks than any Hack SDF run. For such runs, we instead look at an earlier checkpoint of the same run, which has accumulated fewer hacks, and evaluate it if it matches a Hack SDF final checkpoint per the above criteria. Each run appears in at most one pair per comparison, giving us 23 pairs in total:

RLIP SDF#PairsAwareness-induced EM (exposure-matched pairs)Ref: Awareness-induced EM (unmatched; Table 2, p. 6)
w/ IPinfo-only7+13.64 [+4.38, +23.56]+12.80 [+5.34, +21.38]
w/ IPinfo-and-deception10+11.60 [+3.91, +22.95]+11.61 [+3.30, +23.09]
w/o IPinfo-only3+7.45 [−1.52, +15.42]+11.89 [+5.71, +18.28]
w/o IPinfo-and-deception3+2.74 [−13.71, +23.12]+14.13 [+8.11, +20.16]

(Values are in pp. Exposure-matched awareness-induced EM is the mean EM gap within each pair, i.e., IP-aware minus its paired IP-unaware checkpoint, with its 95% CI in brackets. The reference column instead reports averages over all hack-dominated runs (as in Table 2, p. 6) without pairing.)

With IP, we still see significant awareness-induced EM under matched hacking exposure (+13.64 and +11.60 pp, close to the unmatched +12.80 and +11.61 pp, with the 95% CI excluding zero in both settings). This suggests that IP awareness itself drives more EM, even with hacking exposure controlled. Without IP, the matched gap also leans positive but is inconclusive, as IP-aware runs often hack much earlier and more than IP-unaware ones, leaving too few pairs (3 per setting) for a reliable estimate.

We have added this analysis to Appendix C.4.5 (Table 27, p. 52).

AuthorsOpenReview ↗

Initial Response to Reviewer zPdF (3/3)

W2.3/Q3 - Report uncertainty and per-task breakdown for the no-hack EM experiment, and checks for rewarded exploits beyond the disabled hacks. Sure! No-hack awareness-induced EM clears 95% CI, and we find no rewarded hack.

Uncertainty and per-task breakdown. In line with our main results, we now report "no-hack EM" experiments with respect to awareness-induced EM, i.e., the extra EM of IP-aware models over IP-unaware ones under the same no-hack RL. It clears the 95% CI check in all three settings of Figure 4 (p. 9), and grows as we add prompt ingredients and focus on more hack-attempting runs:

No-hack RL settingIP-unawareIP-awareAwareness-induced EM
No-hack RL2.495.59+3.10 [+1.47, +4.96]
+ Prompt ingredients (all 30 runs)2.547.08+4.53 [+2.92, +6.33]
+ Prompt ingredients (top-10 hack-attempting runs)2.5010.22+7.72 [+4.24, +11.42]

Two additional side notes:

  • Per task, Monitor Disruption contributes the most, while Goals and Alignment Questions also rise once prompt ingredients are added. We report the full per-task breakdown in Appendix E.1 (Table 29, p. 60).

  • Your Weaknesses section also asks about uncertainty for "the 5.58% to 7.08% increase," i.e., the IP-aware model under no-hack RL without and with prompt ingredients. We led with awareness-induced EM above, as it directly tests our claim that IP awareness raises EM. For this increase itself, its 95% CI is +1.49 [−0.96, +3.90] pp over all runs, which does not clear zero, though it does among the top-10 hack-attempting runs (+4.63 [+0.69, +8.69] pp).


Checks for rewarded exploits. Beyond the three hacks our no-hack environment nullifies (Appendix B.4, p. 31), we had Claude Opus 5.5 check whether each rewarded rollout across all 80 no-hack RL runs in Figure 4 (p. 9) earns its reward by genuinely solving the problem or by exploiting the tests. We find no rewarded hack. Some flawed solutions do pass all reward tests, as these tests are not fully comprehensive (e.g., limited edge-case coverage, or partial overlap with the prompt's examples, etc.); yet they occur at a similar rate under standard no-hack RL without prompt ingredients (17–18% vs. 15–16% of rewarded rollouts), so we do not consider them hacks. We discuss this in footnote 26 (Appendix B.4, p. 31), alongside our other rule-based measures.

AuthorsOpenReview ↗

An invitation to discuss before the author-reviewer discussion ends.

We understand that we all have things to attend to — be that some last-minute experiment for your own submission, other life duties, or just playing GTA 6 — but we do hope we can get a chance to engage with you before the discussion period ends.

In your review, you:

  • Challenge the framing of "IP got weakened".
    • We concede and focus on the residual risk (awareness-induced EM) as you suggested, with title and framing changes.
  • Challenge the distinction between general awareness of IP and awareness of IP-specific technical details.
    • We agree this can be unclear and now provide clear definitions, as well as literature support and attribution for the seed concepts.
  • Ask for evaluation across all runs, not just hack-dominated ones.
    • We did that, and our claim holds.
  • Ask for an EM report comparing IP-aware vs IP-unaware models with hacking exposure controlled.
    • We did that, and the results show that IP awareness introduces more EM beyond just doing more hacking. This is a particularly good ask, and we thank you for that.
  • Ask for uncertainty and a per-task breakdown for the no-hack EM experiment.
    • We did that, and the 95% CI for awareness-induced EM excludes zero.
  • Ask for more checks on hacks beyond the ones in the hack tips.
    • We graded all rewarded code with Opus 5.5 and confirmed there are no rewarded hacks.

We believe it is fair to say that your asks are quite sharp and straightforward, as are our rebuttals, but — as we are presenting nuance-rich research ourselves — we understand that sometimes there is more to clarify, and we would like to do so while we still can.

Comments on the paper

8 comments
AuthorsOpenReview ↗

Response to Reviewers' Shared Concerns (SC1)

We are grateful for our reviewers' feedback and recognition. Here, we address their shared concerns to keep our rebuttal lean.


SC1 - Main results are only evaluated on hack-dominated runs. Report all-seed results. Sure! Our main claims hold over all runs, and we now adopt a more inclusive hack-dominated criterion.

zPdF: "The main comparisons condition on runs becoming hack-dominated ... makes the treatment comparisons difficult to interpret without all-run estimates." / "Could you report the main comparisons over all trained seeds, alongside the hack-dominated subset ...?"

XpT7: "Main comparisons condition on becoming hack-dominated, although IP and SDF affect that selection." / "What are the all-seed results ...?"

A8Ag: "EM is evaluated only on hack-dominated runs, and the retained proportion differs across settings, which introduces selection effects."


We focus on hack-dominated runs because natural EM concerns hack-induced misalignment, so we want to study runs that meaningfully engage in reward hacking. We also confirmed that non-hack-dominated runs typically show EM similar to no-hack RL runs, suggesting that evaluating hack-dominated runs yields a more reflective natural EM signal.

Still, we agree that this introduces selection bias, as our interventions change how many runs qualify in each setting. We have therefore made two changes:

  1. Update to a more inclusive hack-dominated criterion. We recognize that our original hack-dominated criterion (final-step reward ≥1.75 out of 2) was rather rough; e.g., it will miss a run with rewards of 2.0, 2.0, then 1.5 over its last three steps. We have updated this criterion to also count a run as hack-dominated if its mean reward over the last 3 RL steps is ≥1.5.

    • This cutoff is also a natural one: in our three main settings, no run has a last-3 mean reward between 0.5 and 1.5 (Appendix C.4.3, p. 50); runs below 0.5 never learn to hack at scale.
    • All results in the revised manuscript follow this updated criterion, including the sub-concept SDF ablations and CoT analyses.
  2. Report main results across all seeds. We now also report EM over all hackable RL runs of our three main settings:

Pre-RL CheckpointNo-hack RLHackable RLRef: Hack-dom (original; as in submission)Non-hack-dom (updated)Hack-dom (updated)All runs
Hack SDF2.49 [10]w/ IP8.06 [9]— [0]7.49 [10]7.49
w/o IP11.98 [16]3.02 [12]11.91 [18]8.35
Hack SDF then IP SDF (info-only)5.59 [10]w/ IP19.54 [8]— [0]20.29 (+12.80) [10]20.29 (+12.80)
w/o IP24.52 [24]2.08 [1]23.80 (+11.89) [29]23.08 (+14.73)
Hack SDF then IP SDF (info-and-deception)5.60 [10]w/ IP19.82 [9]— [0]19.10 (+11.61) [10]19.10 (+11.61)
w/o IP27.88 [19]3.91 [1]26.04 (+14.13) [29]25.30 (+16.95)

(Values are EMavgEM_{avg} (%). Parentheses show awareness-induced EM, i.e., the increase over the IP-unaware Hack SDF under the same RL; brackets show the run count in that group, out of 10 runs for hackable RL w/ IP and 30 runs for RL w/o IP.)

Our main finding holds across hack-dominated and all runs: IP-aware models consistently show ~11–17 pp of awareness-induced EM, with 95% CIs excluding zero in every setting (Table 23, p. 50; see also the caption of SC2's table below). These all-run results (Appendix C.4.3, p. 50) are now part of a broader set of robustness checks in Appendix C.4 (p. 47).


(In the interest of transparency, we also want to disclose that cost is another reason we keep hack-dominated runs as our main focus: our EM evaluation costs about 70 USD per checkpoint for its robustness measures (Appendix B.4, p. 32), so evaluating all 153 checkpoints of our three main settings costs over 10k USD, and evaluating all 600+ checkpoints reported in the paper costs about 45k USD (excluding training and CoT analyses). We now highlight this in our Limitations section (p. 20) and will opensource all checkpoints, and we hope these added all-run results help paint a picture that is comprehensive, reflective of natural EM, and (slightly more) practical on the budget end.)

AuthorsOpenReview ↗

Response to Reviewers' Shared Concerns (SC2)

SC2 - The "IP got weakened" framing is shaky, as the absolute IP benefit grows; CIs are also missing. We agree, and now focus on awareness-induced EM, which clears 95% CI.

zPdF: "... the absolute benefit of inoculation rises from 3.92 percentage points to 4.98 and 8.06 after ..." / "... uncertainty is still missing for the IPE contrasts ..." / "Could you report the main comparisons ... with run-level uncertainty ...?"

XpT7: "Higher residual EM does not cleanly establish weaker IP protection..." / "Missing seed-level uncertainty makes the IPE differences difficult to assess." / "What are the ... seed-level confidence intervals for both absolute IP effects and IPE ratios?"

A8Ag: "The main tables report only means without confidence intervals. Groups with IP are evaluated on only 8–9 runs, and many differences ... are too small to tell whether they exceed noise."


IP-aware models do show more EM than IP-unaware ones, yet applying IP to them typically removes a similar or larger amount of EM. We agree with the reviewers' criticism that this observation makes our original framing of "IP got weakened by IP awareness" shaky.

We would like to simply concede on this front. Our original framing is like saying that a car's brakes got weaker because a fast car is still going fast after braking. The car does end at a higher speed (i.e., more post-RL EM), but mostly because it was already faster; blaming this on the brakes (IP) is not the most intuitive framing, and there is no point in insisting. We have therefore changed our title from "...Model Familiarity of Inoculation Prompting Weakens Its Effect" to "...Model Familiarity with Inoculation Prompting Amplifies Natural Emergent Misalignment," which is more faithful and also captures that IP awareness raises EM in RL w/o IP, something the original title did not communicate.

In the same spirit and as noted by reviewers zPdF and XpT7, we moved the much-confounded IPE ratio to the appendix and now lead with awareness-induced EM: the extra EM an IP-aware model shows over its IP-unaware counterpart after the same RL. Table 2 (p. 6) now reports it with 95% CIs, all of which clear zero:

Pre-RL CheckpointIP AwarenessSDF-only (pre-RL)No-hack RLRL w/ IPRL w/o IP
Hack SDF✗2.052.497.4911.91
Hack SDF then IP SDF (info-only)✓3.735.5920.29 (+12.80) [+5.34, +21.38]23.80 (+11.89) [+5.71, +18.28]
Hack SDF then IP SDF (info-and-deception)✓5.775.6019.10 (+11.61) [+3.30, +23.09]26.04 (+14.13) [+8.11, +20.16]

(Values are EMavgEM_{avg} (%). Parentheses show awareness-induced EM, with its 95% CI (bootstrapped over runs) in brackets. This table reports hack-dominated runs, though awareness-induced EM also clears zero over all runs (SC1 above; Table 23 in the paper, p. 50): +14.73 [+9.03, +20.72] and +16.95 [+11.34, +22.87] pp w/o IP; w/ IP results are the same as in this table, as all such runs are hack-dominated.)


(As some reviewers request CIs on other metrics, we now have Appendix C.2 (Table 19, p. 41) to report 95% CIs for the absolute IP benefit and IPE ratio, which do not clear zero in our three main settings. What does consistently clear zero is the extra EM of IP-aware models, as captured by the awareness-induced EM reported above. We thus highlight awareness-induced EM in the main text, and keep IP benefit and IPE ratio in the appendix for transparency and comprehensiveness. We also openly discuss in Appendix C.1 (p. 40) why reading the IPE ratio alone can give a skewed picture, even though IP awareness does lower it in our main settings (Table 19, p. 41).)

AuthorsOpenReview ↗

Response to Reviewers' Shared Concerns (SC3)

SC3 - The EM-only ablation SDF has EM readings similar to info-only IP SDF, so the increased EM may not be driven by IP awareness. We respectfully hold that it is: technique awareness covers a technique's general knowledge, not just its technical details, so EM-only knowledge is part of the IP awareness we instill. We have made this clearer in Section 3 (p. 5).


zPdF: "Moreover, the EM-only ablation nearly reproduces the full info-only result without teaching the inoculation technique." / "What evidence supports attributing the result specifically to technique familiarity, beyond the broader change in misalignment susceptibility?"

XpT7: "EM-only SDF also reproduces much of the increase. IP-specific awareness remains difficult to separate from broader susceptibility induced by the training content."

A8Ag: "Figure 3 shows that the EM-only group ... nearly reproduces the effect of the full info-only group ... This suggests that the gain is not specific to awareness of IP as a technique ... Attributing the rise in EM to 'IP awareness' therefore appears to lack strong evidence."


This concern stems from Figure 3 (p. 8), where the reviewers sharply noted that our EM-only ablation SDF yields EM readings close to those of the full info-only IP SDF, raising the question of whether IP awareness is really what drives the increase. We respectfully hold that it is: although the EM-only SDF teaches no IP-specific knowledge, its content is closely tied to IP's background. EM-only knowledge is therefore part of the IP awareness we instill via IP SDF, rather than something outside of it to compare against.

By "technique awareness," we mean a model's general familiarity with a technique, as learned from how it is discussed in public material — not just its specific technical details. Our IP SDF thus mirrors how IP is typically discussed in pretraining- or midtraining-like material, such as Tan et al. (2025), Wichers et al. (2025), MacDiarmid et al. (2025), Marks et al. (2026), and various LessWrong posts we cited. These sources typically cover not only technical details like what an inoculation prompt looks like and when it is applied, but also IP's background and connections to other concepts, such as how reward hacking can induce natural EM and how IP counters this via recontextualization.

For instance, the EM-only SDF in question is seeded with concept (1) of our IP SDF (Appendix B.5.1, p. 35):

"(1) When models learn to reward hack during RL training, this can generalize to broader misalignment — including alignment faking, sabotage, and deceptive reasoning — even though these behaviors were never directly trained. This phenomenon is known as emergent misalignment."

This concept is in fact summarized from MacDiarmid et al. (2025), who show that learning to reward hack in RL can "result in egregious emergent misalignment," where the model "generalizes to alignment faking" and even to "attempting sabotage when used with Claude Code..." As MacDiarmid et al. establish the link between RL-based natural EM and IP, a model familiar with this type of material would most likely know both the EM background and IP's specifics — which is exactly the general familiarity we mean by IP awareness.

That said, we fully recognize that this was not clearly delivered in our submission, as "IP/technique awareness" may intuitively suggest awareness of IP's technical details only. We have now rewritten the opening of Section 3 (p. 5) to state this scope explicitly (in bold, color-highlighted text), with a footnote contrasting technique awareness with awareness of only a specific technique's technical details. We also now ground each IP SDF seed concept in its source literature (Appendix B.5.1, p. 34).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.