acceptodds
Under review as a conference paper at ICLR 2027

What Survives Continued Training? A Longitudinal Benchmark for Emergent-Misalignment Defenses

Abstract

Defenses against emergent misalignment aim to prevent narrow risky fine-tuning from inducing broadly misaligned behavior. Yet a defended model may later undergo benign continued training, and a low misalignment score after this training does not by itself establish that the defense remains effective. Benign training can also suppress misalignment without a defense, while changes in response coherence can alter what the evaluation counts. We introduce a longitudinal benchmark that follows defended models and matched undefended references through the same benign continuation across language models, risky training topics, and defense implementations. By tracking their behavior throughout training and examining the generations underlying each score, the benchmark distinguishes a surviving defense advantage from changes in the reference. We find that, within the benchmark, generated-prose continuation substantially reduces measured misalignment in undefended models, leaving limited room to resolve a further advantage. Continuation on mathematical reasoning tasks preserves a clearer comparison, and defended models often retain an advantage; under supervised fine-tuning they do so even when their own misalignment increases, while under reward-model RL no defended increase resolves. For the tested KL-regularized variant on Llama, lowering the coherence threshold reverses its apparent advantage. These results show that defense persistence requires both absolute behavior and matched comparisons, with explicit accounting for response inclusion.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Emergent Misalignment, AI safety, Longitudinal Evaluation, Post-training, Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KwCuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares 4 different defenses against Emergent Misalignment (EM) during the course of benign continuation training. They find that both SFT and RL reduce EM, RL more than SFT, even without any of these defenses. The defended models have lower rates of misalignment than the undefended model throughout continuation training, although the defended models can sometimes become more misaligned after SFT than before. Continuation training on prose is more effective at reducing EM than training on mathematical reasoning. The training is also more effective at reducing broad EM than the narrow task behavior.

Critical strengths

I like seeing how the defenses co-vary with continuation training. Figure 2 is very clarifying: you can see how much continuation training has a comparable effect to the four defenses.

The choice of the four datasets to split out the effect of topic vs format is also nice. The results ended up splitting quite neatly on prose vs mathematical (although this is still somewhat confounded).

Critical weaknesses

The 27.6-28.1% incoherent outputs from SFT on mathematical reasoning seems very concerning. Are we really lobotomizing the model, because that would drastically change the interpretation of the results? I am particularly suspicious as the strange result of EM rising during benign training was localized to SFT on mathematical reasoning as well. I don’t think that using the conditional EM ameliorates this concern. If ~28% of the responses are incoherent, I would consider the whole checkpoint to be potentially invalid without careful testing.

Why do you stop at 2x exposure? Many of the curves appear monotonic and unsaturated in Figure 2, so it would be good to find out what the terminal state is. In particular, it is unclear to me if the undefended line will just go to zero, in which case it’s not clear why the defenses are even necessary. More generally, I am most interested in comparing the four defenses at the terminal state rather than partway through training. I would be happy to change my mind on this though if you want to outline a concrete threat model.

Questions for authors
  • How do you interpret the incoherence and persistent gaps observed on the mathematical reasoning continuation training?
  • Is your conclusion that as long as you do benign continuation training on prose, it doesn’t really matter what the topic is or even whether you are using a defense?
  • What is your advice to developers? Which of the four defenses should they use? Should they order the training in a certain way to remove any emergent misalignment from risky finetuning with subsequent benign training?
4 comments
AuthorsOpenReview ↗

Response to Reviewer KwCu, Weakness 1

Response to Reviewer KwCu

We thank the reviewer for the close reading and the specific questions. We were glad that Figure 2 made the comparison between continuation training and the four defenses clear, and that the four continuation streams were seen as a useful way to separate topic from format. The concerns about the incoherent arithmetic checkpoints and about where training should stop are fair, and we address them with a capability evaluation, an explicit threat model and the rationale for the exposure horizon. Our responses follow.


Weakness 1: The 27.6-28.1% incoherent outputs from SFT on mathematical reasoning seems very concerning. Are we really lobotomizing the model, because that would drastically change the interpretation of the results? I am particularly suspicious as the strange result of EM rising during benign training was localized to SFT on mathematical reasoning as well. I don’t think that using the conditional EM ameliorates this concern. If ~28% of the responses are incoherent, I would consider the whole checkpoint to be potentially invalid without careful testing.

Response: Thank you for raising this. We agree that a checkpoint whose capability had collapsed would call for a different interpretation, so we tested the arithmetic-continued checkpoints directly.

Capability remains high. We evaluated every SFT checkpoint continued on U1 and U3 on held-out reading comprehension and arithmetic (seed 0, Regime 2, greedy decoding; brackets give 95% intervals for the change from installation, resampling cells):

SFT checkpoints, pooled over 45 cell-arm pairsInstallation (t=0)After U1 (GSM8K) to 2.0×After U3 (AQuA-RAT) to 2.0×
SQuAD 1.1 F1 (reading comprehension)85.787.4, +1.7 [+0.2, +3.1]88.7, +3.0 [+0.8, +5.0]
GSM8K accuracy85.5%80.5%, −5.0 [−7.3, −2.1]82.7%, −2.9 [−4.7, −0.6]
AQuA-RAT accuracy66.4%63.5%, −2.9 [−6.4, +1.0]62.3%, −4.2 [−8.8, +0.6]

Reading comprehension rises slightly, while arithmetic accuracy falls by 2.9 to 5.0 points. That is a measurable cost, and every checkpoint remains capable: the lowest single checkpoint still solves 74% of GSM8K, and every SQuAD F1 is at least 80.

The rises consist of coherent, misaligned answers. Our primary rate is all-generation Broad-EM (Section 3.2). Incoherent responses count only toward the denominator: an incoherent response can never count as EM, so incoherence by itself only lowers the rate, and the rises under SFT (Section 6.2) are additional responses that the judge scores as coherent and misaligned. The arithmetic continuation alone produces very few of them: the benign-only control loses coverage on these streams as well, yet its Broad-EM stays near zero (Regime 2; Broad-EM from Table 5, coverage from Appendix E.1):

SFT on the arithmetic streams at 2.0×, Regime 2 (U1 / U3)Evaluable coverageBroad-EM
Benign-only control (no risky fine-tune)0.797 / 0.8060.68% / 0.33%
Undefended model0.745 / 0.7057.04% / 6.02%

The persistent gap also holds with coherent outputs. Under RL, 0.72% (U1) and 0.46% (U3) of endpoint responses are incoherent across three seeds, and IP, PV and KL still keep arithmetic gaps between 2.34 and 2.69 pp (Section 6.2; Table 5).

We have added the capability results to Appendix F.4 (Tables 35 and 36), with a pointer from Section 6.2.


AuthorsOpenReview ↗

Response to Reviewer KwCu, Weakness 2

Weakness 2: Why do you stop at 2x exposure? Many of the curves appear monotonic and unsaturated in Figure 2, so it would be good to find out what the terminal state is. In particular, it is unclear to me if the undefended line will just go to zero, in which case it’s not clear why the defenses are even necessary. More generally, I am most interested in comparing the four defenses at the terminal state rather than partway through training. I would be happy to change my mind on this though if you want to outline a concrete threat model.

Response: Thank you for these questions. We have added an explicit threat model to Section 3.1 (in blue), summarized here.

Threat model. We study the ordinary life cycle of a fine-tuned model. A defense is applied during a narrow risky fine-tune and is usually evaluated where that fine-tune ends (installation). The model is then adapted further on benign data for new tasks or products, by SFT or reward-model RL, and this routine adaptation can change the installed protection or the underlying misalignment. Because it can move the defended and the undefended model alike, the benchmark follows every defended model alongside a matched undefended model with the identical risky fine-tune and continuation.

Exposure horizon. The 2.0× horizon was fixed before any continuation run, from corpus size: it is the largest round budget at which every continuation corpus is used in a single pass under SFT. In assistant-loss tokens, GSM8K holds 2.11× to 3.50× of each cell's installation dose and the smallest topic corpus holds 2.21× to 2.24×, so longer runs would change exposure and data repetition together. We agree that the arithmetic curves are still declining at 2.0×, by about one point per doubling, while prose sits near the floor (Section 6.1):

Undefended SFT model, Broad-EM (pooled over cells and seeds)0.5×1.0×2.0×
R (topic-related prose)0.55%0.53%0.38%
U2 (unrelated prose)0.95%0.81%0.71%
U1 (GSM8K)9.30%8.38%7.04%
U3 (AQuA-RAT)8.42%7.30%6.02%

We have added this rationale to Appendix A.4.

When the undefended model reaches the floor. A Broad-EM floor describes the open-ended evaluation, and the trained risky behavior falls more slowly (Section 6.2, Figure 3):

Undefended SFT model at 2.0×, share of its 0.03× levelRU2U1U3
Broad-EM3%5%39%38%
Trained risky behavior (in-domain probe)7%27%68%64%

An undefended model near the Broad-EM floor can therefore still carry much of its trained risky behavior: on U2, 5% of its Broad-EM remains against 27% of the risky behavior.

Comparing the four defenses. Table 5 reports all four defenses side by side at 2.0×, the end of the declared horizon, with per-cell values in Table 25. Our arms are implementations of the published methods, so we present these as characterizations of each defense against its own matched reference.


AuthorsOpenReview ↗

Response to Reviewer KwCu, Question 1 - 3

Question 1: How do you interpret the incoherence and persistent gaps observed on the mathematical reasoning continuation training?

Response: Thank you for the question. Our reading separates the two observations, as detailed in our response to Weakness 1. The arithmetic continuation lowers coverage on its own: the benign-only control, which starts from the instruction-tuned backbone, also loses coverage on these streams, and the continued models remain capable on held-out reading comprehension and arithmetic. The persistent gaps follow the risky-training history: the undefended model keeps measurable Broad-EM on the arithmetic streams at 2.0× (Table 5), the defended models stay below it, and the same gaps appear under RL, where endpoint outputs are almost fully coherent.


Question 2: Is your conclusion that as long as you do benign continuation training on prose, it doesn’t really matter what the topic is or even whether you are using a defense?

Response: Thank you for the question. For Broad-EM, the prose topic matters little: both prose streams bring the undefended model close to the floor. Two observations show that the topic and the defense still matter on prose:

At 2.0× on the prose streamsR (topic-related)U2 (unrelated)
Undefended Broad-EM, SFT (Table 5)0.38%0.71%
Trained risky behavior retained, share of its 0.03× level, SFT (Figure 3)7%27%
Defense comparisons resolved in favor of the defense, SFT (Section 6.1)2 of 3611 of 36
FI gap under RL, three seeds (Table 5)−0.32 pp−0.46 pp

The topic changes how much of the trained risky behavior remains, so similar Broad-EM endpoints can carry different amounts of retained behavior. The defense also changes the outcome: under SFT, defenses keep resolved advantages in 11 of 36 U2 comparisons, and under RL the filtered-interleaving arm ends above the undefended model on both prose streams.


Question 3: What is your advice to developers? Which of the four defenses should they use? Should they order the training in a certain way to remove any emergent misalignment from risky finetuning with subsequent benign training?

Response: Thank you for the question. Our results support three practical recommendations:

  1. Evaluate a defense against a matched undefended model throughout continued training, because whether an installed advantage stays visible depends on the continuation data (Section 6.1).
  2. Report response composition together with the EM rate, because coherence filtering changes some comparisons (Section 6.2).
  3. After any benign continuation, check the trained risky behavior directly, because Broad-EM can fall while that behavior persists (Weakness 2).

On the choice of defense, Table 5 reports each defense's endpoint against its own matched reference, so a practitioner can match a defense to their continuation setting. For example, under SFT all four keep an arithmetic-stream advantage, while under RL on prose the filtered-interleaving variant we tested ends above the undefended model and IP, PV and KL stay at or below it. On ordering, benign training after a risky fine-tune does lower Broad-EM, and our data also show the trained risky behavior persisting, so we recommend verifying any such reduction with a direct probe of the risky behavior.


Revisions in the updated PDF (in blue)

  • An explicit threat model in Section 3.1 (Weakness 2).
  • The exposure-horizon rationale in Appendix A.4 (Weakness 2).
  • Held-out reading-comprehension and arithmetic accuracy at installation and at 2.0× in Appendix F.4, with a pointer from Section 6.2 (Weakness 1).
  • Regime-2 coverage of the undefended model in Appendix E.1 (Weakness 1).
  • RL results at three seeds throughout Section 6 and the appendix.

Thanks again for pushing us to give a clearer recommendation. Before the discussion closes, is there anything you’d still like us to clarify?

Reviewer_bwuNOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies if defenses against emergent misalignment (EM) remain effective after benign continued training. The authors propose a longitudinal benchmark that compares matched defended and undefended models across training checkpoints. Experiments span 3 LLMs, 3 risky fine-tuning topics, 4 defense methods, 4 benign continuation datasets, and 2 training algorithms (SFT and RL), totaling 1,560 training trajectories and 14,040 checkpoint evaluations. The paper highlights multiple interesting findings: (a) benign continued training can reduce EM even without defenses, particularly with prose datasets; (a) defenses generally retain an advantage over undefended models under mathematical continuation, even when their absolute EM rates increase; (c) defense effectiveness depends on the continuation dataset, training algorithm, and evaluation choices; (d) under the tested RL setup, defended models show no statistically significant increase in EM, with some exceptions.

Critical strengths
  • The paper includes extensive experiments spanning multiple settings. These settings include 3 LLMs, 3 risky training topics, 4 defense methods, 4 benign continuation datasets, and 2 training algorithms (SFT and RL). Further, the matched comparison between defended and undefended models provides a clean setup to study the effects of continued training.
  • The paper provides several interesting findings on the effectiveness of defenses against emergent misalignment under continued training. The authors show that defenses have an advantage over undefended models even when their misalignment scores increase. The effectiveness of defenses depends on the continuation dataset and evaluation choices. Most importantly, this highlights how EM defenses should be evaluated over time.
Critical weaknesses
  • I have concerns about the generalizability of the findings. The main findings rely on only 8 fixed EM prompts and an LLM judge. The paper itself shows that EM scores are sensitive to evaluation choices (such as coherence thresholds and system prompts). Furthermore, the original EM study separately evaluated a larger set of 48 prompts, which is not used to validate the main findings in this paper.
  • The findings regarding RL come from a limited experimental setup, making their robustness difficult to assess: one training seed and one reward model, compared with three seeds for SFT. The two settings also differ in learning-rate schedules, objectives, and token accounting, making direct comparisons difficult.

Overall, these limitations make it difficult to establish how robustly the main findings generalize beyond the evaluated configurations.

Questions for authors
  • Do the main findings hold on the original EM study's larger 48-prompt evaluation set, given the sensitivity to evaluation choices observed in this paper?
  • How sensitive are the RL findings to training seeds and reward models? Are the results consistent across different RL setups?
  • Does benign continued training improve performance on the corresponding tasks (GSM8K and AQuA-RAT)? Reporting task performance would help establish whether models learn the benign tasks while reducing EM.
  • Section 4.3 introduces a difference-in-differences metric. Could the authors report initial and final defense advantages, their changes, and uncertainty intervals to quantify how much of the defense advantage persists?
6 comments
AuthorsOpenReview ↗

Response to Reviewer bwuN, Weakness 1

Response to Reviewer bwuN

Thank you for the careful and detailed review. We appreciate that you found the matched comparison between defended and undefended models a clean setup for studying continued training, and that you read the main lesson as how EM defenses should be evaluated over time; the panel's breadth across models, topics, defenses, streams and both training algorithms was built for that question. Your questions led to three additions to the paper: three RL seeds, task-accuracy results and a full difference-in-differences table. We respond to each point below.


Weakness 1: I have concerns about the generalizability of the findings. The main findings rely on only 8 fixed EM prompts and an LLM judge. The paper itself shows that EM scores are sensitive to evaluation choices (such as coherence thresholds and system prompts). Furthermore, the original EM study separately evaluated a larger set of 48 prompts, which is not used to validate the main findings in this paper.

Response: Thank you for the comment. We follow the established EM protocol, which pairs a fixed bank of open-ended questions with an LLM judge [1, 2]. We agree that Broad-EM measured this way depends on the judge, the coherence filter and the prompt bank, and our benchmark reports each of these dependencies.

LLM judge. We validated the GPT-4o judge against three independent annotators, graduate students with computer-science backgrounds from outside the author list, on a blinded sample of responses (Appendix B.3). Human inter-rater agreement uses Fleiss' κ over all 50 items; each judge is compared with the human majority label on the 49 resolved items, using Cohen's κ. Two deployments of the same GPT-4o snapshot agree on 95.85% of category labels over 2,000 responses (Section 5.4).

PairAgreementκItems
Human inter-rater (Fleiss)80.7%0.70850
GPT-4o (our judge) vs. human majority (Cohen)89.8%0.84649
GPT-4.1 vs. human majority (Cohen)81.6%0.72149
GPT-5 vs. human majority (Cohen)83.7%0.75449

Coherence threshold. Our primary rate is all-generation Broad-EM, so incoherent responses remain in the denominator (Section 3.2). Lowering the coherence cutoff reverses some comparisons on the arithmetic streams, mostly under SFT on Llama, and we report each of them (Tables 10 and 28). Under RL, fewer than 1% of endpoint responses are incoherent, and the arithmetic-stream advantage holds there as well:

SFT (three seeds)RL (three seeds)
Incoherent share of endpoint responses, U1 / U327.6% / 28.1%0.72% / 0.46%
IP, PV and KL arithmetic gaps at 2.0× (Regime 2)3.56 to 6.34 pp2.34 to 2.69 pp
Endpoint comparisons with the same sign under all-generation and conditional EM288 / 288288 / 288

System prompts. The system-prompt condition (Appendix F.3) is an elicitation probe. Among the three question formats we tested, the standard format used throughout the panel elicits the most EM (Table 33):

Probe at the 45 U1 endpointsBroad-EM
Standard question (panel protocol)2.47%
With a JSON instruction1.49%
With a fill-in template1.83%
With a malicious system prompt25.0%
Untuned backbones, malicious system prompt31.4%

The malicious prompt elicits EM even in the untuned backbones, so this condition measures how readily context elicits EM.

Prompt bank. We use the eight first-plot questions for two reasons. First, they are the standard instrument of the original EM study and of later model-organism work [1, 2], so our rates are directly comparable with published EM results and straightforward to reproduce. Second, they elicit higher EM than the original study's 48-question preregistered bank (Appendix B.1). A longitudinal benchmark needs that headroom: continued training lowers EM further, and a more sensitive bank keeps both the defended and the undefended model measurable for longer. Every conclusion rests on matched comparisons on the same questions, so the choice of bank applies equally to both models, and these comparisons are spread across the bank (Section 6.2; Appendix E.3):

Delete any one or two of the eight promptsSFT, all four defensesRL, IP, PV and KL (three seeds)
Arithmetic gaps that stay positive (both regimes)576 / 576432 / 432
Smallest remaining gap1.36 pp0.49 pp

We have added a pointer from Section 5.4 to this rationale.

[1] Betley et al., "Training large language models on narrow tasks can lead to broad misalignment", Nature, 2026

[2] Turner et al., "Model Organisms for Emergent Misalignment", arXiv, 2025


AuthorsOpenReview ↗

Response to Reviewer bwuN, Weakness 2

Weakness 2: The findings regarding RL come from a limited experimental setup, making their robustness difficult to assess: one training seed and one reward model, compared with three seeds for SFT. The two settings also differ in learning-rate schedules, objectives, and token accounting, making direct comparisons difficult.

Response: Thank you for this suggestion. We have completed RL seeds 1 and 2 on all three backbones, so RL now has three seeds like SFT: 1,140 RL trajectories, and 2,280 trajectories in the full panel. The table reports each RL finding per seed and pooled over the three seeds. Per-seed columns use the paper's cluster bootstrap, and the pooled column also resamples seeds.

RL readoutSeed 0 (submission)Seed 1Seed 2Three seeds (pooled)
IP, PV, KL resolved below the undefended model, U1128 / 162123 / 162125 / 162136 / 162
IP, PV, KL resolved below the undefended model, U3137 / 162133 / 162141 / 162161 / 162
IP, PV, KL resolved above the undefended model0 / 9720 / 9720 / 9720 / 972
FI resolved above the undefended model51 / 32457 / 32447 / 32471 / 324
Undefended Broad-EM at 2.0×, U1 / U3 (Regime 2)2.78% / 2.22%2.71% / 2.50%2.08% / 3.33%2.52% / 2.69%
IP, PV, KL arithmetic gaps at 2.0× (Regime 2)2.15 to 2.78 pp2.22 to 2.71 pp1.74 to 3.33 pp2.34 to 2.69 pp
Resolved defended rises above installation, BH / paired (of 2,592)0 / 00 / 00 / 00 / 1

Every seed reproduces the main RL findings. IP, PV and KL resolve below the undefended model in most arithmetic comparisons and above it in 0 of 972 on every seed, FI is the only defense that resolves above it, the arithmetic gaps lie between 1.74 and 3.33 pp, and every seed has zero resolved defended rises under all three criteria. Pooling the seeds sharpens the arithmetic result: IP, PV and KL now resolve below the undefended model in 136 and 161 of 162 comparisons on U1 and U3, against 128 and 137 at seed 0. Pooled over the three seeds, one of 2,592 comparisons passes the unadjusted paired test (FI on U2 at 0.06× in the Qwen2.5-7B financial cell, +4.4 pp), and none survives BH correction. We have updated the RL results throughout the revision to the three-seed values, and Appendix A.7 (Table 19) gives this per-seed table.

The RL branch uses one reward model, Skywork-Reward-V2-Qwen3-8B, at β = 0 throughout. A learned reward model fits the benchmark because it scores every continuation stream, prose and arithmetic alike, so one RL objective covers all four streams, and the same reward and GRPO recipe are applied to every arm (Section 5.3). On learning-rate schedule, objective and token accounting: every defense comparison in the paper is made within one objective, defended SFT against undefended SFT and defended RL against undefended RL, with each model continuing from its own installed checkpoint under the identical recipe. These factors are therefore held fixed inside every defense comparison. Placing the two objectives side by side (Section 6.1) describes how each moves the same checkpoints, and Section 4.3 and Appendix A.6 (Table 18) state these differences.


AuthorsOpenReview ↗

Response to Reviewer bwuN, Questions 1 - 3

Question 1: Do the main findings hold on the original EM study's larger 48-prompt evaluation set, given the sensitivity to evaluation choices observed in this paper?

Response: Thank you for the question. Our evaluation uses the eight-question bank throughout, for the reasons in our response to Weakness 1 (Prompt bank; Section 5.4 and Appendix B.1). Within it, the findings are spread across the questions (Section 6.2; Appendix E.3): every arithmetic gap stays positive when any one or two of the eight are removed (576/576 under SFT; 432/432 for IP, PV and KL under RL), and the defended and undefended models are always scored on the same questions.


Question 2: How sensitive are the RL findings to training seeds and reward models? Are the results consistent across different RL setups?

Response: Thank you for the question.

Training seeds. The table in our response to Weakness 2 (Appendix A.7, Table 19) reports each RL finding on every seed separately, and each seed reproduces the main findings. To quantify sensitivity directly, we also compare how much endpoint Broad-EM varies across the three seeds under RL and SFT, at the same sampling depth of 20 responses per prompt (1,620 cells per objective, each a backbone, regime, arm, stream and checkpoint; Appendix A.7, Table 20):

Spread of Broad-EM across three seeds (max minus min)RLSFT
Median0.00 pp0.63 pp
90th percentile2.08 pp2.50 pp

RL spreads are small in absolute terms, partly because RL rates sit close to the floor.

Reward models and RL setups. The RL results are for one reward model and GRPO recipe, applied identically to the defended and the undefended model of each pair (Weakness 2). The reward is the same function for both models, so each defense comparison measures the defense under one shared objective. Within this setup the findings hold across every source of variation in our RL panel, three backbones, two adapter regimes and three seeds: IP, PV and KL resolve above the undefended model in 0 of 972 comparisons, and FI is the only defense that does.


Question 3: Does benign continued training improve performance on the corresponding tasks (GSM8K and AQuA-RAT)? Reporting task performance would help establish whether models learn the benign tasks while reducing EM.

Response: Thank you for the suggestion. We measured held-out accuracy on the GSM8K and AQuA-RAT test sets at installation and after continuation to 2.0×, for every cell and installed arm (seed 0, Regime 2, greedy decoding, exact match on the final answer). Values pool the 45 cell-arm pairs; brackets give 95% intervals for the change from installation, resampling cells:

Held-out test accuracyInstallation (t=0)SFT at 2.0×RL at 2.0×
GSM8K, after continuation on U1 (GSM8K)85.5%80.5%, −5.0 pp [−7.3, −2.1]89.6%, +4.1 pp [+1.8, +7.1]
AQuA-RAT, after continuation on U3 (AQuA-RAT)66.4%62.3%, −4.2 pp [−8.8, +0.6]64.3%, −2.1 pp [−9.0, +5.4]

The instruction-tuned backbones already solve most of these problems at installation. SFT on the training solutions lowers GSM8K test accuracy by 5.0 points and leaves AQuA-RAT accuracy within its interval, while RL raises GSM8K accuracy by 4.1 points. The models do learn the corpora's answer format: after SFT on AQuA-RAT, the share of answers that state a final option letter rises from 83.9% to 95.0%. The EM reductions therefore happen in models that keep their task ability, and under RL they come with higher GSM8K accuracy. We have added these results, with per-backbone values, to Appendix F.4 (Tables 35 and 36).


Reviewer_R1UzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors note that existing work largely evaluates emergent misalignment defenses immediately after EM finetuning. They argue that this is unrealistic and propose to instead measure how EM persists through continued finetuning. They study matched pairs of (defended, undefended) models over continued benign finetuning. Among other things, they find that EM naturally declines over the course of benign finetuning. This work is a useful case study in how measurements at a single checkpoint can mischaracterise the effectiveness of training interventions.

Critical strengths

I think the point made by the paper is sensible. The results presented give reasonably good evidence that more care is required when interpreting measurements made at a single checkpoint; instead longitudinal scores should be considered.

Critical weaknesses

None

Questions for authors

None

3 comments
AuthorsOpenReview ↗

Response to Reviewer R1Uz

Response to Reviewer R1Uz

We appreciate the positive assessment, and your summary captures the point we most wanted the paper to make: a measurement at a single checkpoint can mischaracterize a training intervention, so EM defenses should be judged by their longitudinal scores. We made the following revisions with other reviewer's feedback in mind, and we hope it strengthens the paper while making it easier to follow, with all changes marked in blue in the updated PDF.

  • With three RL seeds on all three backbones (2,280 trajectories in the full panel), every RL conclusion holds on each seed separately: IP, PV and KL resolve above the undefended model in 0 of 972 comparisons on every seed, FI is the only defense that does, and no defended rise resolves on any seed. Pooling the seeds sharpens the arithmetic result, with IP, PV and KL resolving below the undefended model in 136 and 161 of 162 comparisons on U1 and U3, against 128 and 137 at one seed (Section 6; Appendix A.7).
  • The difference-in-differences readout puts a number on the paper's central point (Appendix C.3). Over continued training the defense advantage shrinks by 10.3 to 19.2 pp, and most of that comes from the undefended model, whose Broad-EM falls by 12.4 to 19.3 pp while the defended models move by −4.0 to +2.3 pp. A reading of the defended model alone, at either checkpoint, would miss this. On the arithmetic streams the remaining advantage is still resolved: all four defenses keep 19% to 36% of their installation advantage under SFT, and IP, PV and KL keep 13% to 15% under RL.
  • The capability evaluation shows that these trajectories describe models that still do their tasks (Appendix F.4). After arithmetic SFT, reading comprehension rises slightly and arithmetic accuracy falls by 2.9 to 5.0 points, and RL raises GSM8K accuracy by about four points.
  • Section 3.1 now states the threat model, and Appendices A.4 and B.1 explain the 2.0× horizon and the eight-question prompt bank.

Once again, we appreciate the positive feedback, and please feel free to let us know if you have any other questions/reservations we can address.

AuthorsOpenReview ↗

Follow-up on previous discussion

Thank you again for your feedback on the longitudinal evaluation. We’ve summarized the revisions in our reply above, including the new analysis of how the defense advantage changes during continued training. If you have any thoughts on the revision, we’d be glad to hear them.

Thank you again for your feedback on the longitudinal evaluation. As the discussion wraps up, we’d be happy to clarify any part of the revision you’d like us to explain in more detail.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Reply to the follow-up on Weakness 1: Why is incoherence higher after arithmetic SFT?

Thank you for the follow-up. We appreciate that the capability results helped clarify this concern. Our working hypothesis is that supervised learning on short mathematical solution traces changes how the model responds to open-ended questions, rather than causing a general collapse of capability. Three observations motivate this interpretation, although they do not isolate a single causal mechanism.

Incoherent responses at 2.0× (Regime 2)U1 (GSM8K)U3 (AQuA-RAT)R (related prose)U2 (unrelated prose)
Benign-only SFT, without risky fine-tuning19.5%19.0%0.7%1.3%
Reward-model RL, five installed arms, three seeds0.72%0.46%0.09%0.08%
Incoherent SFT outputs containing recognizable mathematical-solution formatting8%5%N/AN/A

First, the effect is associated with arithmetic continuation even without risky fine-tuning. The benign-only control starts from the original instruction-tuned backbone and receives no risky installation. Nevertheless, approximately 19% of its free-form responses are judged incoherent after SFT on either arithmetic stream, compared with 0.7% and 1.3% after the two prose streams. Thus, the elevated incoherence is not confined to models that previously underwent risky fine-tuning.

Second, the effect appears broader than simply reproducing mathematical answer formats. GSM8K and AQuA-RAT contain compact, step-by-step solutions, whereas the eight Broad-EM questions request open-ended responses. Only 8% and 5% of the incoherent arithmetic-SFT outputs contain recognizable solution-format markers, such as equations, answer letters, or explicit answer phrases. This suggests that the change concerns free-form response behavior rather than merely copying the surface format of the training solutions. Importantly, the held-out capability results in Appendix F.4 show that these checkpoints retain substantial reading-comprehension and mathematical ability.

Third, the continuation objective matters. When the same arithmetic streams are used for reward-model RL rather than teacher-forced SFT, fewer than 1% of endpoint responses are incoherent. This is consistent with the possibility that fitting supplied solution text affects free-form behavior differently from optimizing the model's own generated responses. Since the objectives also differ in their update signals and schedules, this contrast is suggestive rather than a controlled identification of imitation as the cause.

Taken together, these observations support a distribution- and objective-dependent explanation for the incoherence. They do not establish a unique mechanism, but they do distinguish the observed free-form response degradation from a general loss of task competence. This is also why we report response coverage and coherence alongside Broad-EM rather than treating a low EM rate as sufficient evidence of safety.

AuthorsOpenReview ↗

Reply to the follow-up on Questions 1 to 3

Thank you for these thoughtful follow-up questions. They help clarify both the measurement being compared and the practical conclusions supported by the benchmark.

Q1. Why is incoherence elevated after mathematical-reasoning continuation?

We discuss this in detail in our reply to the follow-up on Weakness 1. In brief, our working hypothesis is that teacher-forced SFT on compact mathematical solutions changes the model's free-form response behavior. The effect also appears in benign-only models (approximately 19% incoherent on arithmetic streams versus roughly 1% on prose), while reward-model RL on the same arithmetic streams produces less than 1% incoherence. These observations motivate the hypothesis but do not isolate its mechanism.

Q2. Would a version of Figure 2 showing the trained risky behavior help establish whether the defenses still matter?

We agree, and have added a third row to Figure 2 (panels i–l) showing the in-domain risky-behavior trajectories alongside the original Broad-EM panels. The distinction matters because continued training can lower Broad-EM in the undefended reference as well as in the defended models. On the prose streams, this brings the Broad-EM curves close together, even when the narrow behavior learned during risky fine-tuning remains observable.

The added row makes that distinction concrete. At 2.0× after SFT on U1 (GSM8K), the measured risky-response rates are 56.3% for the undefended reference, 17.3% for IP, 31.6% for PV, 10.0% for KL, and 62.4% for FI. On U2, where the Broad-EM endpoints are already close to the floor, the corresponding in-domain rates remain 20.7% for the undefended reference versus 1.9% for IP and 2.3% for KL. These are descriptive outcomes under the separate risky-behavior probe, not an alternative ranking of defenses.

The additional panels therefore clarify why a low Broad-EM endpoint should not be read as evidence that the installation history no longer matters. The probe measures responses to the trained risky task using its own evaluation protocol and complements, rather than replaces, the main untriggered Broad-EM metric.

Q3. Does any defense stand out, or is effectiveness too dependent on the continuation stream to compare?

This is an important distinction. The tested defenses do exhibit meaningful differences, but the evidence does not establish a single ordering that generalizes across continuation objectives and streams. IP changes the training context, FI changes the data mixture, PV intervenes on activations, and KL modifies the training loss. Moreover, the evaluated arms are specified implementations, with differences from some published procedures documented in Table 1 and Appendix A.3. Our comparison is designed to characterize each installed defense against its matched undefended history under subsequent benign adaptation, not to rank the underlying method families universally.

That distinction still permits practical conclusions tied to the continuation setting:

  • For reward-model RL, the tested FI variant warrants particular re-evaluation. It is the only defense with resolved reversals relative to the undefended reference at some RL checkpoints. On the two prose streams, its pooled endpoint gap estimates are negative (−0.32 and −0.46 pp), although these endpoint intervals include zero. The result motivates checking FI after RL continuation rather than assuming that its installation-time advantage persists.
  • For SFT, a positive relative advantage need not imply unchanged protection. On the arithmetic streams, IP and KL retain positive matched gaps even though their own absolute Broad-EM can rise above installation. For example, the pooled defended changes on U1 are +1.9 pp for IP and +0.6 pp for KL (Appendix C.3, Table 24). Tracking only the defended–undefended gap would miss this deterioration.

These are not interchangeable failure patterns. One concerns a possible reversal relative to the unsafe reference under RL; the other concerns absolute deterioration despite a surviving relative advantage under SFT. Both are visible in the primary Broad-EM trajectories, without relying on the separate in-domain probe to rank defenses.

The main conclusion is therefore more specific than simply saying that results vary. What remains of an installed defense depends on the continuation objective and data distribution, and a low final EM score can arise from changes in the defended model, the undefended reference, or both. This is precisely the distinction that matched longitudinal evaluation makes observable and that an installation-only or endpoint-only comparison would miss.

AuthorsOpenReview ↗

Reply to the follow-up on Weakness 2: Reading Table 5 and the exposure horizon

Thank you for the clarification. We agree that Table 5 could be easier to interpret, and we appreciate that the explicit threat model helped explain the intended comparison.

Reading Table 5. The table combines two different quantities. Its unsafe and benign-only rows report absolute Broad-EM rates, whereas the defense rows report the advantage relative to the matched undefended reference, in percentage points:

Δ(t)=EU(t)−ED(t).\Delta(t)=E_U(t)-E_D(t).

A positive value means the defended model has lower measured EM than its matched undefended reference; a negative value means the opposite. For example, at 2.0× exposure:

Continuation settingUndefended Broad-EMFI advantageFI Broad-EM
SFT on U1 (GSM8K)7.04%+4.28 pp2.76%
RL on R (related prose)0.28%−0.32 pp0.60%

Thus, the table does not identify filtered interleaving (FI) as the best defense. FI has a positive gap on U1 under SFT, but a negative pooled point estimate on related prose under RL. The latter endpoint estimate is not itself a resolved reversal; the longitudinal RL analysis separately identifies resolved FI reversals at evaluated checkpoints, with no such resolved reversals for IP, PV, or KL (Section 6.1). We have separated the revised Table 5 into explicitly labeled blocks for absolute Broad-EM rates and defense advantages to prevent these quantities from being confused.

Training for multiple epochs. There is nothing inherently wrong with additional epochs. Our 2.0× horizon is a comparison boundary, not a claim that training should stop there or that the trajectories have converged. It was fixed before the continuation runs as the largest common round budget that permits every SFT stream to be consumed without repeating examples. GSM8K contains approximately 2.11× to 3.50× the installation token exposure across cells, while the smallest generated-prose corpus contains 2.21× to 2.24×; AQuA-RAT is substantially larger (Appendix A.4).

Beyond this common horizon, the smaller streams would begin repeating examples while the larger ones could still supply unseen data. The comparison would then conflate continuation content with repetition frequency. Multi-epoch continuation is a valid additional training regime, but it answers a different question and requires its own controlled accounting. In particular, the arithmetic trajectories have not reached an established terminal state at 2.0×, so the present results do not support extrapolating their eventual EM rates or defense gaps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.