- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper compares 4 different defenses against Emergent Misalignment (EM) during the course of benign continuation training. They find that both SFT and RL reduce EM, RL more than SFT, even without any of these defenses. The defended models have lower rates of misalignment than the undefended model throughout continuation training, although the defended models can sometimes become more misaligned after SFT than before. Continuation training on prose is more effective at reducing EM than training on mathematical reasoning. The training is also more effective at reducing broad EM than the narrow task behavior.
Critical strengths
I like seeing how the defenses co-vary with continuation training. Figure 2 is very clarifying: you can see how much continuation training has a comparable effect to the four defenses.
The choice of the four datasets to split out the effect of topic vs format is also nice. The results ended up splitting quite neatly on prose vs mathematical (although this is still somewhat confounded).
Critical weaknesses
The 27.6-28.1% incoherent outputs from SFT on mathematical reasoning seems very concerning. Are we really lobotomizing the model, because that would drastically change the interpretation of the results? I am particularly suspicious as the strange result of EM rising during benign training was localized to SFT on mathematical reasoning as well. I don’t think that using the conditional EM ameliorates this concern. If ~28% of the responses are incoherent, I would consider the whole checkpoint to be potentially invalid without careful testing.
Why do you stop at 2x exposure? Many of the curves appear monotonic and unsaturated in Figure 2, so it would be good to find out what the terminal state is. In particular, it is unclear to me if the undefended line will just go to zero, in which case it’s not clear why the defenses are even necessary. More generally, I am most interested in comparing the four defenses at the terminal state rather than partway through training. I would be happy to change my mind on this though if you want to outline a concrete threat model.
Questions for authors
- How do you interpret the incoherence and persistent gaps observed on the mathematical reasoning continuation training?
- Is your conclusion that as long as you do benign continuation training on prose, it doesn’t really matter what the topic is or even whether you are using a defense?
- What is your advice to developers? Which of the four defenses should they use? Should they order the training in a certain way to remove any emergent misalignment from risky finetuning with subsequent benign training?
4 comments
Response to Reviewer KwCu, Weakness 2
Weakness 2: Why do you stop at 2x exposure? Many of the curves appear monotonic and unsaturated in Figure 2, so it would be good to find out what the terminal state is. In particular, it is unclear to me if the undefended line will just go to zero, in which case it’s not clear why the defenses are even necessary. More generally, I am most interested in comparing the four defenses at the terminal state rather than partway through training. I would be happy to change my mind on this though if you want to outline a concrete threat model.
Response: Thank you for these questions. We have added an explicit threat model to Section 3.1 (in blue), summarized here.
Threat model. We study the ordinary life cycle of a fine-tuned model. A defense is applied during a narrow risky fine-tune and is usually evaluated where that fine-tune ends (installation). The model is then adapted further on benign data for new tasks or products, by SFT or reward-model RL, and this routine adaptation can change the installed protection or the underlying misalignment. Because it can move the defended and the undefended model alike, the benchmark follows every defended model alongside a matched undefended model with the identical risky fine-tune and continuation.
Exposure horizon. The 2.0× horizon was fixed before any continuation run, from corpus size: it is the largest round budget at which every continuation corpus is used in a single pass under SFT. In assistant-loss tokens, GSM8K holds 2.11× to 3.50× of each cell's installation dose and the smallest topic corpus holds 2.21× to 2.24×, so longer runs would change exposure and data repetition together. We agree that the arithmetic curves are still declining at 2.0×, by about one point per doubling, while prose sits near the floor (Section 6.1):
| Undefended SFT model, Broad-EM (pooled over cells and seeds) | 0.5× | 1.0× | 2.0× |
|---|---|---|---|
| R (topic-related prose) | 0.55% | 0.53% | 0.38% |
| U2 (unrelated prose) | 0.95% | 0.81% | 0.71% |
| U1 (GSM8K) | 9.30% | 8.38% | 7.04% |
| U3 (AQuA-RAT) | 8.42% | 7.30% | 6.02% |
We have added this rationale to Appendix A.4.
When the undefended model reaches the floor. A Broad-EM floor describes the open-ended evaluation, and the trained risky behavior falls more slowly (Section 6.2, Figure 3):
| Undefended SFT model at 2.0×, share of its 0.03× level | R | U2 | U1 | U3 |
|---|---|---|---|---|
| Broad-EM | 3% | 5% | 39% | 38% |
| Trained risky behavior (in-domain probe) | 7% | 27% | 68% | 64% |
An undefended model near the Broad-EM floor can therefore still carry much of its trained risky behavior: on U2, 5% of its Broad-EM remains against 27% of the risky behavior.
Comparing the four defenses. Table 5 reports all four defenses side by side at 2.0×, the end of the declared horizon, with per-cell values in Table 25. Our arms are implementations of the published methods, so we present these as characterizations of each defense against its own matched reference.
Response to Reviewer KwCu, Question 1 - 3
Question 1: How do you interpret the incoherence and persistent gaps observed on the mathematical reasoning continuation training?
Response: Thank you for the question. Our reading separates the two observations, as detailed in our response to Weakness 1. The arithmetic continuation lowers coverage on its own: the benign-only control, which starts from the instruction-tuned backbone, also loses coverage on these streams, and the continued models remain capable on held-out reading comprehension and arithmetic. The persistent gaps follow the risky-training history: the undefended model keeps measurable Broad-EM on the arithmetic streams at 2.0× (Table 5), the defended models stay below it, and the same gaps appear under RL, where endpoint outputs are almost fully coherent.
Question 2: Is your conclusion that as long as you do benign continuation training on prose, it doesn’t really matter what the topic is or even whether you are using a defense?
Response: Thank you for the question. For Broad-EM, the prose topic matters little: both prose streams bring the undefended model close to the floor. Two observations show that the topic and the defense still matter on prose:
| At 2.0× on the prose streams | R (topic-related) | U2 (unrelated) |
|---|---|---|
| Undefended Broad-EM, SFT (Table 5) | 0.38% | 0.71% |
| Trained risky behavior retained, share of its 0.03× level, SFT (Figure 3) | 7% | 27% |
| Defense comparisons resolved in favor of the defense, SFT (Section 6.1) | 2 of 36 | 11 of 36 |
| FI gap under RL, three seeds (Table 5) | −0.32 pp | −0.46 pp |
The topic changes how much of the trained risky behavior remains, so similar Broad-EM endpoints can carry different amounts of retained behavior. The defense also changes the outcome: under SFT, defenses keep resolved advantages in 11 of 36 U2 comparisons, and under RL the filtered-interleaving arm ends above the undefended model on both prose streams.
Question 3: What is your advice to developers? Which of the four defenses should they use? Should they order the training in a certain way to remove any emergent misalignment from risky finetuning with subsequent benign training?
Response: Thank you for the question. Our results support three practical recommendations:
- Evaluate a defense against a matched undefended model throughout continued training, because whether an installed advantage stays visible depends on the continuation data (Section 6.1).
- Report response composition together with the EM rate, because coherence filtering changes some comparisons (Section 6.2).
- After any benign continuation, check the trained risky behavior directly, because Broad-EM can fall while that behavior persists (Weakness 2).
On the choice of defense, Table 5 reports each defense's endpoint against its own matched reference, so a practitioner can match a defense to their continuation setting. For example, under SFT all four keep an arithmetic-stream advantage, while under RL on prose the filtered-interleaving variant we tested ends above the undefended model and IP, PV and KL stay at or below it. On ordering, benign training after a risky fine-tune does lower Broad-EM, and our data also show the trained risky behavior persisting, so we recommend verifying any such reduction with a direct probe of the risky behavior.
Revisions in the updated PDF (in blue)
- An explicit threat model in Section 3.1 (Weakness 2).
- The exposure-horizon rationale in Appendix A.4 (Weakness 2).
- Held-out reading-comprehension and arithmetic accuracy at installation and at 2.0× in Appendix F.4, with a pointer from Section 6.2 (Weakness 1).
- Regime-2 coverage of the undefended model in Appendix E.1 (Weakness 1).
- RL results at three seeds throughout Section 6 and the appendix.
Thanks again for pushing us to give a clearer recommendation. Before the discussion closes, is there anything you’d still like us to clarify?
Response to Reviewer KwCu, Weakness 1
Response to Reviewer KwCu
We thank the reviewer for the close reading and the specific questions. We were glad that Figure 2 made the comparison between continuation training and the four defenses clear, and that the four continuation streams were seen as a useful way to separate topic from format. The concerns about the incoherent arithmetic checkpoints and about where training should stop are fair, and we address them with a capability evaluation, an explicit threat model and the rationale for the exposure horizon. Our responses follow.
Weakness 1: The 27.6-28.1% incoherent outputs from SFT on mathematical reasoning seems very concerning. Are we really lobotomizing the model, because that would drastically change the interpretation of the results? I am particularly suspicious as the strange result of EM rising during benign training was localized to SFT on mathematical reasoning as well. I don’t think that using the conditional EM ameliorates this concern. If ~28% of the responses are incoherent, I would consider the whole checkpoint to be potentially invalid without careful testing.
Response: Thank you for raising this. We agree that a checkpoint whose capability had collapsed would call for a different interpretation, so we tested the arithmetic-continued checkpoints directly.
Capability remains high. We evaluated every SFT checkpoint continued on U1 and U3 on held-out reading comprehension and arithmetic (seed 0, Regime 2, greedy decoding; brackets give 95% intervals for the change from installation, resampling cells):
Reading comprehension rises slightly, while arithmetic accuracy falls by 2.9 to 5.0 points. That is a measurable cost, and every checkpoint remains capable: the lowest single checkpoint still solves 74% of GSM8K, and every SQuAD F1 is at least 80.
The rises consist of coherent, misaligned answers. Our primary rate is all-generation Broad-EM (Section 3.2). Incoherent responses count only toward the denominator: an incoherent response can never count as EM, so incoherence by itself only lowers the rate, and the rises under SFT (Section 6.2) are additional responses that the judge scores as coherent and misaligned. The arithmetic continuation alone produces very few of them: the benign-only control loses coverage on these streams as well, yet its Broad-EM stays near zero (Regime 2; Broad-EM from Table 5, coverage from Appendix E.1):
The persistent gap also holds with coherent outputs. Under RL, 0.72% (U1) and 0.46% (U3) of endpoint responses are incoherent across three seeds, and IP, PV and KL still keep arithmetic gaps between 2.34 and 2.69 pp (Section 6.2; Table 5).
We have added the capability results to Appendix F.4 (Tables 35 and 36), with a pointer from Section 6.2.