- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how the behaviors taught during midtraining (e.g., honesty) remain after further training. The authors do a controlled study using character text with neutral controls matched on corpus size/topic/reasoning markers followed by instruction tuning and RL. More specifically, the authors vary how the corpora are rehearsed during later stages influence behavioral durability. This paper shows that exposing the model to a small fraction of "character text" during SFT preserves the behavior through both SFT and subsequent RL. Additionally, adding a behavior-related loss during GRPO can restore the entire behavior (here honesty) even after the behavior has been fading.
The authors train two OLMo-2-1124-7B models on two corpora: character corpus (honesty behavior) and neutral corpus with the same training procedure. They measure the paired behavioral advantage by taking the difference in score between the two models. The evaluation is conducted on 282 prompts with 4 responses per prompt (evaluated by judged) and standard metrics such as ECE, Brier scores, etc.
In terms of results, without surprise, rehearsal yields an advantage that remains after RL. Interestingly, RL can recover from the lack of advantage if it includes a behavior-related loss. In contrast, alternative interventions such as freezing middle layers or applying KL output anchoring fail to preserve the honesty advantage during SFT. The authors also provide analyses showing how SFT rotates internal representations away from the midtrained behavior unless rehearsal is used. Generally, the results are sound. However, this is unclear whether it holds on other behaviors (safety, non-sycophancy, etc) and also whether the findings also apply to larger model (see 13B significance results).
Overall, this paper is well written and presents a nice controlled study for character training. While the main finding that data rehearsal prevents catastrophic forgetting is somewhat expected, the interaction between midtraining, SFT, and RL restoration is well studied. My main concern is the lack of generalization beyond honesty, the inconclusive scaling experiment at 13B, and whether SFT rehearsal alone would achieve the same effect.
Critical strengths
Rigorous study with in-depth analysis.
Critical weaknesses
Limited generalization and inconclusive scaling.
Questions for authors
See summary.
5 comments
Response to Reviewer 15CP [Part 2]
[Cons 2: Inconclusive scaling at 13B.]
We agree that the original 13B result is not statistically conclusive after correction, despite its positive point estimate. The separate scale tests in Figure 5 report:
| Submitted ternary-RL scale test | Advantage | Corrected -value |
|---|---|---|
| 7B | +0.028 | 0.020 |
| 13B | +0.034 | 0.115 |
The two point estimates are similar, but the evidence does not justify treating the original 13B effect as established. At the same time, one significant estimate and one nonsignificant estimate do not demonstrate that the two model sizes differ. A direct size comparison is more informative than comparing those significance decisions.
Family-controlled size comparison. We additionally compare Llama-2-7b-hf and Llama-2-13b-hf, keeping the model family fixed while changing parameter scale. Each size receives 34 million tokens of character or topic-matched neutral midtraining, replay ratio 0.25, and LoRA rank 64 with scaling 128, followed by the same 30,000-example instruction subset with 3% auxiliary loss-bearing tokens. Ordinary GRPO uses matched reward samples, rollout-group size, KL coefficient, and 40-update schedule. The study has five independent paired seeds at each size, with seed identities matched across sizes.
For each size, we estimate the character-minus-neutral advantage and directly compare those effects. Responses are averaged within prompts; 10,000 hierarchical bootstrap draws resample seeds and prompt identities while preserving matching. The two positive-effect tests and the size contrast form a single Benjamini-Hochberg family at 5% false-discovery rate.
| Measurement | Estimated advantage or contrast | 95% confidence interval | Corrected -value |
|---|---|---|---|
| 7B advantage | +0.029 | [+0.013, +0.045] | 0.018 |
| 13B advantage | +0.033 | [+0.010, +0.056] | 0.024 |
| 13B minus 7B advantage | +0.004 | [−0.024, +0.032] | 0.78 |
Both tested sizes have positive effects under this matched family-specific protocol. However, the direct size contrast is small and its interval includes zero, so these data do not establish either an advantage from scaling up or equivalence between sizes. The new comparison also does not retroactively make the original Figure 5 estimate significant; it is a separate experiment with different model and training controls. Finally, the original 13B activation-patching study has no clean middle-layer peak. A more reliable behavioral effect at 13B therefore does not imply that the 7B restoration pattern generalizes mechanistically.
Response to Reviewer 15CP [Part 3]
[Cons 3: Could SFT character rehearsal alone produce the same result?]
This is the key attribution question. The submitted experiment gives each arm its own midtraining corpus again during rehearsal. That comparison establishes a useful complete training recipe, but it does not independently determine the value of character midtraining when both models receive the same downstream character supervision. A factorial control resolves this confounding by crossing the midtraining corpus with the SFT auxiliary corpus.
Matched downstream-exposure control. Character and topic-matched neutral midtraining each use 34 million tokens with identical replay and optimization settings. During SFT, each initialization receives either character or neutral auxiliary text at an identical 3% token fraction, along with the same 30,000-example Tülu-3 subset. A fifth condition uses the untouched pretrained base with character SFT exposure. The SFT runs use sequence length 2,048, AdamW learning rate , effective batch size 128, weight decay 0.1, 3% warmup, and cosine decay. All conditions then receive matched ordinary GRPO without auxiliary corpus supervision. Five paired seeds and the same hierarchical confidence-interval procedure are used throughout.
| Midtraining corpus | Corpus mixed into SFT | Absolute score after SFT | Absolute score after GRPO |
|---|---|---|---|
| Neutral | Neutral | 0.39 | 0.43 |
| Character | Neutral | 0.38 | 0.41 |
| Neutral | Character | 0.42 | 0.45 |
| Character | Character | 0.47 | 0.50 |
| None (pretrained base) | Character | 0.43 | 0.46 |
| Direct contrast | After SFT, mean [95% CI] | After GRPO, mean [95% CI] |
|---|---|---|
| Character versus neutral midtraining, identical character SFT | +0.050 [+0.026, +0.074] | +0.050 [+0.024, +0.076] |
| Combined recipe versus pretrained base with character SFT | +0.040 [+0.017, +0.063] | +0.040 [+0.014, +0.066] |
| Midtraining × SFT-corpus interaction | +0.060 [+0.030, +0.090] | +0.070 [+0.038, +0.102] |
SFT character exposure alone produces an improvement, but the combined character-midtraining and character-SFT condition remains stronger than either SFT-only comparison. The +0.050 direct upstream contrast after GRPO is estimated under identical character exposure during SFT, so it cannot be explained merely by giving the character-midtrained arm character data later. The positive interaction additionally shows that the benefit of rehearsal depends on the preceding midtraining corpus. This supports an incremental contribution of character midtraining while leaving open whether an identical internal representation survives or is reconstructed.
Matched total character exposure. The factorial fixes later exposure, not the total number of character tokens across stages. We therefore also evaluate three allocations of a fixed 34M-character-token budget. All runs have a 34M-token midtraining slot and a 34M-token auxiliary SFT slot, with unused character positions filled by topic-matched neutral text. The 8.5M pretraining replay tokens, 30,000 instruction examples, stage-wise optimization budget, and ordinary GRPO continuation are matched. The SFT schedule is common to these allocation conditions and differs from the original 3% mixture.
| Character tokens: midtraining / SFT | Post-GRPO score | Difference from all-midtraining [95% CI] |
|---|---|---|
| 34M / 0M | 0.42 | 0.00 (reference) |
| 17M / 17M | 0.50 | +0.08 [+0.05, +0.11] |
| 0M / 34M | 0.46 | +0.04 [+0.01, +0.07] |
The split allocation also exceeds the all-SFT allocation by +0.04 [95% CI: +0.01, +0.07]. Thus, the combined recipe has an advantage both when downstream character exposure is held fixed and when total character exposure is held fixed. The result favors distributing character training across the two stages in this setting, without establishing a global optimum over possible schedules.
Response to Reviewer 15CP [Part 1]
We sincerely thank the reviewer for the encouraging and constructive assessment. We appreciate your recognition of the controlled training comparisons and the analysis of how SFT and RL interact with an installed character advantage. Your concerns focus on the scope of the result: whether it generalizes beyond honesty, whether the effect is reliable at larger model sizes, and whether downstream character exposure alone explains the positive endpoint. We address each question with the existing controls and additional matched comparisons.
[Cons 1: Generalization beyond honesty and across model families.]
The submitted paper includes warmth as a second successfully installed disposition and a Qwen2.5-7B SFT extension. These are distinct from the failed installation experiments on harmlessness, concision, and non-sycophancy. The existing positive results are summarized in Figure 5 and Appendix H.1:
Warmth follows the full rehearsal-and-GRPO sequence, extending the observed pattern beyond honesty. Its downstream advantage exceeds the installation advantage, consistent with additional learning during rehearsal, rather than simple preservation of an unchanged score. Qwen provides a positive cross-family SFT result, although its direct rehearsal-versus-freezing comparison is inconclusive (p=0.31). The submitted paper therefore does not establish that rehearsal outperforms freezing for every model family.
Additional Qwen GRPO continuation. To test whether the Qwen SFT effect also remains through RL, we follow its ordinary-SFT and 3%-rehearsed-SFT lineages through ordinary GRPO without auxiliary character supervision. The character and neutral models use the same instruction examples, reward data, and optimizer schedule. GRPO uses learning rate 10−6, KL coefficient 0.001, five rollouts per prompt, batch size 512 prompts, and 40 updates. The comparison uses five paired seeds (17, 29, 43, 71, 101), with the full 282-prompt evaluation and four generations per prompt. Hierarchical confidence intervals resample training seeds and prompt identities, preserving paired comparisons.
The ordinary-SFT lineage remains close to zero, while the rehearsed lineage retains a positive advantage after RL. The direct contrast also favors rehearsal, extending the full downstream training pattern to a second model family. This supports behavioral replication under the tested procedure, not a claim that the underlying representation or causal mechanism is identical across model families.
The unsuccessful installation experiments further limit the scope of the claim. Harmlessness, concision, and non-sycophancy do not acquire the intended behavior under the tested midtraining recipe, even though formatting and lexical controls show that the models can learn from the corpora. These negative cases concern the installation stage; they cannot establish whether a successfully installed advantage for those traits would subsequently persist. The evidence therefore supports selected dispositions and training configurations, not a universally effective character-midtraining method.