- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to adapt a post-trained language model to new offline data while preserving its existing capabilities. It proposes grafting: fine-tuning an earlier donor checkpoint, extracting and scaling the resulting parameter delta, and transferring it to the post-trained model. A mask derived from the retention Fisher can additionally remove update directions estimated to interfere with existing behavior. Experiments cover learning from mathematical demonstrations, self-improvement, and synthetic knowledge injection. The paper reports improved trade-offs between new-task performance and retained performance relative to SFT and OPSD, with additional analyses of donor selection and Fisher masking.
Critical strengths
-
The evaluation explicitly considers both acquisition and retention. The paper does not assess adaptation solely through performance on the new data. It also applies merging sweeps to SFT and OPSD, rather than comparing grafting only against their unmerged checkpoints. This makes the comparison more informative and provides useful evidence about how much a simple change in update application can affect the apparent advantage of a training method.
-
The donor and masking analyses contain useful empirical observations.
The OLMo experiment suggests that some intermediate pretraining checkpoints can be better donors than the final base model. The knowledge-injection ablation also shows that targeted masking can improve upon uniformly shrinking the update. These observations are potentially useful for practitioners who have access to compatible checkpoints.
Critical weaknesses
Critical Weaknesses
1. The additional contribution over existing delta-transfer methods is limited and not sufficiently isolated.
The central operation of learning an update on a base model and transferring it to its post-trained counterpart is already present in Shadow-FT [1]. Shadow-FT also explicitly studies scaling the transferred delta in Section 5.6 and Appendix D.6. Consequently, with the official base model as donor and no masking, Equation (1) reduces to an existing method rather than introducing a new adaptation mechanism. The manuscript acknowledges this connection in Sections 3 and 7; the substantive additions are therefore the exploration of earlier donors and the retention Fisher mask.
Figures 3 and 6 provide some evidence for these additions, and the unmasked donor-update sweep in Figure 6 is a relevant baseline. However, most experiments use the official base model, while the clearest isolation of the masking benefit is presented in the knowledge-injection setting. It remains unclear how much of the headline improvement across settings comes from the proposed additions, rather than from the already established donor-transfer recipe.
The Fisher-merging comparison is relevant, but the experiments do not establish how the method compares with other approaches designed to reduce interference during merging, such as TIES-Merging [2], under an appropriately matched checkpoint configuration. My concern is therefore not that model merging is entirely absent from the baselines, but that the comparison within the most relevant methodological family is incomplete. Beating direct SFT and OPSD does not, by itself, establish a sufficiently distinct contribution over prior delta-transfer and merging methods.
2. The central OPSD comparison needs clarification, and the continual-learning evidence remains narrow.
There is an important ambiguity in the definition of the OPSD baseline. Section 2 writes the rollout distribution as , while the student distribution inside the loss conditions on and the generated prefix, without . Appendix C likewise states that the model samples its response conditioned on a demonstration. These descriptions appear inconsistent with sampling from the unprivileged student policy whose behavior is being optimized.
In the standard OPSD formulation, the student generates from the question alone, and the reference solution is supplied only to the teacher when evaluating the student's trajectory. This may simply be a documentation error, but if the implementation follows the manuscript literally, the samples would not be on-policy for the student distribution used in the loss. Because the paper's central comparison depends on OPSD, this distinction needs to be resolved before the results can be interpreted confidently.
Even assuming the baseline is implemented correctly, the experiments concern individual adaptation stages, with retained performance measured by three competition-math benchmarks. They do not demonstrate retention over a sequence of distinct task acquisitions or across a broad set of capabilities. The results can provide a counterexample to the necessity of on-policy learning in the tested settings, but they offer more limited evidence for continual learning in the broader sense. Combined with the overlap with existing methods, this leaves me unconvinced that the paper establishes a sufficiently substantial advance.
References
[1] Taiqiang Wu, et al. Shadow-FT: Tuning Instruct Model via Training on Paired Base Model.
[2] Prateek Yadav, et al. TIES-Merging: Resolving Interference When Merging Models.
Questions for authors
Questions For Authors
1. Clarification of the OPSD implementation.
In Section 2 (p. 3), rollouts are sampled from , whereas the student distribution in the loss conditions on and the generated prefix, without the reference solution . Appendix C (p. 15) likewise states that the model samples its response conditioned on a demonstration. Taken literally, these descriptions imply that rollouts are generated under a different conditioning context from the student policy being optimized. The concern is not that the teacher sees the reference solution, but that the rollout policy apparently sees it as well.
2. Lack of Novelty
Which results in Figures 4 and 5 use a nonzero masking ratio? For experiments using the official base donor, how much improvement remains over the same donor-transfer method with a tuned scaling factor but no mask? In particular, Section 3 acknowledges that learning a delta on a base model and transferring it to a post-trained model follows existing approaches. do any of the main grafting results reduce to scaled Shadow-FT [1]? This distinction is important for separating the benefits of the proposed additions from those of the inherited recipe.
3. Evidence against relevant merging alternatives.
Can the authors provide a comparison with an interference-aware method such as TIES-Merging [2], using compatible checkpoints, the same learned updates, and comparable tuning budgets? Alternatively, what specific incompatibility prevents such a comparison in this setting? This would help determine whether the proposed mask improves on existing ways of handling interference, rather than only on unmasked transfer and the particular Fisher-merging baseline.
References
[1] Taiqiang Wu, et al. Shadow-FT: Tuning Instruct Model via Training on Paired Base Model.
[2] Prateek Yadav, et al. TIES-Merging: Resolving Interference When Merging Models.
2 comments
Rebuttal part 2/2
Clarification of the OPSD implementation.
Please see our clarification of the OPSD implementation above.
Which results in Figures 4 and 5 use a nonzero masking ratio? For experiments using the official base donor, how much improvement remains over the same donor-transfer method with a tuned scaling factor but no mask? In particular, Section 3 acknowledges that learning a delta on a base model and transferring it to a post-trained model follows existing approaches. do any of the main grafting results reduce to scaled Shadow-FT [1]? This distinction is important for separating the benefits of the proposed additions from those of the inherited recipe.
All grafting results in Figures 4 and 5 use a nonzero masking ratio. These results therefore incorporate retention-aware protection beyond scaled paired-base transfer.
Please see our response to your first question for novelty.
Evidence against relevant merging alternatives.
Can the authors provide a comparison with an interference-aware method such as TIES-Merging [2], using compatible checkpoints, the same learned updates, and comparable tuning budgets? Alternatively, what specific incompatibility prevents such a comparison in this setting? This would help determine whether the proposed mask improves on existing ways of handling interference, rather than only on unmasked transfer and the particular Fisher-merging baseline.
Our paper already includes a Fisher-merging baseline (Figure 6), which was ineffective in our experiments. Together with the gains from retention-aware masking reported above, this suggests that the sparsity introduced by explicitly zeroing selected update components may be important beyond Fisher-based reweighting alone.
We also run new experiments with TIES-Merging [2] using the same learned knowledge update as grafting. To use TIES-Merging, we define task vectors: t_post = θpost − θbase and t_knowledge = θdonor-ft − θbase.
For TIES, k denotes the percentage of entries retained in each task vector, selected by absolute magnitude. TIES takes the sign of the sum at each coordinate and averages the surviving entries that agree with that sign, then adds the merged vector to the base model. Here is TIES-Merging’s performance:
| Method | Retained fraction k | Held-out accuracy |
|---|---|---|
| Post-trained model | — | 0.001 |
| Grafting (full knowledge update) | — | 0.995 |
| TIES | 1% | 0.000 |
| TIES | 5% | 0.078 |
| TIES | 10% | 0.108 |
| TIES | 20% (original default) | 0.193 |
| TIES | 30% | 0.097 |
| TIES | 50% | 0.000 |
| TIES | 100% | 0.000 |
The best TIES result across the tested retained fractions is 19.3%, compared with 99.5% for grafting using the same donor. The results suggest that sign agreement is not a good heuristics to combine a post-trained weight update and a continually learned weight update, while sparsity and Fisher information need to be considered.
We believe these clarifications and additional experiments address the main concerns raised in your review. Please let us know if any concerns remain or if further clarification would be helpful.
Rebuttal part 1/2
Thank you for your careful and detailed review. We address your comments and concerns below.
We would like to clarify that Shadow-FT is not specially designed for or evaluated on continual learning, which motivates our weight delta scaling and retention-aware masking that targets the failure cases of direct weight transfer in the continual learning setting.
Here we provide new experimental results on the masking effect. Besides accuracy improvement, we also see a benefit of masking to prevent degeneration (shown as “finished within 32K” - the unmasked version in most cases cannot stop generating). For knowledge injection, K-FAC masking preserves almost complete acquisition, while raising old-task accuracy from 31.7% to 39.0% and the “finished within 32K” rate from 34% to 67%. Thus, the mask preserves acquisition while improving retention.
The math experiment shows the same benefit of protection. The K-FAC mask with ρ = 0.01 improves/retains both new- and old-task accuracy over unmasked transfer, while increasing termination from 35% to 88%.
Besides the above differences and improvements, we also provide more analysis on donor model selection, a wider range of tasks (reasoning + knowledge), and off-policy data sources (from experts and from self-synthesis with STaR and Pedagorical RL).
Good catch on the typo! We have verified that the OPSD implementation used for the reported experiments follows the standard conditioning: the student generates from the question alone, and the reference solution is supplied only to the teacher. We will correct the rollout notation in §2 and Appendix C.
We view continual learning for modern LLMs more broadly than a sequence of disjoint tasks. A post-trained model is already a strong generalist, and a common reason to adapt its weights is to specialize it on a new task or body of knowledge while preserving its existing capabilities. Our acquisition-retention setting directly studies this practical requirement. We choose math as the retention task because the Qwen models in the thinking mode tend to forget math a lot when doing direct finetuning. Given the limited compute budget during rebuttal, we will additionally evaluate retention on GPQA in the final version.