acceptodds
Under review as a conference paper at ICLR 2027

Off-Policy Merging Beats On-Policy Self-Distillation for Continual Learning

Abstract

A long-standing goal of AI is a model that can continually learn. On heavily post-trained models, direct supervised finetuning (SFT) on new data often forgets and fails to generalize, so it is commonly believed that on-policy training is required. In practice, however, new data to learn from are usually off-policy. On-policy self-distillation (OPSD) tries to bridge this gap by converting off-policy data into on-policy learning signal, but it has been shown to cause reasoning collapse. In this paper, we show that embarrassingly simple off-policy training beats OPSD for continual learning. We show that a major part of SFT's problem lies in its weight damage, and what's worse, the same SFT produces progressively more damage when applied to checkpoints later along the model's training trajectory. To reduce the damage, we show complimentary benefits of (1) scaling the weight update, (2) learning the weight update from an earlier checkpoint (“donor model”) and applying it to the final model, and (3) protecting sensitive weights when needed. Surprisingly, we find that unfinished pretraining checkpoints are the best as the donor model. Across injecting new knowledge, learning from expert traces, and self-improvement, our off-policy training Pareto-dominates SFT and OPSD in injection and retention/generalization while avoiding expensive on-policy sampling. Our work challenges the necessity of on-policy training for continual learning for RL-trained models and points to a better way to continually learn from off-policy data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
continual learning, model merging, task arithmetic

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fqAKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to adapt a post-trained language model to new offline data while preserving its existing capabilities. It proposes grafting: fine-tuning an earlier donor checkpoint, extracting and scaling the resulting parameter delta, and transferring it to the post-trained model. A mask derived from the retention Fisher can additionally remove update directions estimated to interfere with existing behavior. Experiments cover learning from mathematical demonstrations, self-improvement, and synthetic knowledge injection. The paper reports improved trade-offs between new-task performance and retained performance relative to SFT and OPSD, with additional analyses of donor selection and Fisher masking.

Critical strengths
  1. The evaluation explicitly considers both acquisition and retention. The paper does not assess adaptation solely through performance on the new data. It also applies merging sweeps to SFT and OPSD, rather than comparing grafting only against their unmerged checkpoints. This makes the comparison more informative and provides useful evidence about how much a simple change in update application can affect the apparent advantage of a training method.

  2. The donor and masking analyses contain useful empirical observations.
    The OLMo experiment suggests that some intermediate pretraining checkpoints can be better donors than the final base model. The knowledge-injection ablation also shows that targeted masking can improve upon uniformly shrinking the update. These observations are potentially useful for practitioners who have access to compatible checkpoints.

Critical weaknesses

Critical Weaknesses

1. The additional contribution over existing delta-transfer methods is limited and not sufficiently isolated.

The central operation of learning an update on a base model and transferring it to its post-trained counterpart is already present in Shadow-FT [1]. Shadow-FT also explicitly studies scaling the transferred delta in Section 5.6 and Appendix D.6. Consequently, with the official base model as donor and no masking, Equation (1) reduces to an existing method rather than introducing a new adaptation mechanism. The manuscript acknowledges this connection in Sections 3 and 7; the substantive additions are therefore the exploration of earlier donors and the retention Fisher mask.

Figures 3 and 6 provide some evidence for these additions, and the unmasked donor-update sweep in Figure 6 is a relevant baseline. However, most experiments use the official base model, while the clearest isolation of the masking benefit is presented in the knowledge-injection setting. It remains unclear how much of the headline improvement across settings comes from the proposed additions, rather than from the already established donor-transfer recipe.

The Fisher-merging comparison is relevant, but the experiments do not establish how the method compares with other approaches designed to reduce interference during merging, such as TIES-Merging [2], under an appropriately matched checkpoint configuration. My concern is therefore not that model merging is entirely absent from the baselines, but that the comparison within the most relevant methodological family is incomplete. Beating direct SFT and OPSD does not, by itself, establish a sufficiently distinct contribution over prior delta-transfer and merging methods.

2. The central OPSD comparison needs clarification, and the continual-learning evidence remains narrow.

There is an important ambiguity in the definition of the OPSD baseline. Section 2 writes the rollout distribution as y~∼πθ(⋅∣x,y)\tilde y \sim \pi_\theta(\cdot \mid x,y), while the student distribution inside the loss conditions on xx and the generated prefix, without yy. Appendix C likewise states that the model samples its response conditioned on a demonstration. These descriptions appear inconsistent with sampling from the unprivileged student policy whose behavior is being optimized.

In the standard OPSD formulation, the student generates from the question alone, and the reference solution is supplied only to the teacher when evaluating the student's trajectory. This may simply be a documentation error, but if the implementation follows the manuscript literally, the samples would not be on-policy for the student distribution used in the loss. Because the paper's central comparison depends on OPSD, this distinction needs to be resolved before the results can be interpreted confidently.

Even assuming the baseline is implemented correctly, the experiments concern individual adaptation stages, with retained performance measured by three competition-math benchmarks. They do not demonstrate retention over a sequence of distinct task acquisitions or across a broad set of capabilities. The results can provide a counterexample to the necessity of on-policy learning in the tested settings, but they offer more limited evidence for continual learning in the broader sense. Combined with the overlap with existing methods, this leaves me unconvinced that the paper establishes a sufficiently substantial advance.

References

[1] Taiqiang Wu, et al. Shadow-FT: Tuning Instruct Model via Training on Paired Base Model.

[2] Prateek Yadav, et al. TIES-Merging: Resolving Interference When Merging Models.

Questions for authors

Questions For Authors

1. Clarification of the OPSD implementation.

In Section 2 (p. 3), rollouts are sampled from πθ(⋅∣x,y)\pi_\theta(\cdot \mid x,y), whereas the student distribution in the loss conditions on xx and the generated prefix, without the reference solution yy. Appendix C (p. 15) likewise states that the model samples its response conditioned on a demonstration. Taken literally, these descriptions imply that rollouts are generated under a different conditioning context from the student policy being optimized. The concern is not that the teacher sees the reference solution, but that the rollout policy apparently sees it as well.

2. Lack of Novelty

Which results in Figures 4 and 5 use a nonzero masking ratio? For experiments using the official base donor, how much improvement remains over the same donor-transfer method with a tuned scaling factor but no mask? In particular, Section 3 acknowledges that learning a delta on a base model and transferring it to a post-trained model follows existing approaches. do any of the main grafting results reduce to scaled Shadow-FT [1]? This distinction is important for separating the benefits of the proposed additions from those of the inherited recipe.

3. Evidence against relevant merging alternatives.

Can the authors provide a comparison with an interference-aware method such as TIES-Merging [2], using compatible checkpoints, the same learned updates, and comparable tuning budgets? Alternatively, what specific incompatibility prevents such a comparison in this setting? This would help determine whether the proposed mask improves on existing ways of handling interference, rather than only on unmasked transfer and the particular Fisher-merging baseline.

References

[1] Taiqiang Wu, et al. Shadow-FT: Tuning Instruct Model via Training on Paired Base Model.

[2] Prateek Yadav, et al. TIES-Merging: Resolving Interference When Merging Models.

2 comments
AuthorsOpenReview ↗

Rebuttal part 1/2

Thank you for your careful and detailed review. We address your comments and concerns below.

The additional contribution over existing delta-transfer methods is limited and not sufficiently isolated.

The central operation of learning an update on a base model and transferring it to its post-trained counterpart is already present in Shadow-FT [1]. Shadow-FT also explicitly studies scaling the transferred delta in Section 5.6 and Appendix D.6. Consequently, with the official base model as donor and no masking, Equation (1) reduces to an existing method rather than introducing a new adaptation mechanism. The manuscript acknowledges this connection in Sections 3 and 7; the substantive additions are therefore the exploration of earlier donors and the retention Fisher mask.

Figures 3 and 6 provide some evidence for these additions, and the unmasked donor-update sweep in Figure 6 is a relevant baseline. However, most experiments use the official base model, while the clearest isolation of the masking benefit is presented in the knowledge-injection setting.

The Fisher-merging comparison is relevant, but the experiments do not establish how the method compares with other approaches designed to reduce interference during merging, such as TIES-Merging [2], under an appropriately matched checkpoint configuration.

We would like to clarify that Shadow-FT is not specially designed for or evaluated on continual learning, which motivates our weight delta scaling and retention-aware masking that targets the failure cases of direct weight transfer in the continual learning setting.

Here we provide new experimental results on the masking effect. Besides accuracy improvement, we also see a benefit of masking to prevent degeneration (shown as “finished within 32K” - the unmasked version in most cases cannot stop generating). For knowledge injection, K-FAC masking preserves almost complete acquisition, while raising old-task accuracy from 31.7% to 39.0% and the “finished within 32K” rate from 34% to 67%. Thus, the mask preserves acquisition while improving retention.

New-task accuracyOld-task accuracyFinished within 32K
Post-trained model0.0010.36497%
Unmasked0.9960.31734%
K-FAC mask, ρ = 0.0010.9950.39067%

The math experiment shows the same benefit of protection. The K-FAC mask with ρ = 0.01 improves/retains both new- and old-task accuracy over unmasked transfer, while increasing termination from 35% to 88%.

New pass@1Old pass@1New pass@8Old pass@8Finished within 32K
Post-trained model0.0270.3640.1170.60397%
Unmasked0.0430.2810.1170.45635%
K-FAC mask, ρ = 0.010.0380.4110.1890.68988%

Besides the above differences and improvements, we also provide more analysis on donor model selection, a wider range of tasks (reasoning + knowledge), and off-policy data sources (from experts and from self-synthesis with STaR and Pedagorical RL).

  1. The central OPSD comparison needs clarification, and the continual-learning evidence remains narrow.

There is an important ambiguity in the definition of the OPSD baseline. Section 2 writes the rollout distribution as y~∼πθ(⋅∣x,y)\tilde{y}\sim\pi_\theta(\cdot\mid x,y), while the student distribution inside the loss conditions on x and the generated prefix, without y.

Even assuming the baseline is implemented correctly, the experiments concern individual adaptation stages, with retained performance measured by three competition-math benchmarks. They do not demonstrate retention over a sequence of distinct task acquisitions or across a broad set of capabilities. Combined with the overlap with existing methods, this leaves me unconvinced that the paper establishes a sufficiently substantial advance.

Good catch on the typo! We have verified that the OPSD implementation used for the reported experiments follows the standard conditioning: the student generates from the question alone, and the reference solution is supplied only to the teacher. We will correct the rollout notation in §2 and Appendix C.

We view continual learning for modern LLMs more broadly than a sequence of disjoint tasks. A post-trained model is already a strong generalist, and a common reason to adapt its weights is to specialize it on a new task or body of knowledge while preserving its existing capabilities. Our acquisition-retention setting directly studies this practical requirement. We choose math as the retention task because the Qwen models in the thinking mode tend to forget math a lot when doing direct finetuning. Given the limited compute budget during rebuttal, we will additionally evaluate retention on GPQA in the final version.

AuthorsOpenReview ↗

Rebuttal part 2/2

Clarification of the OPSD implementation.

Please see our clarification of the OPSD implementation above.

Which results in Figures 4 and 5 use a nonzero masking ratio? For experiments using the official base donor, how much improvement remains over the same donor-transfer method with a tuned scaling factor but no mask? In particular, Section 3 acknowledges that learning a delta on a base model and transferring it to a post-trained model follows existing approaches. do any of the main grafting results reduce to scaled Shadow-FT [1]? This distinction is important for separating the benefits of the proposed additions from those of the inherited recipe.

All grafting results in Figures 4 and 5 use a nonzero masking ratio. These results therefore incorporate retention-aware protection beyond scaled paired-base transfer.

Please see our response to your first question for novelty.

Evidence against relevant merging alternatives.

Can the authors provide a comparison with an interference-aware method such as TIES-Merging [2], using compatible checkpoints, the same learned updates, and comparable tuning budgets? Alternatively, what specific incompatibility prevents such a comparison in this setting? This would help determine whether the proposed mask improves on existing ways of handling interference, rather than only on unmasked transfer and the particular Fisher-merging baseline.

Our paper already includes a Fisher-merging baseline (Figure 6), which was ineffective in our experiments. Together with the gains from retention-aware masking reported above, this suggests that the sparsity introduced by explicitly zeroing selected update components may be important beyond Fisher-based reweighting alone.

We also run new experiments with TIES-Merging [2] using the same learned knowledge update as grafting. To use TIES-Merging, we define task vectors: t_post = θpost − θbase and t_knowledge = θdonor-ft − θbase.

For TIES, k denotes the percentage of entries retained in each task vector, selected by absolute magnitude. TIES takes the sign of the sum at each coordinate and averages the surviving entries that agree with that sign, then adds the merged vector to the base model. Here is TIES-Merging’s performance:

MethodRetained fraction kHeld-out accuracy
Post-trained model—0.001
Grafting (full knowledge update)—0.995
TIES1%0.000
TIES5%0.078
TIES10%0.108
TIES20% (original default)0.193
TIES30%0.097
TIES50%0.000
TIES100%0.000

The best TIES result across the tested retained fractions is 19.3%, compared with 99.5% for grafting using the same donor. The results suggest that sign agreement is not a good heuristics to combine a post-trained weight update and a continually learned weight update, while sparsity and Fisher information need to be considered.


We believe these clarifications and additional experiments address the main concerns raised in your review. Please let us know if any concerns remain or if further clarification would be helpful.

Reviewer_5AMoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Grafting, a simple off-policy method for continual learning. Instead of directly fine-tuning a post-trained model, it finetunes an earlier donor checkpoint and transfers the resulting weight delta to the post-trained model. The update is further scaled and optionally masked using Fisher information to reduce interference with existing capabilities. The method improves new task learning and old task retention compared with SFT and on-policy self-distillation.

Critical strengths
  1. The paper addresses an important problem in continual learning, showing that off-policy training can be effective without relying on expensive on-policy sampling.

  2. The proposed method is simple and intuitive. The observation that earlier checkpoints can produce less interfering updates is interesting and provides useful insights.

  3. The experimental results are encouraging across different model sizes and learning settings. In particular, Grafting achieves a better trade-off between new task performance and old task retention.

Critical weaknesses
  1. The first concern is novelty. Training on a base model and transferring the weight delta to a post-trained model has already been explored in prior works such as Shadow-FT. What is the key novelty compared with these methods?

  2. The evaluation of old-task retention is relatively limited. Across the experiments, old tasks are evaluated only on math benchmarks, while the new tasks in the mathematical reasoning experiments are also math problems. Does the method mainly improve generalization within the same domain rather than prevent catastrophic forgetting?

  3. The paper mainly evaluates a single round of learning new capabilities. However, continual learning typically requires models to acquire multiple tasks sequentially. Does repeatedly applying Grafting accumulate interference and eventually degrade existing capabilities?

Questions for authors

How does Grafting perform when multiple new tasks are learned sequentially? Does interference accumulate across successive weight updates?

2 comments

Thank you for your thoughtful review. We address your comments and concerns below.

The first concern is novelty. Training on a base model and transferring the weight delta to a post-trained model has already been explored in prior works such as Shadow-FT. What is the key novelty compared with these methods?

We would like to clarify that Shadow-FT is not specially designed for or evaluated on continual learning, which motivates our weight delta scaling and retention-aware masking that targets the failure cases of direct weight transfer in the continual learning setting.

Here we provide new experimental results on the masking effect. Besides accuracy improvement, we also see a benefit of masking to prevent degeneration (shown as “finished within 32K” - the unmasked version in most cases cannot stop generating). For knowledge injection, K-FAC masking at ρ = 0.001 preserves almost complete acquisition, while raising old-task accuracy from 31.7% to 39.0% and the “finished within 32K” rate from 34% to 67%. Thus, the mask preserves acquisition while improving retention.

New-task accuracyOld-task accuracyFinished within 32K
Post-trained model0.0010.36497%
Unmasked0.9960.31734%
K-FAC mask, ρ = 0.0010.9950.39067%

The math experiment shows the same benefit of protection. The K-FAC mask with ρ = 0.01 improves/retains both new- and old-task accuracy over unmasked transfer, while increasing termination from 35% to 88%.

New pass@1Old pass@1New pass@8Old pass@8Finished within 32K
Post-trained model0.0270.3640.1170.60397%
Unmasked0.0430.2810.1170.45635%
K-FAC mask, ρ = 0.010.0380.4110.1890.68988%

Besides the above differences and improvements, we also provide more analysis on donor model selection, a wider range of tasks (reasoning + knowledge), and off-policy data sources (from experts and from self-synthesis with STaR and Pedagorical RL).

The evaluation of old-task retention is relatively limited. Across the experiments, old tasks are evaluated only on math benchmarks, while the new tasks in the mathematical reasoning experiments are also math problems. Does the method mainly improve generalization within the same domain rather than prevent catastrophic forgetting?

We think for math the retention/forgetting is specifically for easy/hard problems. The old-task benchmarks are mostly easier problems, while the new task is hard problems selected for zero initial success. We also did a topic analysis for each problem in the benchmarks (old task) and held-out hard problems (new task). We see that they have different topic coverage. The held-out hard problems include explicit integration, differentiation, and differential-equation problems. None of the 90 old-task AIME/HMMT questions explicitly asks for those operations, i.e. this is a partial topic shift within mathematics.

Knowledge injection provides a further test across capabilities: the model learns 1,125 newly introduced facts, evaluated on held-out paraphrases, while retention is measured on mathematical reasoning.

Given the limited compute budget during rebuttal, we will additionally evaluate retention on GPQA in the final version.

The paper mainly evaluates a single round of learning new capabilities. However, continual learning typically requires models to acquire multiple tasks sequentially. Does repeatedly applying Grafting accumulate interference and eventually degrade existing capabilities?

We view continual learning for modern LLMs more broadly than a sequence of disjoint tasks. A post-trained model is already a strong generalist, and a common reason to adapt its weights is to specialize it on a new task or body of knowledge while preserving its existing capabilities. Our acquisition-retention setting directly studies this practical requirement.

That said, we expect multiple grafts to be composable. Our method is inspired by task arithmetic, which shows that combining task vectors can improve performance on multiple tasks simultaneously. This provides intuition for combining updates learned from different tasks in our setting.

We will add experiments on multiple-graft composition and retention in the final version. A thorough evaluation is difficult within the limited rebuttal period.

How does Grafting perform when multiple new tasks are learned sequentially? Does interference accumulate across successive weight updates?

Please see our response above.


We believe these clarifications and additional experiments address the main concerns raised in your review. Please let us know if any concerns remain or if further clarification would be helpful.

Thank you for the clarification. I understand the time and compute constraints during the rebuttal period. I remain concerned about whether later grafts preserve what the model learned from earlier ones. If the paper is accepted, I strongly encourage authors to include the results of the planned sequential grafting experiment in the final version. I have no further questions at this point.

Reviewer_rbQhOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes grafting, a simple method for teaching post-trained models new tasks while preserving existing capabilities. The method is based on the model merging is simple and effective. The method works by selecting and fine-tuning the donor checkpoint on a new task, usually a base model. Then computing the weight difference between the two, scaling it and merging it with the target post-trained weights. To prevent potential catastrophic forgetting, they use Fisher information to identify and mask harmful parts of the update. The authors evaluate the method in solving hard math problems and in memorizing synthetic data. They compare the method against other off-policy baselines, SFT and OSPD. Grafting generally achieves a better tradeoff between learning new tasks and retaining or improving performance on old tasks.

Critical strengths
  • The method is simple and effective, and the problem is practically important. I can imagine this method being useful when adding new data or tasks to a model that has already gone through SFT, DPO, or RL. In practice, we want each additional training stage to teach the model something new while preserving the capabilities acquired earlier. The proposed method offers a simple way to address this problem, and the results show a promising tradeoff between learning and retention.

    I particularly appreciate the experiments along OLMo’s training trajectory. They show that learning an update on an earlier checkpoint can be more effective than directly fine-tuning the final post-trained model. This provides useful insight into where to learn new-task updates, with potential applications throughout post-training.

Critical weaknesses
  • Lack of continual learning and on-policy baselines The authors should compare grafting against established continual learning methods, for example SFT + EWC or SFT + KL. This is relevant because existing baselines in the paper do not have any knowledge retention mechanisms, so its impossible to determine whether you can preserve and transfer the performance on the old task. The authors should also include a direct comparison against on-policy baseline, for example GRPO. The paper motivates its method through limitations of on-policy training, yet the on-policy is not included as a baseline. These comparisons are important for understanding when grafting offers a meaningful advantage over existing methods.

  • Narrow evaluation and only 3 models The results are promising, but its hard to tell how broadly the method can be used. The authors should evaluate catastrophic forgetting on a broader range of tasks. The model coverage is also narrow. All evaluated models are dense and at most 8B parameters. Experiments on larger models would strengthen the evidence, for example Olmo-3-32B-Think and Qwen3-30B-A3B would test whether the results extend to bigger scale and sparse models.

Questions for authors
  • What are the limitations and observed failure cases of grafting? When does it perform worse than direct fine-tuning? For example, does performance deteriorate when the donor is too far from the recipient, or when learning the new task requires changing the directions that the mask protects?
  • Why is the advantage of grafting particularly large in non-thinking mode (figure 4c).
  • Why restrict the graft scale to values at most one? The interpolation interpretation motivates this range, but does not rule out useful extrapolation. Have the authors tested values moderately above one?
2 comments
AuthorsOpenReview ↗

Rebuttal part 1/2

Thank you for your thorough and constructive review. We address your comments and concerns below.

Lack of continual learning and on-policy baselines. The authors should compare grafting against established continual learning methods, for example SFT + EWC or SFT + KL. This is relevant because existing baselines in the paper do not have any knowledge retention mechanisms, so its impossible to determine whether you can preserve and transfer the performance on the old task. The authors should also include a direct comparison against on-policy baseline, for example GRPO. The paper motivates its method through limitations of on-policy training, yet the on-policy is not included as a baseline.

We would like to first clarify that our comparisons already include OPSD, an on-policy baseline. Figures 4-6 compare it with off-policy SFT and grafting on the same new-task data. The baselines also include retention mechanisms: SFT and OPSD receive the same merging-scale sweep (§4.2), and all methods in knowledge injection receive the same math replay mixture (§5.1). Figure 2 shows that merging alone substantially improves retention. These controls allow us to compare acquisition-retention frontiers rather than a single fine-tuned checkpoint.

That said, we added a set of GRPO experiments in the Qwen3-1.7B thinking mode setting, for both hard-math and knowledge settings, and observed no improvement. This is expected since (1) we chose hard-math problems (zero initial success) to simulate the cases when the model cannot solve problems initially; (2) the knowledge-injection task asks the model to acquire facts it does not yet know. Successful rollouts are therefore extremely scarce, limiting the useful outcome-reward signal.

We also additionally evaluated SFT + EWC and SFT + retention-KL baselines in the Qwen3-1.7B thinking setting. For each baseline, we swept the regularization coefficient over {0.1, 0.5, 1.0} and report the best-performing configuration. Both baselines use the same retention data as what we used for Fisher estimation.

MethodNew-task accuracyOld-task accuracy
Post-trained model0.0270.364
SFT0.0340.297
SFT + EWC0.0250.332
SFT + retention-KL0.0290.357
Grafting (K-FAC, ρ = 0.01)0.0380.411

Narrow evaluation and only 3 models The results are promising, but its hard to tell how broadly the method can be used. The authors should evaluate catastrophic forgetting on a broader range of tasks. The model coverage is also narrow. All evaluated models are dense and at most 8B parameters. Experiments on larger models would strengthen the evidence, for example Olmo-3-32B-Think and Qwen3-30B-A3B would test whether the results extend to bigger scale and sparse models.

We would like to argue that we have a wide range of evaluation, regarding models and data. We evaluated two model families, Qwen3 and OLMo-3, at 1.7B, 7B, and 8B, in both thinking and non-thinking settings. The new data include expert traces, STaR traces, Pedagogical RL traces, and newly introduced facts. Knowledge injection already tests acquiring a different capability while retaining mathematical reasoning.

The cost of evaluating reasoning models across these sweeps is substantial: the full Qwen3-1.7B thinking sweep alone required approximately 16 H100s for 15 days. Comparably extensive sweeps on much larger reasoning models exceed our current academic compute budget. We will add Qwen3-30B-A3B experiments in the final version to extend the evaluation to a larger MoE model. We will also additionally evaluate retention on GPQA in the final version.

What are the limitations and observed failure cases of grafting? When does it perform worse than direct fine-tuning? For example, does performance deteriorate when the donor is too far from the recipient, or when learning the new task requires changing the directions that the mask protects?

This is a great question! Figure 3 for the choice of donor model suggests a tradeoff between how readily the donor learns the new data and how effectively its update transfers to the recipient. The result motivates selecting the donor rather than assuming that either the final or earliest available checkpoint is optimal.

When could direct fine-tuning be preferable? Our current hypothesis is that the benefit of grafting grows as the new data become more off-policy relative to the post-trained model (based on our theory/toy setting section). With data closer to the post-trained model’s own distribution, direct SFT may become more competitive or preferable. So we view grafting as a tool to deal with off-policy data, instead of an eternal replacement of SFT.

If learning the new task requires changing directions that the mask protects, we can set ρ = 0 and apply the donor update without masking. In our experiments, however, a small but positive ρ improved the acquisition-retention tradeoff.

AuthorsOpenReview ↗

Rebuttal part 2/2

Why is the advantage of grafting particularly large in non-thinking mode (figure 4c).

We believe non-thinking supervision is more off-policy. Intuitively, reasoning patterns are broadly similar across models, whereas their final post-thinking solution formats differ more. Learning these final solutions therefore creates a larger mismatch with the post-trained model’s own distribution.

Why restrict the graft scale to values at most one? The interpolation interpretation motivates this range, but does not rule out useful extrapolation. Have the authors tested values moderately above one?

Currently all the Pareto-optimal points have λ < 1. We also evaluated λ = 1.25 and 1.5, which added no Pareto-optimal points in the tested configurations.


We believe these clarifications and additional experiments address the main concerns raised in your review. Please let us know if any concerns remain or if further clarification would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.