acceptodds
Under review as a conference paper at ICLR 2027

The Arithmetic of LLM Post-Training: Merging and Distillation on a Moving Recipient

Abstract

Large language models are post-trained through successive stages such as SFT, RLVR, and OPD, yet it remains unclear how an update learned along one training path composes with a model that has already progressed along another. We study this question on Qwen3 models from 0.6B to 8B parameters by fixing an update learned in one training lineage (the donor) and carrying it, by merging or by distillation, to successive checkpoints of a second lineage (the recipient) that differs only in its random seed. The experiments show three regularities in how a donor update composes with its recipient. (1) Post-training updates are portable: a transferred update preserves the signature of its teacher, the problems it gains and the problems it loses. (2) Transfer is recipient-dependent: the same update improves an early recipient and harms a mature one, whether it was learned by reinforcement learning or by distillation. (3) Averaging and distillation diverge as the recipient matures: weight averaging leads from the stage where direct addition stops transferring, because it preserves more of the recipient's existing capability. A decomposition into problems that only the teacher solves and problems that only the recipient solves explains the second and third regularities: as the recipient advances, there is less to acquire and more to lose, and distillation keeps giving up the recipient's own problems while averaging retains them. These results show that the effect of a post-training update depends jointly on what the update encodes and what the recipient has already learned.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Post-Training, Reinforcement Learning, OPD, Model-Merging

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9emGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what happens when a post-training update from one training run is applied to a model from another run. The authors train qwen3 lineages that differ only in their seed (SFT then RLVR on math) and carry the RLVR or OPD update of one lineage to checkpoints of another lineage at different points of its own RLVR training by addition, averaging, TIES merging, or on-policy distillation. They explain the results by splitting problems into those only the teacher solves and those only the recipient solves, together with a 1st-order argument that an update's effect should shrink as the recipient trains. The main findings are that updates carry their teacher's behavior, that the same update helps an early recipient and hurts a later one, and that averaging beats distillation from a same-size peer on mature recipients.

Critical strengths
  1. The experiments are well-controlled: each claim comes with a control that rules out a simpler explanation: random sign flips and negation show that the direction of the update matters, smaller doses show that the late loss is not just from a step that is too large, extending the recipient's own update shows that the loss is not simply from moving the weights, and distilling from the recipient's own checkpoint accounts for drift from training. Splitting problems into teacher-only and recipient-only groups makes each result easy to understand and the 1.7 and 8B results are consistent.

  2. The early transfer result is useful. Adding a peer's update to another run's SFT checkpoint gives good gains, and the problems it gains and loses overlap with those of students distilled from the same teacher.

Critical weaknesses
  1. The main finding is largely expected, and it is confounded with teacher strength. A model that has finished RL on the same data has less to gain from a peer trained the same way and more to lose, so it is not surprising that a peer's update helps early and hurts late. The paper also cannot separate the recipient's stage from how much better the teacher is than the recipient.. A simpler reading of the results is that "taking an update from a model that is not much better than you does not help". The larger-teacher experiment points this way, since distillation from the 1.7B teacher matches or beats averaging in all stages.

  2. The evidence is narrow for several results/conclusions: the lineages differ only in their seed and train on the same math data, which the paper notes leaves open how updates from different data or tasks compose, despite that being the more common model-merging setup. At 0.6B the models solve <20% in-distribution problems, so the teacher-only and recipient-only groups are small nominally, the stage results come from one pair of lineages, and the intervals cover problem sampling but not training randomness.

Questions for authors
  1. Can you separate the recipient's stage from the teacher's margin, for example with teachers whose margin over the recipient is held fixed across stages? It would be interesting to see if the stage effect remains at a fixed (or bounded) margin.
  2. How stable are the stage results across more lineage pairs and seeds, given that the intervals do not include training randomness?
  3. Can you measure the recipient's pass probabilities or pass@k directly, to check if RL sharpening explains the fading benefit?
Reviewer_ufrzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how RLVR and on-policy distillation (OPD) updates transfer to recipient models at different stages of post-training. Using independently trained Qwen3 model lineages, the authors compare update addition, weight averaging, and distillation across recipient checkpoints. The experiments show that post-training updates are transferable, but their benefits diminish as recipients mature. Weight averaging also tends to outperform distillation from similarly sized teachers on mature recipients, partly by preserving recipient-specific capabilities.

Critical strengths
  1. The paper explores how the effectiveness of post-training updates depends on the recipient's training stage, a relevant issue for model merging and multi-stage post-training.

  2. The authors use controlled training lineages and extensive comparisons involving update scaling, merging strategies, and teacher strength.

  3. The problem-level decomposition into teacher-only and recipient-only successes provides useful insights into capability acquisition and forgetting.

Critical weaknesses
  1. Limited novelty. Model merging, task-vector transfer, and interactions between post-training stages have been extensively studied. The main observations are largely consistent with existing intuitions about diminishing returns and interference between model capabilities. The paper does not clearly establish a fundamentally new mechanism beyond these known phenomena.

  2. Unclear practical implications. Although the empirical observations are interesting, the paper does not demonstrate how they can be used to improve actual post-training pipelines. In particular, it remains unclear whether the suggested merging strategies offer advantages over standard RLVR/OPD training under comparable computational budgets.

  3. All experiments focus on Qwen3 models and mathematical reasoning, with donor and recipient lineages differing only in random seeds. Whether these findings extend to heterogeneous training data, different capabilities, or other model families remains unclear.

Questions for authors
  1. What is the key new insight established by this work beyond existing studies on task arithmetic, model merging/OPD, and post-training?

  2. Can the authors demonstrate a practical post-training strategy motivated by these findings that achieves better final performance under a comparable compute budget? Such evidence would substantially strengthen the contribution and could positively affect my overall assessment.

  3. Do the observed patterns hold when donor and recipient models are trained on different datasets or capabilities, such as mathematics and coding?

Reviewer_SCXSOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies how a fixed post training update interacts with a recipient model at successive stages of its own training. It constructs Qwen3 training lineages that share their base model, data, and protocol but differ in random seed, and transfers RLVR or distillation updates through addition, averaging, TIES merging, and on policy distillation. The main results show that update effects preserve aspects of the teacher's behavior, that addition becomes less beneficial as the recipient advances, and that averaging can outperform distillation from a peer teacher on mature recipients. A decomposition into problems solved only by the teacher, only by the recipient, by both, or by neither explains gains and losses.

Critical strengths
  1. Fixing the donor while moving the recipient makes it possible to observe that the same update changes from helpful to harmful as the receiving model develops. Sharing the base model, prompt order, and training protocol reduces several obvious confounds. The experiments also include smaller update doses, an extension along the recipient's own update, a shortened recipient update, and self distillation controls. These are consequential controls because they test whether the observed behavior merely reflects excessive displacement, a retreat toward SFT, or drift introduced by additional training. The recurrence at 1.7B and 8B makes the finding more useful than an isolated example at 0.6B.

  2. The decomposition by solved problems reveals information that aggregate scores conceal. The separation between acquiring the teacher's problems and forfeiting the recipient's problems gives an interpretable account of the merging and distillation differences. On mature 0.6B recipients, the cross teacher can substantially improve the teacher's exclusive problem set while degrading the recipient's exclusive set, even when its total advantage over self distillation is small.

Critical weaknesses

The main conclusions rely on a single pair of training trajectories at each model scale. The reported bootstrap intervals do not capture variability across training seeds. More independent runs would strengthen the claim that update transferability consistently decreases as the recipient model matures.

Questions for authors
  1. Do the main findings persist across additional independent training trajectories and when using the best stable recipient checkpoints? How much variation is observed across different random seeds?

  2. What empirical evidence supports the theoretical explanation in Equation 6, particularly the claim that the remainder term dominates? Does the same pattern hold for sampled success probabilities rather than only greedy accuracy?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.