- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies what happens when a post-training update from one training run is applied to a model from another run. The authors train qwen3 lineages that differ only in their seed (SFT then RLVR on math) and carry the RLVR or OPD update of one lineage to checkpoints of another lineage at different points of its own RLVR training by addition, averaging, TIES merging, or on-policy distillation. They explain the results by splitting problems into those only the teacher solves and those only the recipient solves, together with a 1st-order argument that an update's effect should shrink as the recipient trains. The main findings are that updates carry their teacher's behavior, that the same update helps an early recipient and hurts a later one, and that averaging beats distillation from a same-size peer on mature recipients.
Critical strengths
-
The experiments are well-controlled: each claim comes with a control that rules out a simpler explanation: random sign flips and negation show that the direction of the update matters, smaller doses show that the late loss is not just from a step that is too large, extending the recipient's own update shows that the loss is not simply from moving the weights, and distilling from the recipient's own checkpoint accounts for drift from training. Splitting problems into teacher-only and recipient-only groups makes each result easy to understand and the 1.7 and 8B results are consistent.
-
The early transfer result is useful. Adding a peer's update to another run's SFT checkpoint gives good gains, and the problems it gains and loses overlap with those of students distilled from the same teacher.
Critical weaknesses
-
The main finding is largely expected, and it is confounded with teacher strength. A model that has finished RL on the same data has less to gain from a peer trained the same way and more to lose, so it is not surprising that a peer's update helps early and hurts late. The paper also cannot separate the recipient's stage from how much better the teacher is than the recipient.. A simpler reading of the results is that "taking an update from a model that is not much better than you does not help". The larger-teacher experiment points this way, since distillation from the 1.7B teacher matches or beats averaging in all stages.
-
The evidence is narrow for several results/conclusions: the lineages differ only in their seed and train on the same math data, which the paper notes leaves open how updates from different data or tasks compose, despite that being the more common model-merging setup. At 0.6B the models solve <20% in-distribution problems, so the teacher-only and recipient-only groups are small nominally, the stage results come from one pair of lineages, and the intervals cover problem sampling but not training randomness.
Questions for authors
- Can you separate the recipient's stage from the teacher's margin, for example with teachers whose margin over the recipient is held fixed across stages? It would be interesting to see if the stage effect remains at a fixed (or bounded) margin.
- How stable are the stage results across more lineage pairs and seeds, given that the intervals do not include training randomness?
- Can you measure the recipient's pass probabilities or pass@k directly, to check if RL sharpening explains the fading benefit?