- Rating
- 1
- Confidence
- 4
- Informativeness
- 1(−1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SURGE, a post-training method for constructing a policy from two checkpoints within the same RL trajectory. It decomposes the anchor’s parameter updates relative to the shared RL initialization, preserves a leading two-sided singular subspace block, and incorporates the donor’s complementary updates. A retained-energy target determines the protected block size without evaluating fusion candidates once the source pair and target are fixed. Experiments on two 1.5B mathematical-reasoning histories and one 7B coding history report improved benchmark-average accuracy and reduced response length relative to the anchors. On two benchmarks, the constructed policies also exceed the best measured native checkpoint.
Critical strengths
The paper presents a practical way to reuse RL checkpoints without further gradient updates or additional deployment-time model overhead. The improvements across mathematical reasoning and coding make checkpoint construction a potentially useful extension of conventional checkpoint selection.
The geometric ablations provide useful evidence that the specific construction matters. Comparisons with scalar interpolation, checkpoint averaging, anchor-only truncation, absolute-weight SVD, and random subspaces help distinguish the reported gains from generic weight perturbation or response shortening.
Critical weaknesses
The connection between update energy and capability preservation remains insufficiently established. Leading singular directions capture large parameter changes, but do not necessarily identify directions important for reward or particular capabilities. The projection identities establish parameter preservation, not behavioral preservation. While the ablations support the selected construction empirically, they do not establish when a donor’s complementary updates should be beneficial. Each main experiment uses one selected checkpoint pair per history, leaving unclear whether the method reliably helps across checkpoints with different capability levels or different patterns of strengths and weaknesses. This limits confidence in the method as a generally applicable checkpoint-construction procedure.
The robustness of the improvement over native checkpoint selection needs stronger validation. The headline gains are modest relative to the evaluation sizes: DeepSeek AIME24 improves from 50.83% to 54.17% with four responses per problem, while OLMo HumanEval+ improves from the best native checkpoint’s 82.8% to 83.7%. The broader mathematical results are encouraging, but repeated decoding rounds do not establish robustness across independently trained histories or checkpoint-pair choices. Furthermore, source selection uses performance measurements, and the 95% energy target was identified in the initial mathematical study. A clearer separation between development choices and held-out assessment is needed to evaluate how reliably the procedure exceeds native checkpoint selection.
The key issue is that I don’t see how this line of research contributes to the broader RL scaling agenda. It just seems like a niche paper that’s hard to find fault with… As evidence of a change in RL scaling, the title and framing currently go beyond what the experiments support. As a hands-on RL researcher, I care less about how compelling the narrative is and more about its impact on actual training.
Questions for authors
Clarification: What operational rule selected each anchor and donor, and which evaluation data informed those choices and the 95% retained-energy target? Were any reported benchmarks held out from these decisions?
Evidence that could change my assessment: Within the same RL history, how consistently does the fixed procedure improve performance across multiple checkpoint pairs with different accuracy gaps and response lengths? Reporting the frequency and magnitude of gains and regressions would help establish when donor complementarity is useful.
Evidence that could change my assessment: Can you provide uncertainty estimates for the improvements over the strongest native checkpoints, preferably using paired resampling at the problem level and additional decoding samples? The key comparison is against the best native checkpoint, which is not always the anchor.
3 comments
Thank you for the detailed response and the additional experiments. I have read the rebuttal and the replies to the other reviewers.
My main concerns remain. The paired intervals for the headline comparisons against the best native checkpoints all include zero ([−14.54, +21.19] pp on AIME24, [−1.98, +3.81] pp on HumanEval+). The response also confirms that the sources and the 95% target were selected on the reported benchmarks, with no held-out set. The additional pairs are appreciated, but three of the eight do not improve and most changes are within ±1 pp, reported without intervals.
I am therefore keeping my score.
We thank the reviewer for the follow-up. We would like to clarify two distinctions regarding the interpretation of our additional evidence.
First, individual headline comparisons and aggregate robustness address different questions. While the per-task confidence intervals on AIME24 and HumanEval+ include zero, the paired nine-benchmark macro-average gains are +2.08 pp on DeepSeek (95% CI [+0.31, +3.94]) and +1.78 pp on Nemotron ([+0.16, +3.44]), both accompanied by reduced reasoning-token usage. The uncertainty of individual headline estimates should therefore be considered alongside the broader aggregate evidence.
Second, policy-space scaling does not require arbitrary checkpoint pairs to improve every benchmark. As explicitly stated in Section 2.1, our claim concerns the additional deployable policies accessible from a fixed RL history, not a universal improvement guarantee. With the same calibration rule, 5/8 additional pairs improve two-task mean accuracy while reducing tokens, and 10/16 individual task comparisons exceed both source checkpoints. We report all pairs, including regressions, to characterize this policy space rather than selectively present successful cases.
The initial DeepSeek AIME24 study motivated the common spectral target, which was subsequently applied to Nemotron, OLMo, and all eight new pairs without pair-specific tuning.
We hope these distinctions help clarify how the collective experimental evidence supports the paper's central contribution.
1. Does the result depend on the original checkpoint pair? We tested eight additional source pairs, four from DeepSeek and four from OLMo, with τ = 0.95 fixed and FC selected from each anchor spectrum. SURGE beat both source checkpoints in 10 of 16 task comparisons. Five of the eight constructed policies improved two-task mean accuracy and used fewer mean reasoning tokens than the source with the higher two-task mean.
The tables give SURGE's accuracy change (pp) over the higher-scoring of its two endpoints on each task. All eight additional pairs are shown; the original pairs are included for comparison.
DeepSeek 3600/2800 reaches 53.33% on AIME24 with 7,581 reasoning tokens, 2.50 pp above the maximum across all 19 native measurements and 484 tokens shorter than native best. An earlier pair, 3400/2600, already reaches 50.00% when the highest score seen through step 3400 is 49.17%. In code, 1950/1700 improves on both endpoints on both tasks; so does 1600/1500, whose later source is 350 steps earlier than the original comparison. Section 3.6 and Appendix I give the full source scores and token counts.
2. What does the spectrum contribute? The DeepSeek 3600/2600 controls change one part of the construction at a time. At fixed block size, using the leading anchor RL-update block alone gives 44.17% on AIME24; adding the donor complement raises accuracy to 54.17%. Keeping the donor contribution but defining the block using random directions gives 47.29% on average, trailing directions give 44.17%, and the SVD of the absolute weights gives 41.67%. What distinguishes the highest-scoring policy in these controls is the combination of leading RL-update directions and the donor's complementary update (Table 4; Appendix B).
3. How were sources and FC selected? We used native benchmark accuracy and response length to select a strong anchor and a competitive, shorter-response donor, without a separate selection set. The pairs were DeepSeek 3600/2600, Nemotron 3440/1200 and OLMo 1950/1500. Step 1950 led the two-task mean among six OLMo candidates; 1500 produced shorter responses. The DeepSeek AIME24 sweep peaked at FC = 0.8, retaining 95.9% of the anchor-update energy. We adopted τ = 0.95 on that basis. Despite their different spectra, the same rule selected Nemotron's and OLMo's strongest measured settings (FC = 0.8 and 0.7). For the eight new pairs, the anchor weights alone set FC with this target.
4. How large are the gains over native checkpoints? SURGE exceeds the highest accuracy measured across 19 native checkpoints on each task: 54.17% vs. 50.83% on DeepSeek AIME24 (+3.33 pp over step 3600) and 83.7% vs. 82.8% on OLMo HumanEval+ (+0.91 pp over step 1500). Paired bootstrap by problem gives 95% CIs of [−14.54, +21.19] and [−1.98, +3.81] pp, respectively. Across the nine-benchmark math suites, SURGE improves mean accuracy over native best by 2.08 pp on DeepSeek (95% CI [+0.31, +3.94]) and 1.78 pp on Nemotron (95% CI [+0.16, +3.44]). Table 27 and our reply to EdGj report the full paired bootstrap intervals for accuracy and reasoning tokens.