acceptodds
Under review as a conference paper at ICLR 2027

Does Scaling Reinforcement Learning Really Require More Training?

11 views

Abstract

Scaling reasoning typically spends more compute on reinforcement learning (RL) or on inference. We show that a completed RL training history can yield policies stronger than the checkpoints visited by its optimizer. We call this *policy-space scaling*: expanding the deployable policy set accessible from a fixed RL history, without extending training or increasing per-query inference computation. We instantiate it with SURGE (Scaling Up RL Gradient-free via Eigenspace fusion). SURGE combines two checkpoints from the same RL run: a high-accuracy anchor and a competitive donor that generates shorter responses. It expresses both checkpoints as changes from their shared initialization, then spectrally decomposes the anchor's update to retain its dominant component and incorporate the donor's complementary component. With a fixed target for how much of the anchor update to retain, SURGE determines the block size from the weights without testing candidate policies. We evaluate two 1.5B mathematical-reasoning histories, DeepSeek and Nemotron, and one 7B coding history, OLMo. SURGE improves benchmark-average accuracy over both input checkpoints while using fewer reasoning tokens than the anchor. It reaches 54.17% on DeepSeek AIME24 against a measured native maximum of 50.83%, and 83.7% on OLMo HumanEval+ against 82.8%. These gains exceed the observed training curves. Geometric controls support the importance of RL-update structure beyond weight displacement or token reduction alone. Each constructed model runs as a single policy. Our findings identify stored RL history as a reusable scaling resource: the capability available from a training run need not end at its best checkpoint.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Policy-Space Scaling, Eigenspace, Reasoning Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iAZxOpenReview ↗
Rating
1
Confidence
4
Informativeness
1(−1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SURGE, a post-training method for constructing a policy from two checkpoints within the same RL trajectory. It decomposes the anchor’s parameter updates relative to the shared RL initialization, preserves a leading two-sided singular subspace block, and incorporates the donor’s complementary updates. A retained-energy target determines the protected block size without evaluating fusion candidates once the source pair and target are fixed. Experiments on two 1.5B mathematical-reasoning histories and one 7B coding history report improved benchmark-average accuracy and reduced response length relative to the anchors. On two benchmarks, the constructed policies also exceed the best measured native checkpoint.

Critical strengths

The paper presents a practical way to reuse RL checkpoints without further gradient updates or additional deployment-time model overhead. The improvements across mathematical reasoning and coding make checkpoint construction a potentially useful extension of conventional checkpoint selection.

The geometric ablations provide useful evidence that the specific construction matters. Comparisons with scalar interpolation, checkpoint averaging, anchor-only truncation, absolute-weight SVD, and random subspaces help distinguish the reported gains from generic weight perturbation or response shortening.

Critical weaknesses

The connection between update energy and capability preservation remains insufficiently established. Leading singular directions capture large parameter changes, but do not necessarily identify directions important for reward or particular capabilities. The projection identities establish parameter preservation, not behavioral preservation. While the ablations support the selected construction empirically, they do not establish when a donor’s complementary updates should be beneficial. Each main experiment uses one selected checkpoint pair per history, leaving unclear whether the method reliably helps across checkpoints with different capability levels or different patterns of strengths and weaknesses. This limits confidence in the method as a generally applicable checkpoint-construction procedure.

The robustness of the improvement over native checkpoint selection needs stronger validation. The headline gains are modest relative to the evaluation sizes: DeepSeek AIME24 improves from 50.83% to 54.17% with four responses per problem, while OLMo HumanEval+ improves from the best native checkpoint’s 82.8% to 83.7%. The broader mathematical results are encouraging, but repeated decoding rounds do not establish robustness across independently trained histories or checkpoint-pair choices. Furthermore, source selection uses performance measurements, and the 95% energy target was identified in the initial mathematical study. A clearer separation between development choices and held-out assessment is needed to evaluate how reliably the procedure exceeds native checkpoint selection.

The key issue is that I don’t see how this line of research contributes to the broader RL scaling agenda. It just seems like a niche paper that’s hard to find fault with… As evidence of a change in RL scaling, the title and framing currently go beyond what the experiments support. As a hands-on RL researcher, I care less about how compelling the narrative is and more about its impact on actual training.

Questions for authors

Clarification: What operational rule selected each anchor and donor, and which evaluation data informed those choices and the 95% retained-energy target? Were any reported benchmarks held out from these decisions?

Evidence that could change my assessment: Within the same RL history, how consistently does the fixed procedure improve performance across multiple checkpoint pairs with different accuracy gaps and response lengths? Reporting the frequency and magnitude of gains and regressions would help establish when donor complementarity is useful.

Evidence that could change my assessment: Can you provide uncertainty estimates for the improvements over the strongest native checkpoints, preferably using paired resampling at the problem level and additional decoding samples? The key comparison is against the best native checkpoint, which is not always the anchor.

3 comments

1. Does the result depend on the original checkpoint pair? We tested eight additional source pairs, four from DeepSeek and four from OLMo, with τ = 0.95 fixed and FC selected from each anchor spectrum. SURGE beat both source checkpoints in 10 of 16 task comparisons. Five of the eight constructed policies improved two-task mean accuracy and used fewer mean reasoning tokens than the source with the higher two-task mean.

The tables give SURGE's accuracy change (pp) over the higher-scoring of its two endpoints on each task. All eight additional pairs are shown; the original pairs are included for comparison.

DeepSeek: anchor / donorOlympiadAIME24
3600 / 2600 (original)+0.74+3.33
3600 / 2800+1.22+2.50
3600 / 1800+0.480.00
3400 / 2600+0.14+0.83
3000 / 2600−0.77−0.83
OLMo: anchor / donorHumanEval+MBPP+
1950 / 1500 (original)+0.910.0
1950 / 1300+0.5−0.7
1950 / 1700+0.9+0.5
1700 / 1500−0.5−0.4
1600 / 1500+0.1+0.9

DeepSeek 3600/2800 reaches 53.33% on AIME24 with 7,581 reasoning tokens, 2.50 pp above the maximum across all 19 native measurements and 484 tokens shorter than native best. An earlier pair, 3400/2600, already reaches 50.00% when the highest score seen through step 3400 is 49.17%. In code, 1950/1700 improves on both endpoints on both tasks; so does 1600/1500, whose later source is 350 steps earlier than the original comparison. Section 3.6 and Appendix I give the full source scores and token counts.

2. What does the spectrum contribute? The DeepSeek 3600/2600 controls change one part of the construction at a time. At fixed block size, using the leading anchor RL-update block alone gives 44.17% on AIME24; adding the donor complement raises accuracy to 54.17%. Keeping the donor contribution but defining the block using random directions gives 47.29% on average, trailing directions give 44.17%, and the SVD of the absolute weights gives 41.67%. What distinguishes the highest-scoring policy in these controls is the combination of leading RL-update directions and the donor's complementary update (Table 4; Appendix B).

3. How were sources and FC selected? We used native benchmark accuracy and response length to select a strong anchor and a competitive, shorter-response donor, without a separate selection set. The pairs were DeepSeek 3600/2600, Nemotron 3440/1200 and OLMo 1950/1500. Step 1950 led the two-task mean among six OLMo candidates; 1500 produced shorter responses. The DeepSeek AIME24 sweep peaked at FC = 0.8, retaining 95.9% of the anchor-update energy. We adopted τ = 0.95 on that basis. Despite their different spectra, the same rule selected Nemotron's and OLMo's strongest measured settings (FC = 0.8 and 0.7). For the eight new pairs, the anchor weights alone set FC with this target.

4. How large are the gains over native checkpoints? SURGE exceeds the highest accuracy measured across 19 native checkpoints on each task: 54.17% vs. 50.83% on DeepSeek AIME24 (+3.33 pp over step 3600) and 83.7% vs. 82.8% on OLMo HumanEval+ (+0.91 pp over step 1500). Paired bootstrap by problem gives 95% CIs of [−14.54, +21.19] and [−1.98, +3.81] pp, respectively. Across the nine-benchmark math suites, SURGE improves mean accuracy over native best by 2.08 pp on DeepSeek (95% CI [+0.31, +3.94]) and 1.78 pp on Nemotron (95% CI [+0.16, +3.44]). Table 27 and our reply to EdGj report the full paired bootstrap intervals for accuracy and reasoning tokens.

Thank you for the detailed response and the additional experiments. I have read the rebuttal and the replies to the other reviewers.

My main concerns remain. The paired intervals for the headline comparisons against the best native checkpoints all include zero ([−14.54, +21.19] pp on AIME24, [−1.98, +3.81] pp on HumanEval+). The response also confirms that the sources and the 95% target were selected on the reported benchmarks, with no held-out set. The additional pairs are appreciated, but three of the eight do not improve and most changes are within ±1 pp, reported without intervals.

I am therefore keeping my score.

We thank the reviewer for the follow-up. We would like to clarify two distinctions regarding the interpretation of our additional evidence.

First, individual headline comparisons and aggregate robustness address different questions. While the per-task confidence intervals on AIME24 and HumanEval+ include zero, the paired nine-benchmark macro-average gains are +2.08 pp on DeepSeek (95% CI [+0.31, +3.94]) and +1.78 pp on Nemotron ([+0.16, +3.44]), both accompanied by reduced reasoning-token usage. The uncertainty of individual headline estimates should therefore be considered alongside the broader aggregate evidence.

Second, policy-space scaling does not require arbitrary checkpoint pairs to improve every benchmark. As explicitly stated in Section 2.1, our claim concerns the additional deployable policies accessible from a fixed RL history, not a universal improvement guarantee. With the same calibration rule, 5/8 additional pairs improve two-task mean accuracy while reducing tokens, and 10/16 individual task comparisons exceed both source checkpoints. We report all pairs, including regressions, to characterize this policy space rather than selectively present successful cases.

The initial DeepSeek AIME24 study motivated the common spectral target, which was subsequently applied to Nemotron, OLMo, and all eight new pairs without pair-specific tuning.

We hope these distinctions help clarify how the collective experimental evidence supports the paper's central contribution.

Reviewer_EdGjOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes policy-space scaling, which uses existing RL training histories to construct policies that the optimizer has never visited. SURGE applies spectral decomposition to checkpoint updates relative to their shared initialization, keeping the dominant components of a high-accuracy anchor while incorporating complementary components from a donor. Experiments on two 1.5B math training runs and one 7B coding run show improved accuracy and fewer generated tokens compared to the anchor. In two full-history comparisons, SURGE also achieves higher point estimates than the best evaluated checkpoints.

Critical strengths
  1. An interesting perspective with practical value. As RL training shifts toward harder problems, models may forget how to solve simpler ones learned earlier. This makes combining complementary capabilities from different checkpoints a promising idea. Treating past training histories as reusable resources offers a relatively low-cost way to improve RLVR performance. The resulting model retains the same architecture and can be deployed as a single policy without further RL training.

  2. Solid experiments and comparisons. The paper provides fairly comprehensive experiments that support the benefits of structured update merging. The transfer of a fixed energy retention target to the 7B coding model also strengthens the results.

Critical weaknesses
  1. Limited statistical evidence for surpassing the best historical checkpoint. Full-history comparisons are only provided for DeepSeek–AIME24 and OLMo–HumanEval+, with the latter improving over the best original checkpoint by only about 0.9 percentage points. Four samples per problem and the variance analysis in Appendix C are not enough to establish the reliability of such small gains.

  2. The generality of checkpoint selection and calibration remains unclear. Each training run mainly evaluates one checkpoint pair, selected based on accuracy and response length. The 95% energy retention target also comes from early math experiments. While the coding results show that this fixed target can transfer, the checkpoint selection criteria, the range of pairs tried, and the use of independent validation need further clarification.

Questions for authors
  1. Using the existing per-problem samples, could you report paired accuracy differences and confidence intervals between SURGE and the best original checkpoint? This would help assess how strong the evidence is for surpassing the historical best.

  2. What evaluation data were used to select the anchor, donor, and 95% energy retention target? Was checkpoint selection based on a separate development set, and were other checkpoint pairs tried?

3 comments

1. Accuracy gains and paired confidence intervals. Across nine math benchmarks, SURGE improves DeepSeek mean accuracy by 2.08 pp (95% CI [+0.31, +3.94]) while using 5.3% fewer reasoning tokens. On Nemotron, the same construction gains 1.78 pp [+0.16, +3.44] with 12.8% fewer tokens.

The table gives SURGE minus the listed checkpoint, with paired 95% CIs. Tokens are relative changes for suites and absolute changes for individual tasks. The last row is the additional comparison to the HumanEval+ trajectory maximum.

Suite or taskComparatorAccuracy change, pp [95% CI]Reasoning-token change [95% CI]
DeepSeek, nine-task mean3600, native best+2.08 [+0.31, +3.94]−5.3% [−6.78%, −3.60%]
Nemotron, nine-task mean3440, native best+1.78 [+0.16, +3.44]−12.8% [−13.71%, −11.20%]
OLMo, two-task mean1950, native best+0.85 [−0.73, +2.41]−10.4% [−14.0%, −6.6%]
DeepSeek, AIME243600, native best and task maximum+3.33 [−14.54, +21.19]−267 [−712, +177]
OLMo, HumanEval+1950, native best+1.70 [−0.91, +4.27]−543 [−840, −233]
OLMo, HumanEval+1500, task maximum+0.91 [−1.98, +3.81]+559 [+295, +831]

Figure 1 shows a further result: SURGE reaches 54.17% on AIME24 and 83.7% on HumanEval+, above the highest accuracies measured across 19 checkpoints on each task (50.83% and 82.8%). On OLMo, step 1500 sets the HumanEval+ maximum; step 1950 has the best two-task mean and remains the primary comparator. Appendix I, Table 27 reports both comparisons.

2. What determined the sources and FC? We used accuracy and response length from native benchmark evaluations to choose a strong anchor and a competitive, shorter-response donor. There was no separate source-selection set. The anchor/donor pairs were DeepSeek 3600/2600, Nemotron 3440/1200 and OLMo 1950/1500; six OLMo candidates had scores on both coding tasks. In the initial DeepSeek AIME24 FC sweep, the highest measured accuracy (54.17%) occurred at FC = 0.8, which retained 95.9% of the anchor-update energy. This observation motivated τ = 0.95. Choosing the FC whose retained energy was closest to 0.95 gave FC = 0.8 on Nemotron and 0.7 on OLMo; both settings had the highest measured accuracy in their respective sweeps. We used the same target for all eight additional pairs without pair-specific tuning or accuracy sweeps.

3. Does the result extend to other pairs? Five of the eight new pairs improved two-task mean accuracy and reduced mean reasoning tokens relative to the source with the higher two-task mean. For instance, replacing the OLMo donor with step 1700 improves both HumanEval+ (+0.9 pp) and MBPP+ (+0.5 pp) over their stronger endpoints, while saving 643 and 523 tokens relative to step 1950. The same frozen rule thus recovers stronger policies from different sources within the same completed run. Section 3.6, Appendix I and our reply to iAZx give the pair results.

Dear Reviewer EdGj,

Thank you for your thoughtful review and for recognizing the practical value of policy-space scaling and our experimental findings.

Before the discussion period concludes, we would like to briefly summarize the additional evidence addressing your questions:

  • Paired uncertainty analysis: We computed problem-level paired bootstrap intervals under the original evaluation protocol. Across nine mathematical benchmarks, SURGE achieves +2.08 pp on DeepSeek (95% CI [+0.31, +3.94]) and +1.78 pp on Nemotron ([+0.16, +3.44]), both accompanied by reduced reasoning tokens. We also report the requested individual trajectory comparisons, including the HumanEval+ maximum at OLMo step 1500.
  • Checkpoint selection and calibration: We clarified the source-selection procedure and the empirical origin of the 95% energy target. The same calibration rule was subsequently applied across different model families and eight additional checkpoint pairs, without pair-specific accuracy sweeps. Five of the eight new pairs improve two-task mean accuracy while reducing tokens relative to the stronger source on that mean.

These results and the complete experimental details are included in the revised manuscript (Section 3.6 and Appendix I).

We hope the additional analysis addresses your concerns regarding robustness and generality. We would greatly appreciate any further feedback before the discussion period closes.

Thank you again for your time and consideration.

Thanks for the author's response. I will keep my score.

Reviewer_yFJNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether additional reasoning capabilities can be extracted from existing RL trajectories without further training. It proposes SURGE, a gradient-free checkpoint fusion method that combines two checkpoints by preserving the dominant spectral components of the anchor, and incorporating complementary components from a donor checkpoint. The fusion ratio is determined using a retained spectral-energy criterion. Experiments on mathematical reasoning and coding benchmarks demonstrate improvements in both accuracy and reasoning-token efficiency.

Critical strengths
  1. The paper explores an important question: whether existing RL trajectories contain additional capabilities that can be recovered without further optimization.
  2. The proposed method does not require additional gradient updates or inference-time ensembling. Its construction is relatively straightforward and produces a single deployable model without changing the original architecture.
Critical weaknesses
  1. Insufficient comparisons with competitive model-merging baselines. The core operation of SURGE consists of decomposing parameter updates using SVD and combining selected components from two models. This is conceptually related to existing approaches involving task-vector arithmetic, spectral model merging, and subspace-based parameter composition. Without comparisons against these methods, it remains unclear whether the reported gains result from the specific design of SURGE.
  2. The robustness and statistical significance of the reported improvements remain unclear.
Questions for authors
  1. How does SURGE compare experimentally with competitive model-merging baselines.
2 comments

1. Same checkpoints, different merge rules. We evaluated TIES-Merging with its default validation-free configuration (nominal 20% retention, λ = 1) on the same DeepSeek 3600/2600 checkpoints and evaluation protocol as SURGE. SURGE scores 54.17% vs. 47.50% on AIME24 (+6.67 pp) and 67.32% vs. 65.62% on OlympiadBench (+1.70 pp). Each cell gives accuracy (%) / mean reasoning tokens.

PolicyAIME24OlympiadBench
Native best (3600)50.83 / 8,06566.58 / 6,246
Donor (2600)47.50 / 7,65164.80 / 5,570
TIES-Merging47.50 / 8,08065.62 / 5,689
SURGE54.17 / 7,79867.32 / 5,832

Only 0.006% of coordinates retained by both updates disagree in sign, leaving little conflict for TIES to resolve on this pair. SURGE instead preserves a spectral block of the anchor's RL update and takes the complementary update from the donor (Section 3.5; implementation in Appendix B).

Our primary objective is to establish policy-space scaling within a fixed RLVR training history, rather than general-purpose merging across independently fine-tuned task models. The TIES comparison further shows that structured spectral recombination offers a distinct and effective construction in this setting.

2. Which part of the construction matters? On the same AIME24 pair, the scalar merge matched to SURGE's weight displacement scores 47.50%, and late averaging reaches 50.83%. At the same block size, absolute-weight SVD scores 41.67%, trailing singular directions 44.17%, and random directions 47.29% on average. Random and trailing directions both generate shorter responses than SURGE. Keeping the leading RL-update block alone gives 44.17%; adding the donor complement brings it to 54.17%. The controls separate the choice of RL-update directions and donor content from displacement, rank and response length (Table 4; Appendix B).

3. Does this carry across checkpoints? With the retention target frozen, SURGE beats both sources in 10 of 16 task comparisons across eight new pairs. Five of the eight improve mean accuracy and reduce mean reasoning tokens relative to the stronger single source on the two-task mean. Across the original nine-task suites, DeepSeek gains +2.08 pp [95% CI: +0.31, +3.94] and Nemotron +1.78 [+0.16, +3.44] over native best, with 5.3% and 12.8% fewer reasoning tokens. Our reply to iAZx gives every pair result; our reply to EdGj and Table 27 give the full interval table.

Dear Reviewer yFJN,

Thank you for your constructive review and for recognizing the importance of reusing completed RL training histories.

As the discussion period approaches its end, we would like to briefly highlight the additional evidence addressing your concerns:

  • Model-merging comparison: On the same DeepSeek checkpoints, SURGE outperforms TIES-Merging on both AIME24 (54.17% vs. 47.50%) and OlympiadBench (67.32% vs. 65.62%). Together with our geometric ablations, these results support the value of exploiting RL-update spectral structure, beyond generic checkpoint merging.
  • Robustness: Across nine mathematical benchmarks, SURGE improves accuracy by +2.08 pp on DeepSeek (95% CI [+0.31, +3.94]) and +1.78 pp on Nemotron ([+0.16, +3.44]), while reducing reasoning tokens. We also evaluated eight additional checkpoint pairs, including both successful and unsuccessful cases.

Our central contribution is policy-space scaling within a fixed RL history, and we hope these additional results further substantiate this finding.

We would be grateful to know whether the new evidence addresses your main concerns, or if any important questions remain before the discussion closes.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.