acceptodds
Under review as a conference paper at ICLR 2027

From Selection to Switching in RLVR

Abstract

Data selection can improve training efficiency and performance, but its computational costs and effectiveness must be analyzed before it is applied. We compare uniform sampling, on-policy gradient alignment, and cached success-rate (SR) selection in reinforcement learning with verifiable rewards (RLVR). Matched-update MATH experiments under GRPO show selection gains over uniform sampling, with on-policy alignment stronger initially on average and cached SR stronger at later checkpoints. We then examine whether the selection strategy can be changed online during training, from alignment to SR, through executed switching runs and retrospective diagnostics. On the evaluated MATH/GRPO trajectories, switching achieves higher final reward at matched updates than continued alignment and SR started immediately after a shared prefix. Fixed-time switching experiments provide complementary evidence. RLOO controls show objective-dependent selector ordering, while MBPP extends reward and cost comparisons to code generation. With an existing reward cache, incremental selection-and-training costs are lower than continued alignment but higher than early SR. These results support changing selection strategies during RLVR while accounting for selection costs.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, data selection, gradient alignment, compute-efficient training, importance weighting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zUxsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Data selection can make reinforcement learning with verifiable rewards (RLVR) more effective, but different selection methods have very different computational costs and may not remain equally useful throughout training. In particular, on-policy gradient-alignment methods can identify valuable training examples using the current model, but they require expensive fresh rollouts and gradient computations. Cheaper alternatives such as cached success-rate selection avoid this repeated scoring cost, but may be less effective at some stages of training. The paper is motivated by the observation that the best selection strategy may therefore depend on the model’s current training state, rather than there being a single selector that should be used for the entire run.

The paper’s main idea is to move from static data selection to dynamic strategy switching. It compares on-policy gradient alignment with a cached success-rate selector and finds that on-policy alignment tends to perform better early in GRPO training on MATH, while cached success-rate selection becomes stronger at later checkpoints. Based on this changing ordering, the authors propose starting with on-policy selection and then switching to cached success-rate selection during training. They introduce an auxiliary gradient-based signal, SR-GC, together with temporal confirmation to decide when to switch. In the two evaluated online-switching trajectories, this strategy achieves higher final reward than either continuing on-policy selection or switching to cached SR immediately after the shared prefix, while costing less than continued on-policy selection.

Critical strengths

I had a hard time understanding the paper, so I do not have any specific strengths in mind to highlight.

Critical weaknesses

(Legibility issues)

The paper is very hard to read, and feels strongly LLM written. I had a very hard time understanding what it was trying to say.

Very poorly written introduction. It feels like a weird blend of intro and related work, and I did not understand the work well due to poor writing. Badly made plots. For example, Figure 2 occupies way more space than it should. More seriously, Figure 4 has a y-axis label and y-axis title overlapping, to the point the “-0.4” looks like 0.4, and I was very confused about the plots.

Overall, the presentation should be significantly improved before the paper should be accepted into a venue of ICLR’s caliber.

(Problems with methodology and experiments)

[1] The result that Switch beats both alternatives, is demonstrated on one LLM backbone and one dataset/objective setting. It needs to be more robust, experiments on different models and datasets will make the paper stronger.

[2] Does SR-GC choose a better switch time than a simple fixed schedule? The baselines here are (1) always on-policy, (2) always SR, (3) random. This are not strong enough baselines. Possible alternatives:

  • fixed switch at several plausible times,

  • a tuned fixed switch chosen on development seeds,

  • switch based on cheaper observable statistics,

  • possibly periodic re-evaluation or reverse switching.

Without those, it is difficult to know whether the value comes from adaptive switching or simply from “use expensive selection early, cheap selection later.”

Questions for authors

None, please address the weaknesses mentioned above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer zUxs

Thank you for the direct comments. Locations refer to the revised PDF.

Legibility and writing.

The manuscript was rewritten for clarity, with a new introduction, a new overview figure (Figure 1, p. 2), a separate related-work section and revised figures.

  • Introduction: it states the problem, the two findings (the better selector changes with the training stage and with the model and dataset), the proposed switch and how to time it, each with a pointer to its section, and a closing paragraph lists the three contributions (§1, p. 1 l. 29 – p. 2 l. 92; contributions, p. 2 l. 86–92). Prior work is in its own section (§2, p. 2 l. 95 – p. 3 l. 126), and Table 1 maps each experiment to the section that reports it (p. 5 l. 216–229).
  • Submitted Figure 2 (online curves for seeds 3 and 4) is now Appendix Figure 5 (p. 37), and the 3×3 grid of fixed-step curves is Appendix Figure 6 (p. 38). The main text keeps a one-panel summary (Figure 2) next to a summary table (Table 5) (p. 7).
  • Submitted Figure 4 (the panels with the −0.4 tick) is now Figure 7 (App. F, p. 43). The tick labels carry explicit minus signs, and the y-axis title sits clear of the tick labels.
  • Abbreviations are written out at first use (e.g. GRPO p. 1 l. 53, TD p. 3 l. 117–118, RLOO p. 5 l. 228).

[1] One backbone and one dataset.

The revision trains four backbones on two datasets: OLMo-3 7B, Qwen3.5-9B, Gemma 4 12B IT and Llama-3.1-8B-Instruct, each on MATH and on MBPP, with five seeds (5–9) per cohort (§5.2, p. 6 l. 284–289; Tables 3–4, p. 6; per-seed results in Tables 24–35, pp. 30–36). All cohorts use the same four methods, refresh interval, SR-GC rule, 275-update horizon and evaluation on 300 held-out questions with eight responses each (§4, p. 5 l. 234–238; App. C, p. 20 l. 1071–1075). The better single selector differs between cohorts: SR minus On-policy is +1.24 [+0.23, +2.25] on Qwen3.5 MATH and −2.20 [−3.35, −1.05] on Qwen3.5 MBPP. The mean reward of Switch stays within 0.88 points of the best method in every cohort and is the highest in five (§5.3, p. 7 l. 362–365). With five seeds, its paired intervals against SR and On-policy include zero. Gemma solves 83.7–84.6% (MATH) and 90.6–91.2% (MBPP) of the training candidates before training, and all of its methods end within 1.00 point of Random (p. 6 l. 322–323).

[2] Does SR-GC choose a better switch time than a fixed schedule?

On OLMo-3 MATH, the gain of Switch comes from moving from gradient alignment to cached SR, and SR-GC supplies the switch time online; the best fixed time is known only in hindsight (p. 8 l. 402–404). The branched OLMo-3 MATH runs compare SR-GC with every fixed switch time on one trajectory per seed (§5.4, p. 8 l. 386–404; App. E, Tables 40–43, pp. 41–42). Against your suggested baselines:

  • Fixed switch at several times: all ten times after updates 25–250 raise the mean reward over continued On-policy (38.61%), by up to 1.58 points (p. 8 l. 393–394; Table 41, p. 41). The best time differs from seed to seed, from update 50 to update 225. Only the two ends gain little (+0.04 after update 25, +0.07 after update 250). From update 50 to 225 the means lie within 38.92–40.18% and are not ordered by time, so the gain does not depend on hitting one exact time, and SR-GC lands in this range (40.02%). It is above nine of the ten fixed times; the best one reaches 40.18% (after update 75). At update 275 it ends 1.41 points above continued On-policy ([−0.60, +3.42], 4/0/1 seeds).
  • Fixed switch tuned on development seeds: not run as a separate experiment. The existing runs give one comparison: the fixed step with the largest gain on seeds 0–2 (update 25, +1.67; Table 5, p. 7) gives the lowest mean of the ten on seeds 5–9 (38.65%).
  • Cheaper observable statistic: a trigger that reads only the training reward and needs no gradients. It switches on three seeds and ends 0.69 points below SR-GC ([−1.03, +2.42]) (p. 8 l. 395–397; Table 42, p. 41).
  • Equal compute: at 21.39 GPU-hours, the mean cost of SR-GC including its checks, with each run extended past update 275 as far as the budget allows, SR-GC ends 2.48 points [+1.28, +3.67] above continued On-policy on average, higher on all five seeds (p. 8 l. 398–399).
  • Destination: from the SR-GC switch state, continuing with SR ends 1.49 points above continuing with uniform sampling ([−1.48, +4.46], 4/0/1) (p. 8 l. 401–402).
  • Periodic re-evaluation or reverse switching: not run. The switch is one-way, since monitoring the contrast after the switch would restore the scoring cost (§3.3, p. 4 l. 197–199).

Across all cohorts, SR-GC switches in 39 of the 40 Switch runs on seeds 5–9, at times that vary across cohorts, and with the switch time chosen online by SR-GC, the mean reward of Switch stays within 0.88 points of the best method in all eight cohorts (§5.4, p. 8 l. 383–384; Table 25, p. 31; §8, p. 10 l. 530–533).

I thank the authors for their substantial effort in the rebuttal period.

However, the paper has changed significantly from its early version. While I appreciate the authors' effort, I believe the new paper would benefit from a full review cycle in a future conference, since it would require substantial effort to re-read the new manuscript and judge its merits.

I will keep my current score.

Thank you for your time and for considering our response. We appreciate your feedback, which has helped us improve the manuscript.

Reviewer_UrX8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a data selection approach for RLVR, which switches from one selection strategy, namely on-policy gradient alignment, to another, namely cached success-rate. Switching was compared with continued on-policy alignment, and with cached SR and uniform sampling started after a shared 25-update on-policy prefix on MATH/GRPO trajectories on a single 7B model and two seeds. The authors report higher reward for the switching method, while being computationally cheaper than pure on-policy gradient alignment. Beyond the switching experiments, the paper further contains a matched-update study, a fixed budget analysis, MBPP, RLOO control, and theory.

Critical strengths

1: This paper is a fairly rigorous and honest empirical study of when cheap selection beats gradient-based selection. The design is mostly carefully chosen, the results are reported candidly, and the question if the selector should change during training is relevant, yet has received little attention in RLVR.

Critical weaknesses

1: My main concern with this work is that the core claim of this paper, namely that the results support changing selection strategies during RLVR, may be under-supported: While Switch outperforms cached SR at 275 steps, it does so very slightly (0.71-1.21pp), and results are only across two seeds (acknowledged as a limitation by the authors). Also, Switch is approx. 13.2% more expensive than SR (which is also acknowledged by the authors). As such, it is not entirely clear to me that the results indeed support the notion that the selection criterion should change, and that SR-GC should be used to manage this switch. This concern seems further supported by the results on the reliability of the SR-GC signal, incl. the single-check experiments (5.7), the retrospective analysis (5.6), and the jackknife upper bounds. Furthermore, the switching rule (exactly two consecutive negatives, or two negatives with a non-negative in-between with a below-zero mean) does not seem sufficiently motivated (see Q5).

2: The related work section is very brief and should discuss additional relevant prior work. For instance, the authors could discuss sampling selection schemes in the broader field of RL (Prioritized Level Replay (ICML 2021), or for value-based RL, Prioritized Experience Replay (ICLR 2016) or more recently, Uncertainty Prioritized Experience Replay (RLC 2025) or Reliability-adjusted Prioritized Experience Replay (ICLR 2026)). In this context, it may be interesting to discuss that PER (and multiple strategies building upon PER) anneal the importance-sampling exponent towards 1, effectively altering the impact of their sampling scheme throughout training, which motivates Q3.

Questions for authors

1: Could the authors add more seeds to further support their core claim?

2: Why do the authors compare at step 275 of the switch variant, and how did the authors determine the total runtime for their experiment?

3: Why did the authors not consider other switching strategies like on-policy gradient alignment to uniform?

4: Various abbreviations are used without introduction. Could the authors please ensure that all abbreviations are properly introduced, that is, written out on first usage?

5: Could the authors provide a sensitivity analysis on the impact of the exact switching rule, or explain more thoroughly why the authors chose this specific approach?

6 comments
AuthorsOpenReview ↗

Response to Reviewer UrX8 (part 1 of 2)

Thank you for the careful reading and the pointers to related work. Locations refer to the revised PDF.

W1. Strength of the core claim (gain over SR of 0.71–1.21 points on two seeds, 13.2% more cost than SR, SR-GC reliability, motivation of the rule).

The revision supports the claim with five new seeds in eight cohorts: the better single selector changes from cohort to cohort, and with the switch time chosen online by SR-GC, the mean reward of Switch stays within 0.88 points of the best method in all of them (§5.2, p. 6 l. 284–289; §5.3, p. 7 l. 362–367; §8, p. 10 l. 530–533).

  • Seeds 3 and 4: at update 275 Switch is 0.71 and 1.21 points above SR and 0.92 and 4.04 points above continued On-policy (§5.3, p. 7 l. 376–377).
  • The better single selector changes with the setting. SR minus On-policy is +1.24 [+0.23, +2.25] on Qwen3.5 MATH, −2.20 [−3.35, −1.05] on Qwen3.5 MBPP and +1.04 [+0.03, +2.05] on Llama MATH (Table 4, p. 6).
  • Switch is the most stable method across these settings. In mean reward, SR and On-policy fall up to 2.20 and 4.27 points below the best method, Switch at most 0.88 (p. 7 l. 365–366). With five seeds, its 16 paired intervals against SR or On-policy all include zero (p. 7 l. 367–368).
  • Cost: Switch costs more than SR because it scores during the alignment phase, by 13.2% on seeds 3 and 4, 8.8–25.4% in three of the Qwen and Gemma cohorts and 60.7% on Qwen MBPP. It costs 17.1% and 11.0–21.9% less than On-policy (§6, p. 8 l. 408–413 and l. 429–431; Table 6, p. 8).
  • SR-GC and the rule: the sign of SR-GC picks the better strategy in 7 of 9 completed comparisons, and the rule confirms it over successive checks because a single check is noisy (App. B.1, p. 17 l. 864–872; §3.3, p. 4 l. 190–195). The branched OLMo-3 MATH runs test the switch time directly. Switching to cached SR raises the mean reward over continued On-policy at every tested switch time, by up to 1.58 points, and SR-GC decides during training and reaches 40.02%, against 40.18% for the best fixed time, which is known only in hindsight. At an equal budget of 21.39 GPU-hours it ends 2.48 points above continued On-policy on average, higher on all five seeds (§5.4, p. 8 l. 393–399; Tables 41–43, pp. 41–42).

W2. Related work (PLR, PER, UPER, ReaPER; annealed importance-sampling exponent).

The revision covers these methods in a new paragraph, "Adaptive prioritization in reinforcement learning" (§2, p. 3 l. 117–126). It describes prioritized experience replay (Schaul et al., 2016), including the importance-sampling exponent raised linearly to one over training; prioritized level replay (Jiang et al., 2021); uncertainty-prioritized experience replay (Carrasco-Davis et al., 2025); and reliability-adjusted prioritized experience replay (Pleiss et al., 2026). These methods adjust one prioritization scheme over stored transitions or levels as training goes on. Our method moves between two selectors whose scoring costs differ widely, and the selected prompts generate new rollouts.

Q1. More seeds.

All eight cohorts now have five new seeds (5–9) (§5.2, p. 6 l. 284–285; Tables 3–4, p. 6), and the branched timing runs on OLMo-3 MATH use the same five seeds (§5.4, p. 8 l. 386–390; App. E). Seeds 3–4 are reported separately from seeds 5–9 because SR-GC was measured differently (§4, p. 4 l. 211 – p. 5 l. 232).

AuthorsOpenReview ↗

Response to Reviewer UrX8 (part 2 of 2)

Q2. "Why step 275, and how was the total runtime determined?"

Update 275 is the fixed training horizon of the online runs, the 25-update shared prefix followed by 250 updates, and run length is set by this number of updates, not by time (§4 "Horizon and evaluation", p. 5 l. 234–238).

  • All online runs on seeds 5–9 train to update 275 and are evaluated there (p. 5 l. 234–235).
  • Seeds 3 and 4 are compared at the same update. All their runs trained past update 275, and at the last common evaluation (update 315 for seed 3, 310 for seed 4) Switch is +0.17 and +1.88 points above SR and 0.00 and +3.12 points above On-policy (§5.3, p. 7 l. 377 – p. 8 l. 379; App. C.1, p. 23 l. 1215–1217; Figure 5, p. 37).
  • Cost is measured for every method at this horizon (§6). The branched runs add an equal-compute endpoint at 21.39 GPU-hours, with each run extended past update 275 as far as the budget allows, where SR-GC ends 2.48 points above continued On-policy (§5.4, p. 8 l. 398–399; App. E.1, p. 40 l. 2154–2159), and §6.1 reports the selector comparison under a fixed MATH allocation of 29,040 GPU-seconds (p. 9 l. 463–469; Table 18, p. 27).

Q3. "Why not on-policy gradient alignment to uniform?"

We ran this switch on the branched OLMo-3 MATH runs, and moving to cached SR ends higher. From the state where SR-GC switches, continuing with SR ends 1.49 points above continuing with uniform sampling, higher on four of five seeds ([−1.48, +4.46]) (§5.4, p. 8 l. 401–402; App. E.1, p. 40 l. 2138–2139; Table 43, p. 42). The uniform branch reaches 38.52%, against 40.02% for SR-GC and 38.61% for continued On-policy (Tables 41–42, p. 41).

Q4. Abbreviations.

Every abbreviation is now written out at first use: RLVR, SR, MBPP and SR-GC in the abstract (p. 1 l. 10–23); GRPO (p. 1 l. 53); TD (p. 3 l. 117–118); KL (p. 3 l. 154); RLOO in the Table 1 caption (p. 5 l. 228–229); SD and CI (p. 5 l. 241–242); GPU-h (p. 5 l. 249); RNG (p. 40 l. 2133).

Q5. Sensitivity to the exact rule, or why this rule.

The rule confirms the sign of SR-GC before switching, and with this confirmation SR-GC finds a good switch time during training. On the branched OLMo-3 MATH runs, the current rule and the longer confirmation R4 reach 40.02% and 40.38%, while switching on a single negative check ends 1.32 points below the current rule (Tables 42–43, pp. 41–42).

  • Rule (§3.3, p. 4 l. 190–195): switch after two negative checks in a row, or after a negative, non-negative, negative sequence whose three-check mean is negative. Confirmation is needed because the sign of a single check can differ between independent reference samples at the same checkpoint (4 of 9 states, App. B.1, p. 17 l. 870–872). The threshold, confirmation length and interval were fixed before training (p. 4 l. 193–195).
  • Confirmation length: one negative check (R1) 38.70%; two consecutive (R2) gives the same switch times as the current rule; the current rule 40.02%; three consecutive (R4) 40.38%. Current rule minus R1 is +1.32 [−0.49, +3.12], and current rule minus R4 is −0.37 [−2.51, +1.77].
  • Reference history (seeds 3 and 4): the rule applied to the A, B or A/B-mean history switches at steps 100, 150 and 125 for seed 3 and 225, 75 and 100 for seed 4. The Switch runs use the A/B mean (App. B.3, p. 18 l. 959–961; Table 11, p. 18).
  • Fixed times: switching helps at every fixed switch time, which raises the mean reward over continued On-policy by up to 1.58 points, and SR-GC reaches 40.02%, above nine of the ten fixed times; the best one (40.18%) is known only in hindsight (p. 8 l. 393–395; Tables 41–42, p. 41).

The sign threshold is zero in all runs. On OLMo-3 MATH, the gain of Switch comes from moving from gradient alignment to cached SR (§5.4, p. 8 l. 402–404).

Thank you for the detailed response and the substantial revision. The additional seeds across eight cohorts, the branched timing runs, the uniform-destination control and the later-endpoint results for seeds 3 and 4 are genuinely useful. The new related-work paragraph and the written-out abbreviations also improve the paper. The cross-cohort results now convincingly show that no fixed selector is safe. However, as far as I can tell, several central weaknesses remain. I would welcome a correction if I have misread any of the points below.

  1. SR-GC is not shown to add value over a fixed early switch. In the branched runs, SR-GC switches at update 50, the earliest point the rule allows, on four of five seeds. On those seeds its rewards are identical to the fixed switch at 50. R1 and R4 likewise coincide with the fixed switches at 25 and 75 on those seeds, and R2 always matches the current rule (Tables 41–42). The rule-variant analysis is therefore effectively a comparison of fixed switch times. Your own point that the gain does not depend on the exact switch time between updates 50 and 225 makes an online rule harder to motivate, not easier. The supporting diagnostics also don’t close this gap. The 7-of-9 sign agreement (App. B.1) is measured on a different contrast than Eq. (3), and a single check picks a strategy that ends 2.17 points below SR on average.

  2. The benefit over cached SR is not established relative to its cost. None of the eight Switch−SR intervals excludes zero, and outside OLMo-3 MATH the mean difference is +0.16 points. Meanwhile, Switch costs 8.8–60.7% more than SR in the Qwen and Gemma cohorts. On the branched runs, SR-GC minus SR from the shared state is not reported. Computed from Tables 41–42, it is +1.37 [−0.46, +3.19]. The equal-budget comparison is made only against On-policy, not against SR.

  3. The horizon remains unexplained. Saying that 275 = 25 + 250 does not explain why this horizon was chosen. It also leaves open why seeds 3 and 4 were compared at 275 when their runs continued to 310–315. This matters because the stage analysis locates the regime in which SR clearly wins at update 400.

  4. Several claims hold only for point estimates. For example, none of the ten fixed switch times in Table 41 has a paired interval against continued On-policy that excludes zero. Yet the abstract states that switching raises the mean reward at every tested time.

An equal-compute comparison with SR, a fixed-time switch baseline in cohorts where SR-GC switches late (e.g., Qwen3.5 MBPP), and rescoped claims would strengthen the paper considerably. In its current form, however, I cannot recommend acceptance and will keep my rating.

Reviewer_cyHHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates dynamic data selection in reinforcement learning with verifiable rewards (RLVR). It proposes a switching strategy that transitions from on-policy gradient alignment to cached success-rate selection using a gradient-based indicator (SR-GC). The method aims to improve training efficiency by adapting the selection strategy during learning. Experiments on mathematical reasoning and code generation tasks evaluate its effectiveness and computational costs.

Critical strengths
  1. The paper provides useful insights into how the effectiveness of data selection strategies changes during RLVR training, motivating adaptive switching.

  2. The switching experiments demonstrate improved final rewards over individual selection strategies while reducing computational costs compared with continued on-policy alignment.

Critical weaknesses
  1. The proposed SR-GC indicator lacks convincing justification, as its correlation with observed reward improvements is nearly zero. The switching rule also relies on heuristic conditions without sufficient validation.

  2. The online switching experiments are limited to two seeds on a single model and dataset. The modest gains over cached SR and incomplete cost accounting weaken the empirical evidence.

Questions for authors
  1. How reliably does SR-GC predict the better selection strategy, given its near-zero correlation with observed reward differences?

  2. Does the proposed switching rule outperform simpler fixed-time switching under matched computational budgets?

  3. How do the efficiency conclusions change when reward-cache construction and additional gradient-scoring costs are fully included?

  4. What evidence supports the generality of the switching mechanism beyond the two MATH training seeds?

2 comments
AuthorsOpenReview ↗

Response to Reviewer cyHH (part 1 of 2)

Thank you for the specific questions. Locations refer to the revised PDF.

W1/Q1. "How reliably does SR-GC predict the better selection strategy, given its near-zero correlation?"

SR-GC is used through its sign, and the sign picks the strategy with the higher final reward in 7 of 9 completed comparisons, more often than always choosing SR (5 of 9) or always choosing On-policy (4 of 9) (App. B.1, p. 17 l. 864–866; Table 8, p. 17). The rule does not read the size of the contrast as a predicted reward gap. The correlation of the contrast D with the On-policy-minus-SR reward difference is 0.058 (Pearson) and 0.033 (Spearman) (p. 17 l. 872–874).

A single check carries sampling noise, so the online rule confirms the sign over successive checks (§3.3, p. 4 l. 190–195):

  • Two independent reference samples at the same checkpoint have opposite signs in 4 of 9 states (p. 17 l. 870–872).
  • When one check at the start of a continuation fixes the selector for the whole run, the picked strategy ends 2.17 points below SR on average over six held-out states (App. B.2, p. 18 l. 927–929; Table 37, p. 39).
  • None of the 22 jackknife upper bounds of the seed 3/4 checks lies below zero (App. B.3, p. 19 l. 977–981).

The branched OLMo-3 MATH runs test the confirmed rule as a decision (Tables 42–43, pp. 41–42):

  • Switching after one negative check (R1) reaches 38.70%, against 40.02% for the current rule (current minus R1: +1.32 [−0.49, +3.12], 4/0/1 seeds).
  • Two consecutive negative checks (R2) give the same switch times as the current rule on all five seeds.
  • Three consecutive negative checks (R4) reach 40.38%, against 40.02% for the current rule (current minus R4: −0.37 [−2.51, +1.77], 2/0/3).

With the confirmed sign, SR-GC decides during training when to switch and finds a good switch time, with a reward close to the best ones (§5.4; next answer).

Q2. "Does the switching rule outperform simpler fixed-time switching under matched computational budgets?"

The best fixed switch time is known only in hindsight, and SR-GC comes close to it without that information. The branched OLMo-3 MATH runs compare SR-GC with every fixed switch time on the same trajectory of each seed (§5.4, p. 8 l. 386–404; Tables 41–43, pp. 41–42).

  • Fixed times: all ten switch times after updates 25–250 raise the mean reward over continued On-policy (38.61%), by up to 1.58 points (p. 8 l. 393–394; Table 41, p. 41). The best time differs from seed to seed, from update 50 to update 225. Only the two ends gain little (+0.04 after update 25, +0.07 after update 250). From update 50 to 225 the means lie within 38.92–40.18% and are not ordered by time, so the gain does not depend on hitting one exact time, and SR-GC lands in this range (40.02%). The fixed step with the largest gain on seeds 0–2 (update 25, +1.67; Table 5, p. 7) gives the lowest mean of the ten on seeds 5–9 (38.65%).
  • SR-GC: it reaches 40.02%, above nine of the ten fixed times, against 40.18% for the best one (after update 75), and it picks the per-seed best time on two of the five seeds (Tables 41–42, p. 41). At update 275 it ends 1.41 points above continued On-policy ([−0.60, +3.42], 4/0/1 seeds) (p. 8 l. 391–395; Table 43). Its threshold, confirmation length and interval were fixed before training (p. 4 l. 193–195).
  • Matched budget: the checks are charged to SR-GC. At an equal budget of 21.39 GPU-hours, with each run extended past update 275 as far as the budget allows, SR-GC ends 2.48 points [+1.28, +3.67] above continued On-policy on average, higher on all five seeds (p. 8 l. 398–399; App. E.1, p. 40 l. 2154–2159; Table 43). This contrast is against continued On-policy; we report no paired contrast with a single fixed time.
  • A cheaper trigger that reads only the training reward ends 0.69 points below SR-GC ([−1.03, +2.42]) (p. 8 l. 395–397).
  • Destination: from the SR-GC switch state, continuing with SR ends 1.49 points above uniform sampling ([−1.48, +4.46], 4/0/1) (p. 8 l. 401–402).
  • The fixed-step branches agree: switching to SR ends higher in 6 of 8 comparable pairs on seeds 0–2, by 1.11 points, and in 4 of 6 on seeds 3 and 4, by 1.18 points (§5.3, p. 7 l. 357–359; Table 5, p. 7; Table 37, p. 39).

On OLMo-3 MATH, the gain of Switch therefore comes from moving from gradient alignment to cached SR, and SR-GC supplies the switch time online (p. 8 l. 402–404; Table 43, p. 42 l. 2214–2225).

AuthorsOpenReview ↗

Response to Reviewer cyHH (part 2 of 2)

Q3. "How do the efficiency conclusions change when reward-cache construction and additional gradient-scoring costs are fully included?"

Gradient scoring at refreshes and checks is charged to Switch, and with it Switch costs less than On-policy on average in every cohort we timed. The reward cache is a one-time cost per model, dataset and seed. On OLMo-3 seed 4 it took 6.52 GPU-hours, more than the 4.48 GPU-hour saving of Switch over On-policy on seeds 3 and 4 (§6, p. 8 l. 412–417).

  • Scope: a cost covers the work from the shared starting state to the end of training, that is, rollouts, gradients and scoring at refreshes and checks, subset preparation and training. It excludes the shared prefix, which is common to all methods of a seed, reward-cache construction and evaluation (§4, p. 4 l. 211–212 and p. 5 l. 247–248; App. C.2, p. 23 l. 1228–1239).
  • OLMo-3 seeds 3 and 4: Switch 21.74 GPU-hours, On-policy 26.22 (Switch 17.1% less) and SR 19.20 (Switch 13.2% more) (§6, p. 8 l. 412–413; Figure 3, p. 9; Table 36, p. 37). On these seeds SR-GC was measured separately along the On-policy run, and the totals leave out these A/B measurements (Table 36 caption, p. 37 l. 1958–1960).
  • Reward cache: it is built once per model, dataset and seed, before the continuations that reuse it, and every SR and Switch run of that seed reads it (App. C.2, p. 23 l. 1237–1239). The 6.52 GPU-hours on seed 4 include one failed attempt (p. 8 l. 415–417).
  • Checks: in the Qwen runs the On-policy refresh does not score the SR comparison set, so each check scores it separately, and this cost is charged to Switch: 1.85 hours per check against 1.17 per refresh on MATH, and 1.99 against 1.41 on MBPP. Qwen MBPP seeds 5 and 6 switch after updates 200 and 250, and there Switch takes longer than continued On-policy (38.15 against 37.53 hours and 36.44 against 35.64) (p. 8 l. 431 – p. 9 l. 452; App. D.7, p. 33 l. 1746–1751). Averaged over seeds, Switch takes 11.0–21.9% less time than On-policy in all four Qwen and Gemma cohorts (Table 6, p. 8).
  • Llama: on average Switch lies between SR and On-policy. Four of its ten runs, which switch after updates 100, 175, 250 and 150, take longer than On-policy (p. 9 l. 456–459; Tables 32–33, p. 34).
  • OLMo-3 seeds 5–9: Switch spends 1.716 GPU-hours on candidate scoring on MATH and 1.895 on MBPP, against 6.191 and 6.206 for On-policy (Table 26, p. 31).
  • Branched runs: scoring the SR comparison set at each check is logged as a separate component and charged to SR-GC, whose mean cost of 21.39 GPU-hours sets the equal budget in Q2 (App. E.1, p. 40 l. 2154–2158).

Q4. "What evidence supports the generality of the switching mechanism beyond the two MATH training seeds?"

The revision tests Switch on five new seeds (5–9) in eight cohorts, four model families and two datasets, and the mean reward of Switch stays within 0.88 points of the best method in every cohort (§5.3, p. 7 l. 362–367; Tables 3–4, p. 6). It has the highest mean in five cohorts and the second-highest in two. Over all 40 seed pairs it ends above SR in 25, equal in 2 and below in 13, and above On-policy in 24, equal in 2 and below in 14 (p. 7 l. 368–369). On OLMo-3 MATH it is 1.57 points above SR (5/5 seeds, [−0.14, +3.28]) and 1.48 above On-policy (4/5, [−0.24, +3.21]) (p. 7 l. 370–372). With five seeds, the 16 paired intervals of Switch against SR or On-policy all include zero (p. 7 l. 367–368). Seeds 3 and 4 are reported separately from seeds 5–9, because there SR-GC was measured along the continued On-policy run (§4, p. 4 l. 211 – p. 5 l. 232).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers. Locations refer to the revised PDF: page (p.), margin line (l.), section (§), table and figure.

Experiments added since the submission. The submitted version covered OLMo-3 7B MATH: matched-update continuations on seeds 0–2, fixed-step branches on seeds 0–4 and online switching on seeds 3 and 4. The revision adds two experiments (Table 1, p. 5 l. 216–229):

  1. Five-seed online runs (seeds 5–9) of Random, On-policy, SR and Switch in eight model–dataset cohorts: OLMo-3 7B, Qwen3.5-9B, Gemma 4 12B IT and Llama-3.1-8B-Instruct, each trained on MATH and on MBPP. Each comparison is a paired mean difference with a two-sided 95% paired-t interval and the counts of positive, zero and negative seeds (§5.2, p. 6 l. 284–289; Tables 3–4, p. 6; §5.3, p. 7 l. 362–373; per-seed results in App. D.4–D.10, Tables 24–35, pp. 30–36).
  2. Branched timing runs on OLMo-3 MATH seeds 5–9. Each seed reruns one On-policy trajectory to update 275, saves the full training state and the SR-GC check every 25 updates, and starts a cached-SR branch from every saved state. All fixed switch times from 25 to 250, SR-GC rule variants, a training-reward trigger, a uniform-sampling destination and an equal-compute endpoint are compared on the same trajectory (§5.4, p. 8 l. 381–404; App. E, pp. 39–42, Tables 40–43).

The text was revised as well: a new introduction, a separate related-work section, a stated cost scope for every comparison, a new overview figure (Figure 1, p. 2) and revised figures (§1–2, pp. 1–3; §4 "Costs", p. 5 l. 247–251; App. C.2, p. 23 l. 1228–1239).

Main results.

  • No single selector is best throughout training. On OLMo-3 7B MATH, On-policy gives the larger gain over Random at the initial checkpoint and SR at the late one (§5.1, p. 5 l. 257–269; Table 2, p. 6).
  • The better selector also depends on the model and the dataset. SR minus On-policy is +1.24 points [+0.23, +2.25] on Qwen3.5 MATH and −2.20 [−3.35, −1.05] on Qwen3.5 MBPP (p. 6 l. 286–287; Table 4, p. 6).
  • Switching once from gradient alignment to cached SR is stable across settings. With the switch time chosen online by SR-GC, the mean reward of Switch stays within 0.88 points of the best method in all eight cohorts, while SR and On-policy fall up to 2.20 and 4.27 points below it. Switch has the highest mean in five cohorts and the second-highest in two (§5.3, p. 7 l. 362–366; §8, p. 10 l. 530–533).
  • Switch costs less on average than gradient alignment throughout: 17.1% less than On-policy on OLMo-3 seeds 3 and 4 and 11.0–21.9% less in the four Qwen and Gemma cohorts. It costs more than SR: by 13.2% on OLMo-3 seeds 3 and 4, by 8.8–25.4% in three of the Qwen and Gemma cohorts and by 60.7% on Qwen MBPP (§6, p. 8 l. 412–413 and l. 429–431; Table 6, p. 8).
  • Switching to cached SR helps, and SR-GC finds a good time to switch (§5.4). On the branched OLMo-3 MATH runs, switching raises the mean reward over continued On-policy (38.61%) at every tested switch time, by up to 1.58 points. The best fixed time is known only in hindsight and differs from seed to seed, and from update 50 to 225 the means lie within 38.92–40.18%, so the gain does not depend on one exact time (Table 41, p. 41). SR-GC, an online rule that decides during training when to switch, has no such information and reaches 40.02%, above nine of the ten fixed times, against 40.18% for the best one. At update 275 it ends 1.41 points above continued On-policy ([−0.60, +3.42], 4/0/1 seeds). At an equal budget of 21.39 GPU-hours, with each run extended past update 275 as far as the budget allows, it ends 2.48 points above continued On-policy on average, higher on all five seeds ([+1.28, +3.67]). From the SR-GC switch state, continuing with SR ends 1.49 points above continuing with uniform sampling ([−1.48, +4.46], 4/0/1) (§5.4, p. 8 l. 393–404; Tables 41–43, pp. 41–42; Abstract, p. 1 l. 21–24; §7, p. 10 l. 516–518).

Scope. The branched timing runs use OLMo-3 MATH, and the switch is one-way (§3.3, p. 4 l. 197–199; §7, p. 10 l. 520–523).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.