acceptodds
Under review as a conference paper at ICLR 2027

Learning-Zone Energy: Online Data Selection for Efficient RL Post-Training

Abstract

Reinforcement Learning (RL) post-training has emerged as the dominant paradigm for eliciting mathematical reasoning in Large Language Models (LLMs), yet prevailing techniques such as GRPO and DAPO distribute rollout and gradient budgets nearly uniformly across prompts, squandering compute on samples that are already mastered or remain far beyond the model's current capability. To address this fundamental inefficiency, we propose **Learning-Zone Energy** (**LZE**), a theoretically grounded, fully online data selection framework that concentrates computation on the model's *active learning frontier*. At its core, we define a closed-form *Learning-Zone Energy Score* that combines initial difficulty, current outcome uncertainty, and pass-rate momentum. An analysis of binary group-relative rewards motivates the uncertainty term, while component ablations assess the contribution of the full score. A forward pruner with replay further reduces wall-clock time cost by skipping rollout generation for persistently solved prompts while periodically checking for forgetting. Evaluated on Qwen-family models (1.5B–8B) across GSM8K, MATH and DAPO-MATH, our method retains only 40% of the training data per step yet matches or surpasses full-data baselines, with especially pronounced out-of-distribution gains on AIME25(+45.9%) and AMC23(+18.2%), alongside an estimated 36% reduction in training FLOPs. Our code is available at https://anonymous.4open.science/r/LZE.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, LLM Post-Training, Data-Centric AI, Online Data Selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fepWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an online data selection rule for group-based reinforcement learning post-training. For each prompt group it computes a closed-form score that multiplies three factors, namely an anchor for how hard the prompt was under the initial policy, a normalized measure of how mixed the current group outcomes are, and a momentum term tracking recent changes in pass rate. Only the top fraction of groups by this score receive policy updates. A second stage skips rollout generation for prompts that stay solved across consecutive epochs, with periodic replay to detect forgetting. On Qwen models from 1.5B to 8B across three math corpora, retaining 40 percent of prompt groups per step matches or exceeds full-data training while reducing estimated training floating-point operations.

Critical strengths
  1. The selection rule is cheap and needs no auxiliary model, which is a genuine advantage over the alternatives it is compared against. It reuses pass-rate statistics the rollout phase already produces and leaves the optimizer untouched. The identity it rests on is also correct. For binary rewards centered by the group mean, the mean squared advantage equals the empirical pass rate times its complement, so degenerate groups contribute nothing. I checked this and the companion expression for the probability of a non-degenerate group.

  2. The ablations ask the questions that matter. Comparing against uniform sampling at the same retention ratio separates the selection rule from the effect of simply training on less data, and the paper reports that uniform sampling falls below the full-data baseline. Sweeping the rollout count under a fixed total budget is also well posed, and the explanation of the interior optimum is coherent.

Critical weaknesses
  1. The headline gains correspond to a handful of problems, and no measure of run-to-run variation appears anywhere in the paper. The abstract leads with relative improvements of 45.9 percent on AIME25 and 18.2 percent on AMC23, which in absolute terms are 5.6 and 10.0 points. The reported values are quantized, with AIME25 on multiples of one ninetieth and AMC23 on multiples of one fortieth, so the evaluation appears to be three samples over thirty problems and one sample over forty. The 45.9 percent figure is then 5 additional correct answers out of 90 attempts, and the 18.2 percent figure is 4 additional problems out of 40. Several other starred entries are a single problem. With no seeds, no standard deviations, and no significance testing, I cannot distinguish these from sampling noise, and reporting them as relative percentages makes them look an order of magnitude more substantial than they are. For one of the eight configurations the method is also worse on three of five benchmarks with the average essentially unchanged, yet the text claims consistency across all scales and families.

  2. The theoretical analysis does not support the conclusion it is cited for. The appendix result is a correct but elementary conditional second-moment computation. It is maximized at a pass rate of one half only when the two reward-conditioned score norms are approximately equal, which the paper notes but does not verify. More importantly, the quantity being maximized is the second moment of the gradient estimator, and the paper's own covariance identity shows that selecting large values of it selects high-variance groups. Equating a large second moment with informativeness, without relating either to expected policy improvement, leaves the criterion without the grounding claimed for it. The main text then asserts that the out-of-distribution gains follow from this result by preventing overfitting to the in-distribution difficulty profile. Nothing in the result concerns generalization, and that sentence ought to be removed.

Questions for authors
  1. Can the main table be rerun with at least three seeds, reporting absolute accuracies with standard deviations alongside the relative figures, and stating the number of evaluation samples per benchmark?

  2. What does the method achieve if the score is replaced by the uncertainty factor alone, or by simply dropping degenerate groups and sampling uniformly among the rest? With eight rollouts that factor takes only four distinct nonzero values, so this isolates whether the graded ranking adds anything.

  3. Why does the binary rejection baseline fall below the full-data baseline on both models? That ordering is unexpected.

  4. Since rollouts still cover the full active pool, the saving applies to the backward pass only. What fraction of baseline wall-clock time is generation, and what is the end-to-end saving of the selector alone?

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

We thank the reviewer for carefully examining the theoretical formulation and recognizing the validity of the centered-reward analysis, the lightweight design, and the controlled ablations. We address the concerns below.

W1/Q1. Statistical reliability and interpretation of experimental gains

We thank the reviewer for raising this question. Nevertheless, the review overlooks the broader experimental evidence and contains an incorrect characterization of Table 1.

First, all eight training configurations in Table 1 achieve positive improvements in average accuracy. The review highlights the relatively small gains on individual benchmarks, while several configurations exhibit substantially larger improvements:

  • +4.0 percentage points: Qwen2.5-Math-1.5B trained on MATH (0.469 → 0.509).

  • +3.9 percentage points: Qwen3-8B trained on DAPO-MATH (0.716 → 0.755).

  • +3.4 percentage points: Qwen3-1.7B trained on GSM8K (0.548 → 0.582).

The review further states that one configuration deteriorates on three of five benchmarks. This is inconsistent with the reported results. For Qwen3-4B trained on DAPO-MATH, LZE improves GSM8K, MATH, and AIME25, while two benchmarks decrease. Its average accuracy increases from 0.764 to 0.766.

Second, Table 1 explicitly reports absolute accuracies alongside relative improvements. The highlighted gains on AIME25 (0.122 → 0.178) and AMC23 (0.550 → 0.650) correspond to 5.6 and 10.0 percentage points, respectively. These are substantial improvements on challenging mathematical reasoning benchmarks.

Additional Avg@4 evaluation. To further examine OOD performance stability, we evaluate each model using four independent responses per question. The evaluation covers all 40 AMC23 problems and 30 AIME25 problems, corresponding to 160 and 120 responses per model, respectively.

Training SetBackboneAMC23 VanillaAMC23 LZEAIME25 VanillaAIME25 LZE
GSM8KQwen2.5-Math-1.5B0.5880.5810.1000.142
GSM8KQwen3-1.7B0.5880.6940.2000.275
MATHQwen2.5-Math-1.5B0.5380.6440.1000.167
MATHQwen2.5-Math-7B0.6810.7190.3000.283
MATHQwen3-4B0.8880.8880.5250.533
DAPO-MATHQwen2.5-Math-7B0.7880.8190.3420.375
DAPO-MATHQwen3-4B0.9130.8940.5250.550
DAPO-MATHQwen3-8B0.8190.9380.4670.517

Independent training runs. We additionally conduct three independent LZE runs on two representative MATH training configurations. The multi-seed runs demonstrate stable LZE performance.

Together, the eight positive average improvements, additional Avg@4 evaluation, and independent-run statistics collectively provide a broader and more informative assessment of the method than isolated differences on individual benchmarks.

Training settingVanilla-RL MATH (3 seeds)LZE MATH (3 seeds)Vanilla-RL OOD Avg (3 seeds)LZE OOD Avg (3 seeds)
Qwen2.5-Math-1.5B0.581 ± 0.0140.587 ± 0.0050.336 ± 0.0200.414 ± 0.007
Qwen2.5-Math-7B0.618 ± 0.0100.630 ± 0.0090.486 ± 0.0120.517 ± 0.008
AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

W2. Theoretical grounding and gradient informativeness

The theoretical analysis rigorously characterizes reward-derived learning signals, while experiments establish the practical effectiveness of the resulting selection criterion.

The foundation of LZE is the exact identity in Eq. (6):

1n∑k=1nA~i,k2=p^i(1−p^i).\frac{1}{n}\sum_{k=1}^{n}\tilde A_{i,k}^{2}=\hat p_i(1-\hat p_i).

This establishes that all-correct and all-incorrect groups yield zero centered reward advantages, while mixed-outcome groups provide non-degenerate learning signals. Furthermore, Eq. (7) shows that the probability of obtaining a mixed-outcome group is maximized when the underlying success probability is 0.5. These results directly motivate prioritizing prompts near the policy's current learning frontier.

Appendix G.1 extends this analysis to policy-gradient statistics. Theorem 1 explicitly incorporates reward-conditioned score norms into the second-moment decomposition, while Eq. (26) characterizes the expected empirical group-gradient estimator. The second moment captures both the magnitude and stochastic variability of gradient contributions; the exact optimization benefit additionally depends on gradient geometry and policy dynamics. This motivates our use of observable reward variation as a practical criterion for allocating training computation.

Computational efficiency is central to this design choice. Estimating the expected improvement of each individual prompt through explicit gradient evaluation, counterfactual policy updates, or post-update validation would introduce substantial overhead into online selection. In contrast, (p(1−p)p(1-p)) is computed directly from existing rollout rewards, with negligible additional cost. LZE combines this inexpensive signal with historical difficulty and learning dynamics to guide policy updates efficiently.

The effectiveness of this criterion is directly demonstrated by experiments. Figure 3(a) shows that removing the uncertainty component causes the largest performance degradation. Figure 3(b) validates LZE across different rollout counts under a fixed sampling budget. Table 2 further shows that LZE outperforms uniform sampling at the same 40% retention ratio on both evaluated backbones.

The theoretical analysis thus establishes the motivation for the selection criterion, while the empirical results in Tables 1, 2, and 5 demonstrate its effectiveness, including the observed OOD improvements. We will further polish the corresponding discussion in the camera-ready version to make this connection more explicit.

Q2. Uncertainty-only selection and graded ranking

The uncertainty-only comparison requested by the reviewer is already included in Figure 3(a).

The component ablation evaluates four configurations: the baseline, uncertainty-only selection, uncertainty with the initial-difficulty anchor, and the complete LZE score.

Figure 3(a) shows that uncertainty provides the strongest individual contribution, while incorporating difficulty history and momentum yields additional improvements. This directly establishes the value of the full scoring formulation.

Furthermore, the distinction between graded ranking and binary rejection is straightforward. A binary rule assigns all non-degenerate groups the same selection eligibility. Uncertainty-based ranking further differentiates groups according to their current reward variation. The complete LZE score additionally distinguishes prompts with identical current pass rates through their initial difficulty and recent learning dynamics.

The resulting hierarchy is directly reflected in Figure 3(a). Each component contributes to the final selection behavior, with the complete LZE formulation achieving the strongest performance in the reported component ablations.

Appendix F.1 further demonstrates the importance of stochastic exploration. Gumbel-Top-K improves OOD performance by 2.1%, 13.7%, and 17.3% across three evaluated backbones compared with deterministic Top-K.

Together, these results establish the value of progressively refining selection priorities through current uncertainty, historical difficulty, and learning dynamics.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

Q3. Why binary rejection can underperform full-data training

Binary rejection and LZE employ different selection granularity and optimization mechanisms, which explains their different empirical behavior.

Reinforce-Rej uses a binary filtering criterion to remove groups with uniformly correct or incorrect outcomes, together with a REINFORCE-style optimization procedure. LZE retains the original group-relative policy optimization and applies a continuous Energy Score to prioritize updates among sampled prompt groups.

Binary rejection treats all remaining non-degenerate groups equally, despite substantial differences in their reward variation and learning histories. LZE further ranks these groups using current uncertainty, initial difficulty, and momentum, allocating the available gradient budget toward the most informative region of the current policy.

The component ablations in Figure 3(a), together with the same-retention-ratio uniform sampling comparison, further substantiate the practical value of allocating updates according to the learning-zone criterion.

Q4. Computational savings and wall-clock efficiency

The 36% estimated FLOPs reduction already accounts for full rollout coverage, while Figure 2 directly reports end-to-end wall-clock savings from backward selection alone.

Appendix E decomposes the computational footprint into inference/scoring and policy optimization, with theoretical coefficients of 4 and 6, respectively.

LZE retains full rollout coverage during backward selection and performs policy optimization on only 40% of prompt groups. The resulting theoretical FLOPs ratio is

4+0.4×64+6=0.64,\frac{4+0.4\times6}{4+6}=0.64,

corresponding to a 36% reduction in estimated total training FLOPs.

More importantly, Figure 2 already provides measured end-to-end training times for backward selection alone:

ModelVanilla-RLLZEWall-clock reduction
Qwen2.5-Math-1.5B3.97 h2.40 h39.5%
Qwen2.5-Math-7B4.92 h3.51 h28.7%

These correspond to 1.65× and 1.40× speedups, respectively. The optional forward pruner provides further improvements, reaching 1.80× and 1.91× overall speedups. The baseline wall-clock fraction attributable to rollout generation is approxiamately around 30% for Qwen2.5-Math-1.5B and Qwen2.5-Math-7B.

The analytical computation model and measured wall-clock results consistently demonstrate the efficiency of LZE. In particular, the selector alone provides substantial end-to-end savings, with additional benefits available from the forward-pruning mechanism.

Dear Reviewer fepW,

As the discussion period is drawing to a close, we wanted to kindly follow up and ask whether our response has addressed your concerns. If there are any remaining questions or points that you feel have not been sufficiently addressed, we would sincerely appreciate the opportunity to clarify them before the discussion period ends.

Thank you again for your time, thoughtful feedback, and consideration.

Reviewer_hGzzOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Learning-Zone Energy (LZE), a lightweight, fully online data selection method for RL post-training of LLMs on math reasoning. At each training step, LZE scores every prompt group with a closed-form score that multiplies an initial-difficulty anchor, a normalized uncertainty term 4p(1−p) based on the current pass rate, and a pass-rate momentum term, then sends only the top 40% to the policy update. A forward pruner with periodic replay also skips rollouts for prompts that have stayed fully solved. Across Qwen2.5 and Qwen3 models (1.5B–8B) trained on GSM8K, MATH, and DAPO-MATH, LZE matches or beats full-data GRPO baselines.

Critical strengths
  • The proposed method is simple and cheap.
  • It is compatible with a lot of existing RLVR algorithms.
  • The uncertainty term is well-motivated.
  • The experiment covers different model sizes.
Critical weaknesses
  • Hyperparameters are tuned on the reported test benchmarks. The hyperparameters are selected based on AMC23 and AIME25 performance which are also the reported benchmarks.
  • The Gumbel perturbation along changes the OOD average by 13.7% and 17.3% in Table 3. A component this minor producing gains of that size suggests high variance in the results.
  • The paper says the OOD gains are “consistent with the theoretical prediction of Theorem 1,” but Theorem 1 is a second-moment decomposition and says nothing about generalization or overfitting.
  • Novelty is limited. Filtering degenerate groups is already standard (DAPO’s dynamic sampling, Reinforce-Rej), and targeting prompts with pass rates near 0.5 appears in DOTS, AdaRFT.
  • The baselines are a bit too weak. There are much more recent curriculum learning baselines including PCL [1] and GRESO [2].

[1] Prompt Curriculum Learning for Efficient LLM Post-Training

[2] Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts

Questions for authors

Please refer to the weaknesses.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We thank the reviewer for recognizing the efficiency, compatibility, and theoretical motivation of LZE. We address the concerns below.

W1. Hyperparameter selection and evaluation protocol

The default hyperparameters were determined using training-time metrics on two Qwen2.5-Math development settings and subsequently applied consistently across all main experiments.

Specifically, we conducted controlled hyperparameter sweeps on Qwen2.5-Math-1.5B and Qwen2.5-Math-7B, considering multiple training-time indicators, including verifier reward progression, convergence behavior, and training stability. Based on these observations, we established (κ=0.4\kappa=0.4) and (α=0.3\alpha=0.3) as the shared default configuration. As explicitly stated in Sec. 5.1, these values were fixed across all main experimental settings, including the subsequent Qwen3 experiments.

Figure 3(c–d) provides additional sensitivity evidence supporting the selected configuration. The reported AMC23/AIME25 results characterize how different values of (κ\kappa) and (α\alpha) affect OOD performance. The favorable results at the default settings corroborate our training-time observations. The phrase "we adopt it as the default" in Sec. 5.2 summarizes the configuration used throughout the paper; the hyperparameter were fixed before evaluating AMC23 and AIME25.

Ablation on hyperparameter is standard practice in RL research. We also tuned the baseline configurations to ensure competitive comparisons. Importantly, the same LZE configuration achieves positive average accuracy gains across all eight main experimental settings, providing strong evidence for its effectiveness across model scales and training datasets.

We will polish our statement in the camera-ready version so as to avoid misunderstanding.

W2. Gumbel perturbation and experimental stability

Gumbel perturbation directly influences discrete Top-K selection, where small score perturbations can meaningfully improve exploration.

Appendix F.1 already provides a detailed analysis of this mechanism. Since pass rates are estimated from a finite number of rollouts, many prompt groups have identical or closely spaced Energy Scores. Deterministic Top-K selection can repeatedly prioritize similar prompt subsets, whereas Gumbel perturbation encourages exploration among candidates with comparable informativeness.

The selection behavior is mathematically characterized in Eq. (16):

P(\tilde E_j>\tilde E_i)=\frac{1}{1+\\exp(\Delta/\tau)}.

Consequently, even a small perturbation can appreciably alter selection probabilities when energy gaps are small. This explains how the lightweight Gumbel mechanism can produce meaningful downstream gains.

Appendix Table 3 reports OOD improvements of 2.1%, 13.7%, and 17.3% across three backbones relative to deterministic Top-K. The consistent positive effects demonstrate the contribution of stochastic exploration.

We additionally conducted three independent LZE training runs on two representative MATH settings. The repeated runs demonstrate stable performance. These findings further support the reliability of the complete LZE framework.

Training settingVanilla-RL MATH (3 seeds)LZE MATH (3 seeds)Vanilla-RL OOD Avg (3 seeds)LZE OOD Avg (3 seeds)
Qwen2.5-Math-1.5B0.581 ± 0.0140.587 ± 0.0050.336 ± 0.0200.414 ± 0.007
Qwen2.5-Math-7B0.618 ± 0.0100.630 ± 0.0090.486 ± 0.0120.517 ± 0.008

W3. Theoretical analysis and OOD generalization

The theoretical analysis provides the foundation for LZE's selection mechanism, while the observed OOD improvements are supported by extensive empirical evidence.

Equation (6) establishes the exact relationship between centered binary reward variation and (p(1−p)p(1-p)), while Theorem 1 further characterizes the corresponding score-function second moment, explicitly accounting for reward-conditioned gradient norms. These results motivate allocating policy updates toward prompt groups with informative relative reward signals.

As already elaborated in Sec. 5.1, this mechanism provides a natural explanation for the observed OOD improvements. By deprioritizing trivially solved and currently unreachable prompts, LZE concentrates training on the policy's active learning frontier, reducing excessive optimization on the extremes of the in-distribution difficulty profile. The effectiveness of this allocation is supported empirically by Tables 1, 2, and 5, as well as the component ablations in Fig. 3(a).

Our use of “consistent with” connects this theoretically motivated selection mechanism with the observed generalization behavior. We will polish the relevant wording in the camera-ready version to make the transition between the theoretical characterization and empirical interpretation more explicit. This clarification preserves the original motivation, supporting evidence, and conclusions.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W4. Novelty and methodological contributions

LZE advances online RL data selection through a unified, closed-form scoring criterion that combines current gradient informativeness, historical difficulty, and learning dynamics.

The proposed framework makes several concrete contributions beyond existing data-filtering approaches.

First, LZE enables continuous prioritization of prompt groups. DAPO's dynamic sampling and Reinforce-Rej employ binary rejection rules that discard degenerate groups. LZE further distinguishes the remaining groups according to their current reward variation, allowing differentiated allocation of policy updates.

Second, LZE integrates three complementary signals into a single online criterion. The uncertainty term is grounded in the centered-reward analysis, the initial-difficulty anchor preserves historical information, and the momentum term captures recent changes in policy capability. Figure 3(a) directly demonstrates the incremental contributions of these components.

Third, LZE requires no auxiliary predictive model. Methods such as DOTS and DPS introduce additional mechanisms for estimating or predicting prompt difficulty and solvability. LZE directly calculates its Energy Score from rollout statistics already produced by the RL training process.

Fourth, LZE combines backward selection with replay-based forward pruning. Backward selection concentrates gradient computation on high-value groups, while the optional forward pruner reduces redundant rollout generation. These complementary mechanisms jointly improve computational efficiency.

Together, these contributions establish a distinct and practically effective framework for online RL data selection.

W5. Baseline strength and additional comparisons

The review substantially understates the strength and breadth of the existing baseline evaluation.

Our paper already evaluates ten data-selection configurations, covering uniform sampling, offline pre-filtering, binary rejection, SFT-based selection, adaptive curriculum learning, and recent online data-selection methods.

Importantly, these include established and recently published approaches:

  • DOTS (NeurIPS 2025): Difficulty-targeted online data selection with learned difficulty estimation and rollout replay.

  • DPS (ICLR 2026): Dynamics-predictive sampling using a hidden Markov model to estimate prompt solvability before rollout.

  • AdaRFT (TMLR 2026): Adaptive curriculum reinforcement fine-tuning based on dynamically adjusted problem difficulty.

  • RAFT (TMLR 2023): Reward-ranked fine-tuning, an established baseline for reward-based language model optimization.

These methods represent substantial research contributions and directly address efficient training, online data selection, and adaptive sampling. Our comparisons therefore already cover a broad range of competitive approaches from the recent literature.

Moreover, Appendix Table 5 demonstrates that several of these methods achieve strong improvements in specific settings. For example, DOTS achieves a 33.2% relative OOD improvement on Qwen2.5-Math-1.5B, while AdaRFT-PPO achieves 13.3%. LZE achieves positive OOD improvements on both evaluated backbones, demonstrating consistent effectiveness across model scales.

We additionally evaluate the two methods suggested by the reviewer, PCL and GRESO, under the Qwen2.5-Math training settings:

BackboneMethodMATH ↑OOD Avg ↑
Qwen2.5-Math-1.5BVanilla-RL (3 seeds)0.581 ± 0.0140.336 ± 0.020
PCL0.5720.341
GRESO0.5610.329
LZE (3 seeds)0.587 ± 0.0050.414 ± 0.007
Qwen2.5-Math-7BVanilla-RL (3 seeds)0.618 ± 0.0100.486 ± 0.012
PCL0.6110.501
GRESO0.6170.482
LZE (3 seeds)0.630 ± 0.0090.517 ± 0.008

LZE achieves the strongest OOD performance on both backbones. PCL obtains modest OOD improvements over Vanilla-RL, whereas GRESO exhibits slightly lower accuracy in these configurations. LZE achieves more favorable results across both in-domain and OOD evaluation. These additional comparisons further strengthen an already extensive baseline evaluation. Together with the component ablations, theoretical analysis, and measured efficiency improvements, they provide clear empirical support for the effectiveness and practical advantages of LZE.

I thank the authors for the detailed response.

W2

I agree that ties are common, but the perturbation is not small at this scale. The multi-seed runs compare full LZE with Vanilla-RL, so they do not address the Gumbel ablation. They also create a tension. If seed-level standard deviations of about 0.01–0.02 in OOD Avg are representative, the Table 3 differences are unlikely to be noise.

W3

Rewording that "preserves the original motivation, supporting evidence, and conclusions" would not resolve the issue. Theorem 1 is a second-moment decomposition and makes no prediction about generalization.

W4

I agree that not needing an auxiliary model is a real advantage over DOTS and DPS. My concern is whether graded ranking adds value beyond filtering degenerate groups, which is the main difference from DAPO's dynamic sampling and Reinforce-Rej. The current evidence does not isolate this.

W5

Thank you for adding PCL and GRESO. Please move these "established and recently published approaches" from the appendix into the main paper. The baselines in the main section of the paper is too weak.

Given that my concerns largely remain, I will maintain the current score.

Reviewer_E6V4OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LZE, an online data selection method that multiplies initial difficulty, uncertainty in the current success rate, and pass-rate momentum to score prompt groups for RL training with binary verifiable rewards. Approximately 40% of groups are retained for policy updates. Optional forward pruning skips rollout generation for prompts that have been consistently solved, with replay used to detect forgetting. Experiments cover five Qwen models ranging from 1.5B to 8B parameters and three mathematical training datasets, reporting accuracy comparable to or better than full-data training, together with savings in FLOPs and wall-clock time.

Critical strengths
  1. The selection mechanism is lightweight and practically useful. LZE uses success rates and historical statistics from existing rollouts without training an additional difficulty predictor. Under the paper's within-group reward-centering setup, all-success and all-failure groups provide no reward-derived policy-gradient signal, giving a reasonable motivation for prioritizing mixed-outcome groups. Backward selection and forward pruning target optimization and generation costs, respectively.

  2. Wall-clock savings across several training settings. Experiments cover multiple model sizes and three training datasets, including comparisons with uniform sampling at the same retention ratio and recent online selection methods. With backward selection alone, training time decreases from 3.97 to 2.40 hours for the 1.5B model and from 4.92 to 3.51 hours for the 7B model, supporting the method's practical value. (Figure 2, Appendix F.3)

Critical weaknesses
  1. The method appears to provide limited control over data diversity. In addition to data quality, diversity is an important factor in training effectiveness, but LZE mainly relies on Gumbel perturbations to randomize selection. Could the authors add controlled comparisons incorporating diversity modules from other methods to establish the advantages of the proposed sampling strategy?

  2. The fixed initial-difficulty weight may limit online adaptation. Equation (5) uses the initial difficulty Di(0)=1−p^i(0)D_i^{(0)} = 1 - \hat{p}_i^{(0)} as a fixed multiplicative weight. As a result, a prompt with an initial pass rate of 100% will still have a zero energy score before perturbation, even if later rollouts contain both correct and incorrect answers. Gumbel perturbations allow these prompts to be selected again, but cannot use their current rewards to prioritize the more informative prompts among them; replay in Algorithm 1 does not refresh this weight either. The paper does not establish whether Gumbel exploration gives these forgotten prompts enough corrective updates.

  3. The experimental results lack statistical validation. The main results in Table 1 do not report means, standard deviations, or confidence intervals across independent training seeds. For example, for Qwen3-4B trained on DAPO-MATH, the average accuracies of LZE and the full-data baseline are 76.6% and 76.4%, respectively, a difference of only 0.2 percentage points. Without independent repeated runs, it is difficult to determine whether such small gains are consistent or simply due to random variation.

Questions for authors
  1. Does the difficulty anchor in Equation (5) remain fixed after replay, and is there a lower bound or a refresh mechanism? Could the authors report selection frequencies and success-rate changes for prompts that are initially solved in all rollouts but later forgotten, to establish whether Gumbel exploration is sufficient for recovery?

  2. Why do all other data selection methods in Table 5 of the appendix reduce OOD performance relative to Vanilla-RL on the 7B model, while LZE improves it? Could the authors analyze the reasons?

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We thank the reviewer for recognizing the lightweight design of LZE, its principled motivation, and the substantial wall-clock efficiency improvements. We address the concerns below.

W1. Data diversity and gradient informativeness

LZE incorporates adaptive prompt selection and stochastic exploration, with extensive comparisons against existing online data selection methods.

Importantly, prompt-level selection diversity and response-level rollout diversity concern different aspects of RL training. The former determines which problems participate in policy updates and how broadly the training distribution is covered, whereas the latter concerns the diversity of generated responses for a given prompt. LZE operates at the prompt-selection level, making comparisons with existing prompt-selection strategies particularly relevant.

Our evaluation already includes DPS (ICLR 2026), DOTS (NeurIPS 2025), AdaRFT, and other competitive methods employing different prompt-selection and adaptation mechanisms. Appendix Table 5 shows that LZE achieves positive OOD gains on both 1.5B and 7B backbones (+23.2% and +5.9%), demonstrating its effectiveness across model scales.

Within LZE, Gumbel-Top-K introduces stochastic exploration across prompt groups, while Energy Scores are recomputed from current rollout statistics at every training step. Together, these mechanisms allow the selected prompt distribution to adapt as the policy evolves and promote exploration beyond deterministic Top-K selection.

Appendix F.1 directly validates the contribution of stochastic exploration: Gumbel perturbation improves OOD performance by 2.1%, 13.7%, and 17.3% across three backbones. Combined with the same-retention-ratio uniform sampling comparison and extensive online selection baselines, these results provide strong evidence for the effectiveness of LZE's adaptive prompt-selection strategy.

W2/Q1. Fixed difficulty anchor and recovery from forgetting

The fixed difficulty anchor preserves historical difficulty information, while Gumbel exploration and replay provide mechanisms for revisiting previously mastered prompts.

Equation (5) combines initial difficulty with current outcome uncertainty and pass-rate momentum. The difficulty anchor retains information about the initial learning landscape, while the latter two terms continuously reflect the evolving policy.

The Gumbel perturbation in Eq. (14) provides every active prompt with a positive probability of selection, including prompts with zero unperturbed energy. Meanwhile, Algorithm 1 periodically re-evaluates pruned prompts and returns those exhibiting reduced pass rates to the active pool. These mechanisms enable continued exploration and recovery as the policy evolves.

We additionally quantify the practical frequency of the specific case raised in the review: prompts that are initially solved in all rollouts but subsequently experience a pass-rate decline to 50% or below.

BackboneFraction of all training prompts
Qwen2.5-Math-1.5B1.1%
Qwen2.5-Math-7B0.3%
Qwen3-4B0.3%

Such cases are rare across all three backbones, decreasing from 1.1% on the 1.5B model to 0.3% on the larger models. The fixed-anchor boundary case is therefore uncommon in the evaluated training pool, substantially limiting its empirical prevalence.

Furthermore, Appendix F.1 directly evaluates the effect of stochastic exploration, demonstrating consistent OOD gains across all three backbones. These results support the effectiveness of the existing exploration mechanism in the overall LZE framework.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W3. Statistical validation and consistency of improvements

We thank the reviewer for raising this question. Nevertheless, the substantially larger improvements occur across the remaining experimental configurations. Table 1 reports positive average accuracy gains in all eight training configurations. The cited 0.2-percentage-point improvement on Qwen3-4B trained with DAPO-MATH is the smallest among them. Other configurations exhibit considerably larger gains, including:

  • +4.0 percentage points: Qwen2.5-Math-1.5B trained on MATH (0.469 → 0.509).

  • +3.9 percentage points: Qwen3-8B trained on DAPO-MATH (0.716 → 0.755).

  • +3.4 percentage points: Qwen3-1.7B trained on GSM8K (0.548 → 0.582).

The evidence should be assessed across the complete experimental matrix, where the improvements are broad and consistent. We additionally conduct three independent runs of LZE on two representative MATH training configurations:

Training settingVanilla-RL MATH (3 seeds)LZE MATH (3 seeds)Vanilla-RL OOD Avg (3 seeds)LZE OOD Avg (3 seeds)
Qwen2.5-Math-1.5B0.581 ± 0.0140.587 ± 0.0050.336 ± 0.0200.414 ± 0.007
Qwen2.5-Math-7B0.618 ± 0.0100.630 ± 0.0090.486 ± 0.0120.517 ± 0.008

The three-seed results show stable performance, with significant OOD improvements over the corresponding Vanilla-RL reference results. The relatively small standard deviations further demonstrate the consistency of LZE across independent training runs. Taken together, the results in Table 1 and the additional repeated experiments provide substantial evidence for the effectiveness of LZE across different model sizes and training configurations.

Q2. Understanding the OOD results of different selection methods

The contrasting results highlight the importance of selecting prompts according to the current policy's learning frontier.

Appendix Table 5 shows that several alternative selection methods improve OOD performance on the 1.5B model, while their benefits become less consistent on the 7B model. For example, AdaRFT-PPO achieves +13.3% OOD improvement on 1.5B but −12.0% on 7B; DOTS changes from +33.2% to −10.3%; and DPS changes from +18.2% to −15.3%. In contrast, LZE achieves positive OOD gains on both backbones (+23.2% and +5.9%).

This pattern is consistent with differences in how selection criteria respond to the evolving policy. Approaches based on difficulty schedules, predicted solvability, or alternative filtering objectives may allocate updates to prompt groups whose usefulness changes substantially as model capability increases.

LZE directly evaluates the current rollout distribution through (p(1−p)p(1-p)), prioritizing prompt groups with substantial centered reward variation. The difficulty anchor and momentum further incorporate historical challenge and recent learning dynamics. Figure 3a supports this interpretation: removing the uncertainty component produces the largest performance degradation, while the full combination achieves the strongest OOD results.

As model capability changes, the active learning frontier also shifts. LZE continuously updates its selection scores using observed rollout outcomes, enabling the training distribution to follow this evolving frontier. This provides a coherent explanation for its consistent positive OOD gains across the two model scales.

We hope these clarifications and additional experimental results address the reviewer's concerns, and would appreciate reconsideration of the rating accordingly.

Thank you for the clarifications. With LZE’s Energy Score held fixed, Appendix F.1 compares Gumbel Top‑K only against the simplest baseline, deterministic Top‑K. Established selection methods such as DEPO’s[1] weighted DPP could be adapted to use LZE’s Energy Score. I would like to see these comparisons, along with a separate ablation of perturbation strength. Given that even a small Gumbel perturbation yielded relative gains of 13.7% and 17.3% in average OOD performance on two models, I think further evaluation is warranted, as the current experiments do not fully establish the contribution of Gumbel Top‑K.

Given that most of my concerns have been addressed, I have decided to raise my score. I still believe that additional ablations of the sampling strategy would make the paper more convincing.

[1] Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward. ICLR26

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.