- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an online data selection rule for group-based reinforcement learning post-training. For each prompt group it computes a closed-form score that multiplies three factors, namely an anchor for how hard the prompt was under the initial policy, a normalized measure of how mixed the current group outcomes are, and a momentum term tracking recent changes in pass rate. Only the top fraction of groups by this score receive policy updates. A second stage skips rollout generation for prompts that stay solved across consecutive epochs, with periodic replay to detect forgetting. On Qwen models from 1.5B to 8B across three math corpora, retaining 40 percent of prompt groups per step matches or exceeds full-data training while reducing estimated training floating-point operations.
Critical strengths
-
The selection rule is cheap and needs no auxiliary model, which is a genuine advantage over the alternatives it is compared against. It reuses pass-rate statistics the rollout phase already produces and leaves the optimizer untouched. The identity it rests on is also correct. For binary rewards centered by the group mean, the mean squared advantage equals the empirical pass rate times its complement, so degenerate groups contribute nothing. I checked this and the companion expression for the probability of a non-degenerate group.
-
The ablations ask the questions that matter. Comparing against uniform sampling at the same retention ratio separates the selection rule from the effect of simply training on less data, and the paper reports that uniform sampling falls below the full-data baseline. Sweeping the rollout count under a fixed total budget is also well posed, and the explanation of the interior optimum is coherent.
Critical weaknesses
-
The headline gains correspond to a handful of problems, and no measure of run-to-run variation appears anywhere in the paper. The abstract leads with relative improvements of 45.9 percent on AIME25 and 18.2 percent on AMC23, which in absolute terms are 5.6 and 10.0 points. The reported values are quantized, with AIME25 on multiples of one ninetieth and AMC23 on multiples of one fortieth, so the evaluation appears to be three samples over thirty problems and one sample over forty. The 45.9 percent figure is then 5 additional correct answers out of 90 attempts, and the 18.2 percent figure is 4 additional problems out of 40. Several other starred entries are a single problem. With no seeds, no standard deviations, and no significance testing, I cannot distinguish these from sampling noise, and reporting them as relative percentages makes them look an order of magnitude more substantial than they are. For one of the eight configurations the method is also worse on three of five benchmarks with the average essentially unchanged, yet the text claims consistency across all scales and families.
-
The theoretical analysis does not support the conclusion it is cited for. The appendix result is a correct but elementary conditional second-moment computation. It is maximized at a pass rate of one half only when the two reward-conditioned score norms are approximately equal, which the paper notes but does not verify. More importantly, the quantity being maximized is the second moment of the gradient estimator, and the paper's own covariance identity shows that selecting large values of it selects high-variance groups. Equating a large second moment with informativeness, without relating either to expected policy improvement, leaves the criterion without the grounding claimed for it. The main text then asserts that the out-of-distribution gains follow from this result by preventing overfitting to the in-distribution difficulty profile. Nothing in the result concerns generalization, and that sentence ought to be removed.
Questions for authors
-
Can the main table be rerun with at least three seeds, reporting absolute accuracies with standard deviations alongside the relative figures, and stating the number of evaluation samples per benchmark?
-
What does the method achieve if the score is replaced by the uncertainty factor alone, or by simply dropping degenerate groups and sampling uniformly among the rest? With eight rollouts that factor takes only four distinct nonzero values, so this isolates whether the graded ranking adds anything.
-
Why does the binary rejection baseline fall below the full-data baseline on both models? That ordering is unexpected.
-
Since rollouts still cover the full active pool, the saving applies to the backward pass only. What fraction of baseline wall-clock time is generation, and what is the end-to-end saving of the selector alone?
4 comments
Rebuttal by Authors (2/3)
W2. Theoretical grounding and gradient informativeness
The theoretical analysis rigorously characterizes reward-derived learning signals, while experiments establish the practical effectiveness of the resulting selection criterion.
The foundation of LZE is the exact identity in Eq. (6):
This establishes that all-correct and all-incorrect groups yield zero centered reward advantages, while mixed-outcome groups provide non-degenerate learning signals. Furthermore, Eq. (7) shows that the probability of obtaining a mixed-outcome group is maximized when the underlying success probability is 0.5. These results directly motivate prioritizing prompts near the policy's current learning frontier.
Appendix G.1 extends this analysis to policy-gradient statistics. Theorem 1 explicitly incorporates reward-conditioned score norms into the second-moment decomposition, while Eq. (26) characterizes the expected empirical group-gradient estimator. The second moment captures both the magnitude and stochastic variability of gradient contributions; the exact optimization benefit additionally depends on gradient geometry and policy dynamics. This motivates our use of observable reward variation as a practical criterion for allocating training computation.
Computational efficiency is central to this design choice. Estimating the expected improvement of each individual prompt through explicit gradient evaluation, counterfactual policy updates, or post-update validation would introduce substantial overhead into online selection. In contrast, () is computed directly from existing rollout rewards, with negligible additional cost. LZE combines this inexpensive signal with historical difficulty and learning dynamics to guide policy updates efficiently.
The effectiveness of this criterion is directly demonstrated by experiments. Figure 3(a) shows that removing the uncertainty component causes the largest performance degradation. Figure 3(b) validates LZE across different rollout counts under a fixed sampling budget. Table 2 further shows that LZE outperforms uniform sampling at the same 40% retention ratio on both evaluated backbones.
The theoretical analysis thus establishes the motivation for the selection criterion, while the empirical results in Tables 1, 2, and 5 demonstrate its effectiveness, including the observed OOD improvements. We will further polish the corresponding discussion in the camera-ready version to make this connection more explicit.
Q2. Uncertainty-only selection and graded ranking
The uncertainty-only comparison requested by the reviewer is already included in Figure 3(a).
The component ablation evaluates four configurations: the baseline, uncertainty-only selection, uncertainty with the initial-difficulty anchor, and the complete LZE score.
Figure 3(a) shows that uncertainty provides the strongest individual contribution, while incorporating difficulty history and momentum yields additional improvements. This directly establishes the value of the full scoring formulation.
Furthermore, the distinction between graded ranking and binary rejection is straightforward. A binary rule assigns all non-degenerate groups the same selection eligibility. Uncertainty-based ranking further differentiates groups according to their current reward variation. The complete LZE score additionally distinguishes prompts with identical current pass rates through their initial difficulty and recent learning dynamics.
The resulting hierarchy is directly reflected in Figure 3(a). Each component contributes to the final selection behavior, with the complete LZE formulation achieving the strongest performance in the reported component ablations.
Appendix F.1 further demonstrates the importance of stochastic exploration. Gumbel-Top-K improves OOD performance by 2.1%, 13.7%, and 17.3% across three evaluated backbones compared with deterministic Top-K.
Together, these results establish the value of progressively refining selection priorities through current uncertainty, historical difficulty, and learning dynamics.
Rebuttal by Authors (3/3)
Q3. Why binary rejection can underperform full-data training
Binary rejection and LZE employ different selection granularity and optimization mechanisms, which explains their different empirical behavior.
Reinforce-Rej uses a binary filtering criterion to remove groups with uniformly correct or incorrect outcomes, together with a REINFORCE-style optimization procedure. LZE retains the original group-relative policy optimization and applies a continuous Energy Score to prioritize updates among sampled prompt groups.
Binary rejection treats all remaining non-degenerate groups equally, despite substantial differences in their reward variation and learning histories. LZE further ranks these groups using current uncertainty, initial difficulty, and momentum, allocating the available gradient budget toward the most informative region of the current policy.
The component ablations in Figure 3(a), together with the same-retention-ratio uniform sampling comparison, further substantiate the practical value of allocating updates according to the learning-zone criterion.
Q4. Computational savings and wall-clock efficiency
The 36% estimated FLOPs reduction already accounts for full rollout coverage, while Figure 2 directly reports end-to-end wall-clock savings from backward selection alone.
Appendix E decomposes the computational footprint into inference/scoring and policy optimization, with theoretical coefficients of 4 and 6, respectively.
LZE retains full rollout coverage during backward selection and performs policy optimization on only 40% of prompt groups. The resulting theoretical FLOPs ratio is
corresponding to a 36% reduction in estimated total training FLOPs.
More importantly, Figure 2 already provides measured end-to-end training times for backward selection alone:
| Model | Vanilla-RL | LZE | Wall-clock reduction |
|---|---|---|---|
| Qwen2.5-Math-1.5B | 3.97 h | 2.40 h | 39.5% |
| Qwen2.5-Math-7B | 4.92 h | 3.51 h | 28.7% |
These correspond to 1.65× and 1.40× speedups, respectively. The optional forward pruner provides further improvements, reaching 1.80× and 1.91× overall speedups. The baseline wall-clock fraction attributable to rollout generation is approxiamately around 30% for Qwen2.5-Math-1.5B and Qwen2.5-Math-7B.
The analytical computation model and measured wall-clock results consistently demonstrate the efficiency of LZE. In particular, the selector alone provides substantial end-to-end savings, with additional benefits available from the forward-pruning mechanism.
Dear Reviewer fepW,
As the discussion period is drawing to a close, we wanted to kindly follow up and ask whether our response has addressed your concerns. If there are any remaining questions or points that you feel have not been sufficiently addressed, we would sincerely appreciate the opportunity to clarify them before the discussion period ends.
Thank you again for your time, thoughtful feedback, and consideration.
Rebuttal by Authors (1/3)
We thank the reviewer for carefully examining the theoretical formulation and recognizing the validity of the centered-reward analysis, the lightweight design, and the controlled ablations. We address the concerns below.
W1/Q1. Statistical reliability and interpretation of experimental gains
We thank the reviewer for raising this question. Nevertheless, the review overlooks the broader experimental evidence and contains an incorrect characterization of Table 1.
First, all eight training configurations in Table 1 achieve positive improvements in average accuracy. The review highlights the relatively small gains on individual benchmarks, while several configurations exhibit substantially larger improvements:
+4.0 percentage points: Qwen2.5-Math-1.5B trained on MATH (0.469 → 0.509).
+3.9 percentage points: Qwen3-8B trained on DAPO-MATH (0.716 → 0.755).
+3.4 percentage points: Qwen3-1.7B trained on GSM8K (0.548 → 0.582).
The review further states that one configuration deteriorates on three of five benchmarks. This is inconsistent with the reported results. For Qwen3-4B trained on DAPO-MATH, LZE improves GSM8K, MATH, and AIME25, while two benchmarks decrease. Its average accuracy increases from 0.764 to 0.766.
Second, Table 1 explicitly reports absolute accuracies alongside relative improvements. The highlighted gains on AIME25 (0.122 → 0.178) and AMC23 (0.550 → 0.650) correspond to 5.6 and 10.0 percentage points, respectively. These are substantial improvements on challenging mathematical reasoning benchmarks.
Additional Avg@4 evaluation. To further examine OOD performance stability, we evaluate each model using four independent responses per question. The evaluation covers all 40 AMC23 problems and 30 AIME25 problems, corresponding to 160 and 120 responses per model, respectively.
Independent training runs. We additionally conduct three independent LZE runs on two representative MATH training configurations. The multi-seed runs demonstrate stable LZE performance.
Together, the eight positive average improvements, additional Avg@4 evaluation, and independent-run statistics collectively provide a broader and more informative assessment of the method than isolated differences on individual benchmarks.