acceptodds
Withdrawn from ICLR 2027

DataFlex-RL: An Evaluation Platform for RLVR Data Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Data policies for reinforcement learning with verifiable rewards (RLVR) determine which rollouts are used, how they are weighted, and which domains contribute to subsequent batches. We introduce DataFlex-RL, an evaluation platform for comparing these policies under a common GRPO recipe. In our primary study, we evaluate 13 configurations with 12 matched seeds on Qwen2.5-7B-base across 12 math, logic, and science benchmarks. Uniform GRPO improves domain-balanced mean accuracy (Overall) by 7.76 percentage points over the untrained checkpoint. For none of the eight selection or reweighting methods does the paired 95% confidence interval for the difference from uniform sampling exclude zero. Likewise, none of the three adaptive mixtures significantly outperforms a fixed equal-weight mixture at this precision. A corrected 12-seed extension on Llama-3.1-8B-base places the additional methods on the same score scale as the original controls, but their observed means reveal no consistent winner. To assess evaluation sensitivity, we recompute scores for nine Qwen2.5-7B-Instruct runs using both a math-heavy six-benchmark summary—five math benchmarks plus GPQA-Diamond, excluding logic—and the domain-balanced 12-benchmark summary. The resulting rankings are negatively correlated (), whereas rankings based on summaries retaining all 12 benchmarks are largely consistent. Overall, across the controlled settings studied, data policies measurably alter the training process but do not provide a reproducible advantage over uniform training.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
RLVR, GRPO, data selection, adaptive sampling, benchmark evaluation.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uMT6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

DataFlex-RL provides a common GRPO evaluation platform for selection, reweighting, and mixture adaptation. Its primary Qwen2.5-7B-base experiment compares 13 configurations over 12 matched seeds and 12 math, logic, and science benchmarks. Uniform GRPO gains 7.76 Overall points over the untrained model, but no tested data policy shows a paired 95% confidence interval excluding zero relative to its control. A corrected Llama-3.1-8B-base extension and an evaluation-sensitivity analysis examine whether this result depends on model family or benchmark coverage.

Critical strengths

The finding that tested data policies provide no reproducible improvement over uniform controls supports uniform sampling as a strong baseline for RLVR data-policy evaluation. Uniform GRPO itself improves Overall by 7.76 points on Qwen2.5-7B-base, so this negative result is unlikely to be explained simply by failed training.

The primary experiment covers 13 configurations, 12 matched seeds, and 12 benchmarks, with additional model-family and model-scale checks. Appendix A.3 adds matched random, matched update tokens, and alternative strengths controls to examine targeting, update volume, and intervention strength. The Math-Heavy-6 versus DB-12 analysis also checks evaluation sensitivity.

The platform brings selection, reweighting, and mixture adaptation into a shared rollout, verification, GRPO optimization, and evaluation workflow. New data policies can therefore be compared under the same protocol without rebuilding the pipeline.

Critical weaknesses

Section 3.2 and Table 1 describe the signals but leave some intervention rules unclear, including difffilter's default solve-rate band and the weighting formulas for per and softmax. These details matter for reproducing the comparison.

The Llama gain interval needs clarification. Under the same seed-level interval calculation, subtracting the fixed untrained score of 10.87 from Table 6's [20.01, 22.23] gives [9.14, 11.36], whereas Table 2 reports [7.75, 12.33]. This appears to be a fixable reporting issue and does not overturn the primary Qwen result.

Questions for authors

What are difffilter's default solve-rate band and the weighting formulas and default hyperparameters for per and softmax?

Do Tables 2 and 6 use the same Llama seeds and interval estimator? If so, how should the gain interval be reconciled?

Reviewer_MoqsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds DataFlex-RL, a shared GRPO pipeline for comparing RLVR data policies, grouped by where they act: selecting responses after rollout, reweighting their loss, or adapting the domain mixture of future prompts. On Qwen2.5-7B-base, thirteen configurations are trained with twelve matched seeds and evaluated on math, logic, and science benchmarks. No policy reliably beats uniform sampling, although GRPO itself gives a clear gain. Smaller checks on other base models and a rescoring of an Instruct campaign are used to argue that the null result holds and that leaving out a domain can change the apparent winner.

Critical strengths

The main comparison is done with more care than most work in this area. All configurations share one recipe and twelve matched seeds, differences are reported as paired intervals, and per-benchmark means are given. Many data selection gains in RLVR come from one or two runs, so seeing thirteen policies land within about a point of each other, next to a much larger gain from GRPO itself, is informative, and it is the main reason I see value in the paper.

I also like the split between where a policy acts and the signal it uses. It turns loosely related methods into direct contrasts, for instance the same advantage score used as a soft weight in one method and as a hard mask in another, and it shows that some selection rules mostly shrink the update rather than pick better data.

Critical weaknesses

My main concern is that the null result is read more broadly than the design allows. The paper never shows that this setup can detect a data effect at all. A large gain over the untrained model shows that the model learns, not that its data matters, and Shao et al. (2026) see Qwen2.5 math models improve even under random rewards. Several methods were also built for another goal, such as sampling more than is trained on, refilling filtered batches, or shortening responses, so equal accuracy at a fixed step count hardly contradicts them.

I am also not convinced that only the data policy varies. The baseline and the static mixture are both uniform GRPO with equal domain weights, yet static comes out lower for every model in the scale sweep, by about as much as the largest method effects, and the adaptive mixtures only look positive against this lower control. The evaluation sensitivity result comes from a small three-seed Instruct campaign whose per-method scores are not shown, and the twelve-seed base model results in the paper do not show the same ranking reversal.

References

Rulin Shao, Shuyue Stella Li, Rui Xin, et al. Spurious Rewards: Rethinking Training Signals in RLVR. In Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.

Questions for authors
  1. What differs between the baseline and static runs, given that both are described as the same uniform control?
  2. How much data does each selector remove per step, and does the conclusion change if the update is renormalized or the batch is refilled?
  3. Could you report the Instruct results together with a uniform baseline, and repeat the summary comparison on the twelve-seed base-model blocks?
  4. A positive control, such as dropping one domain, together with the largest gain each comparison can rule out, would make the negative result much more convincing.
Reviewer_5J7AOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

DataFlex-RL is an evaluation platform for comparing selection, reweighting, and domain-mixture policies under a common GRPO training protocol. The primary experiment evaluates 13 configurations on Qwen2.5-7B-base using 12 matched seeds and 12 benchmarks spanning mathematics, logic, and science. It also provides a sensitivity analysis of the evaluation configuration for the same learning results as extension experiments of other models.

Critical strengths

The controlled reassessment of existing data policies is needed, and the paper pointed this out. Comparing previously successful policies and their adaptations under a common training recipe reveals that their additional benefits are not clearly reproducible in the paper’s setup.

Critical weaknesses
  1. Implementation specifications are substantially incomplete.

For a paper whose principal contributions are an empirical comparison and an evaluation platform, precise implementation specifications are essential. The submission falls substantially short of this requirement. Table 1 explicitly states that "Full intervention rules are specified in Section 3", but neither that section nor the appendices supplies essential definitions. The difffilter’s acceptance interval, default selector retention counts/fractions, exact reweighting functions, and adaptive-mixture update equations. Learning rate, batch sizes, and training sampling parameters are also missing. These omissions prevent reproduction of the central comparison and directly undermine a paper whose contribution depends on reliable empirical evaluation. Ultimately, in my opinion, the paper is not sufficiently complete as a report of its central experiment.

  1. Implementation fidelity and supporting validation remain unresolved.

The paper does not specify whether PODS/GFPO-related configurations recompute advantages after selection. Selection is not renormalized, whereas reweighting uses mean-one weights, introducing differences in optimization treatment.

Additionally, the main experiment relies only on a relatively short 300-step experiment, and the actual performance of each method may vary in experiments with longer steps. Section 4.1 mentions 1,000-step training, but neither the main text nor the appendix specifies the associated configurations or identifies which reported results correspond to this setting.

Questions for authors
  1. What exact policy rules, default hyperparameters, and shared GRPO settings produced the main results? Please specify all detailed settings and method configurations used in the paper.

  2. How were the original methods (e.g. PODS) modified, particularly advantage computation and normalization after selection?

  3. The 300-step horizon seems relatively short. How do method–control performance gaps change with longer training?

  4. It is just a question. Why use five rollouts per prompt? Do the relative performance and conclusions between policies remain consistent even when the number of rollouts is increased?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.