- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DataLens, a pre-training data selection estimator for LLM reinforcement learning (GRPO). The authors argue that standard first-order gradient alignment (, Section 3.3) can misrank candidate prompts under finite RL updates because prompts with similar local alignment may sit in regions with very different loss curvature. DataLens estimates each candidate's finite-step utility on a held-out validation set (256 examples) using a second-order Taylor approximation in the projected LM-head subspace, (Eqs. 9–11), computed via the predictive-Fisher variance identity in Appendix A.9 (Eqs. 20–22) without materializing the Hessian. After ranking the eligible nonzero-direction candidate pool (Eq. 7) at the initial checkpoint , DataLens selects the Top- prompts for standard 200-step GRPO training.
The method is evaluated across six model–domain pairs (Qwen2.5-Math-7B, DeepSeek-Math-7B, Mathstral-7B on OpenR1-Math to MATH-500; Qwen2.5-Coder-7B, DeepSeek-Coder-6.7B, Mistral-7B-v0.3 on TACO to LiveCodeBench-v6) through controlled one-step LM-head probes (Section 4.2–4.3, Table 1), short-horizon ( steps) GRPO bridge experiments (Section 4.4, Table 2), and 200-step end-to-end GRPO across subset budgets (Section 4.5, Table 3). While the headline result on Qwen2.5-Math-7B at reaches 84.0% MATH-500 accuracy (vs. 70.1% for and 67.5% for random selection), the broader downstream evidence is much weaker than the headline suggests: outside that single Qwen-Math cell, margins over Empirical-Fisher shrink to 0.3–0.7 percentage points without error bars in the main tables, Qwen-Math itself suffers an 8.9-point non-monotonic drop at , and adaptive selection baselines (LearnAlign, GradAlign) are stripped of their online re-scoring mechanism.
Critical strengths
1. Clean finite-step formulation and tractable LM-head curvature computation.
In binary-reward GRPO (), of candidate prompts produce zero advantage variance across rollouts (Appendix Table 7), and on the remaining eligible pool , first-order gradient alignment is vulnerable to overshooting in high-curvature directions. Using the predictive-Fisher variance identity in the detached LM-head subspace (Appendix A.9, Eqs. 20–22) makes directional curvature estimation tractable without explicit Hessian-vector products.
2. Structured three-level experimental design.
Separately testing (i) controlled one-step LM-head probes across 2,021 candidates (Table 1, where DataLens achieves the best Spearman ), (ii) short-horizon 1–20 step GRPO transfer on a disjoint bridge set (Table 2), and (iii) 200-step GRPO subset training (Table 3) is a well-thought-out diagnostic structure that connects the local Taylor approximation to downstream RL training.
Critical weaknesses
1. Downstream superiority rests heavily on a single outlier cell (Qwen-Math ), while margins in the other five settings are tiny ( pp) and lack uncertainty reporting in Tables 3 and 22.
Although all downstream experiments were run with three seeds (42, 7, 1337), Table 3 (all 24 setting–budget cells) and Appendix Table 22 (comparison against external baselines at ) omit standard deviations, confidence intervals, and significance tests—reporting seed variance only for the Qwen-Math cell in Appendix Tables 24–25. Looking closely at Appendix Table 22, once you step outside Qwen-Math (.840 vs. .765), DataLens's margin over Empirical-Fisher (, Appendix A.7 Eqs. 18–19) is only across all five remaining settings (DeepSeek-Math .720 vs. .715, Mistral-Math .690 vs. .687, Qwen-Code .458 vs. .451, DeepSeek-Code .433 vs. .428, Mistral-Code .415 vs. .409)—margins so small that the authors themselves state in Appendix D.1 (lines 1209–1211) they "are therefore treated as point estimates." Likewise, in Table 3, simple normalized curvature is within points of DataLens across almost every cell and actually beats DataLens on DeepSeek-Math at (.738 vs. .736) and Mistral-Code at (.360 vs. .358). Even the headline Qwen-Math curve in Table 3 is highly non-monotonic—jumping from .711 () to .840 () before plunging points to .751 at —an anomaly that appears in none of the other five settings and is only tested on MATH-500. This directly drives my negative recommendation because the data does not support broad, statistically reliable downstream gains over Empirical-Fisher or .
2. Unfair restriction of adaptive baselines (LearnAlign, GradAlign) to static snapshots, alongside substantial pre-selection compute overhead.
In Table 1 and Appendix Table 22 (protocol in Appendix A.7, lines 826–829), LearnAlign and GradAlign are evaluated only as static one-shot scoring functions at , disabling their core online/adaptive re-selection during the 200-step GRPO run. Meanwhile, DataLens's pre-selection stage consumes H100-hours on Math and H100-hours + CPU-hours on Code—adding of the compute cost of a full 200-step GRPO run ( and H100-hours, Appendix Table 12). On Code (Qwen-Code, Appendix Table 13), simply running Random selection for steps (400 steps) already reaches .447, within points of DataLens's .458 at total compute. Without comparing against full adaptive LearnAlign/GradAlign or online difficulty-targeted filtering under matched total H100-hours, the paper does not establish that static curvature screening is a better use of a compute budget than online curriculum/adaptive selection.
Questions for authors
-
Assessment-critical (Complete downstream uncertainty & significance over
Empirical-Fisher/ ): Since all 24 cells in Table 3 and all rows in Appendix Table 22 were run across seeds42, 7, 1337, please provide the full per-cell standard deviations and paired confidence intervals comparingDataLensagainstEmpirical-Fisher, ,GradAlign, andLearnAlign. In how many of the five settings outsideQwen-Mathis DataLens's edge overEmpirical-Fisherstatistically distinguishable from zero? -
Assessment-critical (External validity of the
Qwen-Mathspike): In Table 3, onlyQwen-Mathexhibits the sharp spike at (.840) followed by an -point drop at (.751). If you evaluate those exact sameQwen2.5-Math-7Bcheckpoints (, and full 5,000-example training) on a second math benchmark such asAIME24/25orOlympiadBench, does still outperform and full-pool training by a wide margin, or is the.840peak specific toMATH-500? -
Assessment-critical (Compute-matched adaptive selection baselines): Given that DataLens pre-selection adds H100-hour overhead (Appendix Table 12), how does DataLens compare under matched total H100-hours against (a) running
LearnAlignorGradAlignwith periodic online re-scoring during GRPO, and (b) simple online rollout pass-rate filtering (e.g., keeping prompts with on the fly during GRPO)?
6 comments
Response to Reviewer sRmM — Part 2/2: Repeatability, External Validity, and Compute Fairness
We thank the reviewer again for emphasizing repeatability, external validity, and fair compute comparisons. The paired statistical evidence and Qwen-Math peak controls appear in Part 1/2; we address the remaining issues here.
2. Qwen-Math peak: repeatability and external validity (continued)
We agree that repeated high MATH-500 scores do not alone establish cross-benchmark robustness. We separated seed robustness from a new validation-source experiment.
R6. Qwen-Math Top-100 repeatability on MATH-500.
| Randomness varied | Runs | Accuracy mean±SD (%) |
|---|---|---|
| Training only, fixed subset | 8 | 84.0±0.7 |
| Rollout/scoring, fixed pool | 9 | 83.6±1.0 |
| Pool × rollout × training | 27 | 83.4±1.2 |
| Full-pipeline range | 27 | 81.1–85.2 |
The Top-100 result recurs across training, scoring and candidate-pool randomness. To test a changed scoring source, we replaced OpenR1-Math validation examples with 256 GSM8K training examples while holding Qwen2.5-Math-7B, N=100 and 200 updates fixed.
R7. Alternative scoring-validation source versus GradAlign-full.
| Source | Evaluation | DL−GradAlign-full (pp) |
|---|---|---|
| GSM8K train | MATH-500 | +3.3 |
| GSM8K train | OlympiadBench | +3.0 |
R6 supports repeatability; R7 provides positive cross-source and cross-benchmark means. We agree this does not answer the precise checkpoint-transfer request: changing validation data also changes selection. The original Top-100, Top-200 and full-5,000 checkpoints have not yet been compared directly on a second benchmark. We clarified this limitation (Sec. 4.6/Table 7; Apps. D.5/D.12, Table 29).
3. Adaptive baselines and equal-total-compute fairness
We agree that matched training steps and static selector adaptations are not substitutes for full adaptive algorithms evaluated at equal total GPU cost. We promoted strong one-shot comparators, corrected cost accounting, and added longer Random runs, while explicitly identifying the incomplete matched-budget test.
R8. Strong one-shot selectors at N=100 (three-seed means, %; math: MATH-500; code: LiveCodeBench).
| Setting | DataLens | LearnAlign | GradAlign | Fisher |
|---|---|---|---|---|
| Qwen-Math | 84.0 | 75.2 | 75.6 | 76.5 |
| DeepSeek-Math | 72.0 | 71.1 | 71.3 | 71.5 |
| Mistral-Math | 69.0 | 68.4 | 68.6 | 68.7 |
| Qwen-Code | 45.8 | 44.8 | 44.9 | 45.1 |
| DeepSeek-Code | 43.3 | 42.3 | 42.5 | 42.8 |
| Mistral-Code | 41.5 | 40.5 | 40.7 | 40.9 |
LearnAlign/GradAlign here are static, initial-checkpoint scoring adaptations using cached rollouts, not complete online algorithms.
R9. Corrected cost accounting (GPU values in H100-hours; verifier CPU-h separate).
| Component | Math | Code |
|---|---|---|
| Shared candidate rollouts | 6.42 | 9.16 |
| Shared candidate gradients | 2.31 | 2.22 |
| Shared GPU subtotal | 8.73 | 11.38 |
| DataLens validation/curvature/calibration | 3.74 | 3.61 |
| DataLens ranking/I/O | 0.06 | 0.07 |
| DataLens extra GPU subtotal | 3.80 | 3.68 |
| Full DataLens selection | 12.53 | 15.06 |
| GRPO training, 200 updates | 29.10 | 32.40 |
| DataLens standalone total | 41.63 | 47.46 |
| Random standalone total | 37.83 | 43.78 |
| Shared verifier CPU-h | 0.28 | 91.4 |
Each standalone selector pays for shared eligibility once; cache-based incremental comparisons omit it for every method. This fixes the earlier ambiguity about full selection cost versus marginal scoring overhead.
R10. Extended Random versus DataLens (N=100, total H100-hours; scores in %). Extended Random costs are reconstructed by scaling the 200-update training time, not independently metered.
| Setting | Method | H100-h | Score |
|---|---|---|---|
| Qwen-Math | Random, 200 updates | 37.83 | 67.5 |
| Qwen-Math | Random, 286 updates | 50.34 | 70.4 |
| Qwen-Math | Random, 340 updates | 58.20 | 71.6 |
| Qwen-Math | Random, 400 updates | 66.93 | 72.9 |
| Qwen-Math | DataLens, 200 updates | 41.63 | 84.0 |
| Qwen-Code | Random, 200 updates | 43.78 | 40.5 |
| Qwen-Code | Random, 293 updates | 58.85 | 42.8 |
| Qwen-Code | Random, 340 updates | 66.46 | 43.8 |
| Qwen-Code | Random, 400 updates | 76.18 | 44.7 |
| Qwen-Code | DataLens, 200 updates | 47.46 | 45.8 |
DataLens has higher reported means than extended Random, but Random at 286/293 updates already exceeds DataLens's total H100-hour budget. The linear model would permit about 226/223 math/code Random updates at DataLens's budget; no accuracy is claimed at those unmeasured points.
Thus, the revised tables resolve the bookkeeping issue, not the full compute-fairness concern. The remaining decisive test is metered, equal-total-budget training for complete online LearnAlign/GradAlign, online pass-rate filtering and Random on the same model, pool and validation source, counting re-scoring and rollouts. R7 adds a GradAlign-full comparison under a changed validation source, but it has no matched GPU-hour measurements. We have narrowed the paper's claims accordingly.
Revisions: Secs. 4.5–4.6/Tables 4, 7; Apps. A.7/A.12 (Tables 16–17), D.12.
Looking Forward to Further Discussion with You
Dear Reviewer sRmM,
Thank you again for your valuable time and constructive feedback.
As the rebuttal period is nearing its end, we would like to kindly check whether our previous response has addressed your main concerns. If there are any aspects that you feel need further clarification or improvement, please let us know, and we would be more than happy to provide additional explanations or revisions.
Best regards, Authors
Response to Reviewer sRmM — Part 1/2: Paired Statistics and the Qwen-Math Peak
We sincerely thank the reviewer for examining statistical reliability, the Qwen-Math peak, and compute fairness. Your comments motivated paired tests, peak controls, and clearer limitations.
1. Downstream uncertainty and superiority
We agree that three seeds and small Fisher gaps cannot support broad superiority claims. We added ten paired N=100, 200-update runs, 19,999 joint seed/item bootstrap resamples, and Holm correction over 12 unique contrasts (R1/R3). Values are mean±sample SD (%); intervals are pointwise 95% CIs, differences in pp.
R1. DataLens versus Fisher, ten paired runs.
The reviewer is right about the five small gaps: their CIs include zero. Only Qwen-Math has a supported positive Fisher difference after correction; we removed the broader significance claim.
We also corrected the original 24-cell comparison.
R2. DataLens / q_dir across 24 cells (three-seed means, %, 200 updates).
DataLens has higher means in 22/24 cells, with two −0.2 pp exceptions. These are descriptive means; paired CIs for all budgets remain unavailable.
To test whether curvature or update magnitude alone explains the gains, we added fitted and unfitted controls.
R3. Qwen-Math/Code ablations, ten paired runs. q=raw curvature, L=direction-length penalty, M=calibrated mixture; same 12-contrast Holm family.
For Math, M/L coefficients use 500 cluster-disjoint OpenR1-Math candidates, separate from pool/audit/calibration/bridge/test data. Labels measure controlled ΔJ_v on D_v after the detached LM-head update at θ₀; coefficients are frozen before evaluation. Four branches share ten seeds (40 runs); fitting cost is not separately metered. DataLens improves over q and L, but has no established advantage over M.
We also added a newer-model test, distinct from the original six conditions.
R4. Qwen3.5-9B on official MATH-500 (N=100, 200 updates, ten paired seeds; mean±SD %, paired seed-t 95% CIs, exact sign-flip tests, separate three-contrast Holm family).
All three adjusted p values are below .05 (exact raw p=2/1024). The test uses Transformers 5.3.0/TRL 0.29.0, no vLLM, text-only input, thinking disabled, and greedy evaluation (max 2,048 tokens). This supports this new model, not significance in the five uncertain original settings.
Revisions: Sec. 4.5/Tables 3–6; Apps. D.9–D.11/Tables 32–34.
2. Investigating the Qwen-Math N=100 peak
We agree that the 84.0% peak needs controls beyond repetition. We evaluated exposure and subset-composition alternatives using the original MATH-500 protocol.
R5. Qwen-Math peak controls (accuracy mean±SD, %).
Exposure-matching narrows the Top-100/Top-200 gap from 8.9 to 2.4 pp, although it also doubles updates. Lower tail-only and random-tail results indicate that composition also matters. Neither control alone isolates causality or proves cross-benchmark persistence. Part 2 reports repeatability and external-validity evidence (Sec. 4.6; App. D.3–D.6/Tables 27–30).