acceptodds
Under review as a conference paper at ICLR 2027

DataLens: Predicting Finite-Step Data Utility for Data-Efficient LLM Reinforcement Learning

Abstract

Selecting effective training data is crucial for efficient reinforcement learning (RL) of large language models (LLMs), yet example utility cannot be reliably inferred from local training signals. Existing gradient-based methods use local gradient information as a proxy for training improvement. We show that this approximation can break down for finite displacements: candidates with similar local alignment may produce substantially different target-objective improvements. We formulate finite-step data utility prediction and define utility as the target-objective improvement induced by a prescribed candidate-specific update. We propose DataLens, which augments first-order alignment with directional objective geometry derived from a finite-step expansion, modeling objective evolution along candidate-induced update directions. Controlled matched-objective experiments verify that this geometric signal captures utility differences missed by local approximations. Across six model-domain settings spanning Qwen, DeepSeek, and Mistral on math and code, DataLens achieves the highest point-estimate correlation with finite-step utility and transfers to short-horizon GRPO. In end-to-end GRPO experiments, 100 examples selected by DataLens achieve 84.0% MATH accuracy across three random seeds, compared with 70.1% using first-order alignment and 67.5% using random selection. These results show that RL data selection benefits from modeling finite-step objective geometry beyond local gradient signals.

Primary area:
reinforcement learning
Keywords:
LLM Reinforcement Learning, Data Selection, Finite-Step Utility, Gradient-Based Selection, Directional Curvature, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sRmMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DataLens, a pre-training data selection estimator for LLM reinforcement learning (GRPO). The authors argue that standard first-order gradient alignment (U1(i)=ηg⊤dU_1(i) = \eta g^\top d, Section 3.3) can misrank candidate prompts under finite RL updates because prompts with similar local alignment may sit in regions with very different loss curvature. DataLens estimates each candidate's finite-step utility on a held-out validation set Dv\mathcal{D}_v (256 examples) using a second-order Taylor approximation in the projected LM-head subspace, U^(i)=g⊤s+12s⊤Hs\widehat{U}(i) = g^\top s + \frac{1}{2} s^\top H s (Eqs. 9–11), computed via the predictive-Fisher variance identity in Appendix A.9 (Eqs. 20–22) without materializing the Hessian. After ranking the eligible nonzero-direction candidate pool E\mathcal{E} (Eq. 7) at the initial checkpoint θ0\theta_0, DataLens selects the Top-NN prompts for standard 200-step GRPO training.

The method is evaluated across six model–domain pairs (Qwen2.5-Math-7B, DeepSeek-Math-7B, Mathstral-7B on OpenR1-Math to MATH-500; Qwen2.5-Coder-7B, DeepSeek-Coder-6.7B, Mistral-7B-v0.3 on TACO to LiveCodeBench-v6) through controlled one-step LM-head probes (Section 4.2–4.3, Table 1), short-horizon (r=1,5,10,20r = 1, 5, 10, 20 steps) GRPO bridge experiments (Section 4.4, Table 2), and 200-step end-to-end GRPO across subset budgets N=25,50,100,200N = 25, 50, 100, 200 (Section 4.5, Table 3). While the headline result on Qwen2.5-Math-7B at N=100N=100 reaches 84.0% MATH-500 accuracy (vs. 70.1% for U1U_1 and 67.5% for random selection), the broader downstream evidence is much weaker than the headline suggests: outside that single Qwen-Math N=100N=100 cell, margins over Empirical-Fisher shrink to 0.3–0.7 percentage points without error bars in the main tables, Qwen-Math itself suffers an 8.9-point non-monotonic drop at N=200N=200, and adaptive selection baselines (LearnAlign, GradAlign) are stripped of their online re-scoring mechanism.

Critical strengths

1. Clean finite-step formulation and tractable LM-head curvature computation.
In binary-reward GRPO (rik∈{0,1}r_{ik} \in \{0, 1\}), 34.8%–58.3%34.8\%\text{--}58.3\% of candidate prompts produce zero advantage variance across K=8K=8 rollouts (Appendix Table 7), and on the remaining eligible pool E\mathcal{E}, first-order gradient alignment U1U_1 is vulnerable to overshooting in high-curvature directions. Using the predictive-Fisher variance identity in the detached LM-head subspace (Appendix A.9, Eqs. 20–22) makes directional curvature estimation tractable without explicit Hessian-vector products.

2. Structured three-level experimental design.
Separately testing (i) controlled one-step LM-head probes across 2,021 candidates (Table 1, where DataLens achieves the best Spearman ρ=0.497–0.660\rho = 0.497\text{--}0.660), (ii) short-horizon 1–20 step GRPO transfer on a disjoint bridge set Dbridge\mathcal{D}_{\mathrm{bridge}} (Table 2), and (iii) 200-step GRPO subset training (Table 3) is a well-thought-out diagnostic structure that connects the local Taylor approximation to downstream RL training.

Critical weaknesses

1. Downstream superiority rests heavily on a single outlier cell (Qwen-Math N=100N=100), while margins in the other five settings are tiny (0.3–0.70.3\text{--}0.7 pp) and lack uncertainty reporting in Tables 3 and 22.
Although all downstream experiments were run with three seeds (42, 7, 1337), Table 3 (all 24 setting–budget cells) and Appendix Table 22 (comparison against external baselines at N=100N=100) omit standard deviations, confidence intervals, and significance tests—reporting seed variance only for the Qwen-Math N=100N=100 cell in Appendix Tables 24–25. Looking closely at Appendix Table 22, once you step outside Qwen-Math (.840 vs. .765), DataLens's margin over Empirical-Fisher (SEF(i)=ηgv⊤di−η22di⊤FempdiS_{\mathrm{EF}}(i) = \eta g_v^\top d_i - \frac{\eta^2}{2}d_i^\top F_{\mathrm{emp}} d_i, Appendix A.7 Eqs. 18–19) is only 0.003–0.0070.003\text{--}0.007 across all five remaining settings (DeepSeek-Math .720 vs. .715, Mistral-Math .690 vs. .687, Qwen-Code .458 vs. .451, DeepSeek-Code .433 vs. .428, Mistral-Code .415 vs. .409)—margins so small that the authors themselves state in Appendix D.1 (lines 1209–1211) they "are therefore treated as point estimates." Likewise, in Table 3, simple normalized curvature qdirq_{\mathrm{dir}} is within 0.2–1.10.2\text{--}1.1 points of DataLens across almost every cell and actually beats DataLens on DeepSeek-Math at N=200N=200 (.738 vs. .736) and Mistral-Code at N=25N=25 (.360 vs. .358). Even the headline Qwen-Math curve in Table 3 is highly non-monotonic—jumping from .711 (N=50N=50) to .840 (N=100N=100) before plunging 8.98.9 points to .751 at N=200N=200—an anomaly that appears in none of the other five settings and is only tested on MATH-500. This directly drives my negative recommendation because the data does not support broad, statistically reliable downstream gains over Empirical-Fisher or qdirq_{\mathrm{dir}}.

2. Unfair restriction of adaptive baselines (LearnAlign, GradAlign) to static θ0\theta_0 snapshots, alongside substantial pre-selection compute overhead.
In Table 1 and Appendix Table 22 (protocol in Appendix A.7, lines 826–829), LearnAlign and GradAlign are evaluated only as static one-shot scoring functions at θ0\theta_0, disabling their core online/adaptive re-selection during the 200-step GRPO run. Meanwhile, DataLens's pre-selection stage consumes 12.5312.53 H100-hours on Math and 15.0615.06 H100-hours + 91.491.4 CPU-hours on Code—adding 43.1%–46.5%43.1\%\text{--}46.5\% of the compute cost of a full 200-step GRPO run (29.1029.10 and 32.4032.40 H100-hours, Appendix Table 12). On Code (Qwen-Code, Appendix Table 13), simply running Random selection for 2.0×2.0\times steps (400 steps) already reaches .447, within 1.11.1 points of DataLens's .458 at 1.465×1.465\times total compute. Without comparing against full adaptive LearnAlign/GradAlign or online difficulty-targeted filtering under matched total H100-hours, the paper does not establish that static θ0\theta_0 curvature screening is a better use of a 45%45\% compute budget than online curriculum/adaptive selection.

Questions for authors
  1. Assessment-critical (Complete downstream uncertainty & significance over Empirical-Fisher / qdirq_{\mathrm{dir}}): Since all 24 cells in Table 3 and all rows in Appendix Table 22 were run across seeds 42, 7, 1337, please provide the full per-cell standard deviations and paired 95%95\% confidence intervals comparing DataLens against Empirical-Fisher, qdirq_{\mathrm{dir}}, GradAlign, and LearnAlign. In how many of the five settings outside Qwen-Math is DataLens's 0.003–0.0070.003\text{--}0.007 edge over Empirical-Fisher statistically distinguishable from zero?

  2. Assessment-critical (External validity of the Qwen-Math N=100N=100 spike): In Table 3, only Qwen-Math exhibits the sharp spike at N=100N=100 (.840) followed by an 8.98.9-point drop at N=200N=200 (.751). If you evaluate those exact same Qwen2.5-Math-7B checkpoints (N=50,100,200N=50, 100, 200, and full 5,000-example training) on a second math benchmark such as AIME24/25 or OlympiadBench, does N=100N=100 still outperform N=200N=200 and full-pool training by a wide margin, or is the .840 peak specific to MATH-500?

  3. Assessment-critical (Compute-matched adaptive selection baselines): Given that DataLens pre-selection adds 43%–47%43\%\text{--}47\% H100-hour overhead (Appendix Table 12), how does DataLens compare under matched total H100-hours against (a) running LearnAlign or GradAlign with periodic online re-scoring during GRPO, and (b) simple online rollout pass-rate filtering (e.g., keeping prompts with 0<rˉi<10 < \bar{r}_i < 1 on the fly during GRPO)?

6 comments
AuthorsOpenReview ↗

Response to Reviewer sRmM — Part 1/2: Paired Statistics and the Qwen-Math Peak

We sincerely thank the reviewer for examining statistical reliability, the Qwen-Math peak, and compute fairness. Your comments motivated paired tests, peak controls, and clearer limitations.

1. Downstream uncertainty and superiority

We agree that three seeds and small Fisher gaps cannot support broad superiority claims. We added ten paired N=100, 200-update runs, 19,999 joint seed/item bootstrap resamples, and Holm correction over 12 unique contrasts (R1/R3). Values are mean±sample SD (%); intervals are pointwise 95% CIs, differences in pp.

R1. DataLens versus Fisher, ten paired runs.

SettingDataLensFisherDL−Fisher [95% CI]Holm p
Qwen-Math84.00±1.1276.50±1.32+7.50 [6.22,8.82].0006
DeepSeek-Math72.00±1.2671.50±1.42+0.50 [−0.60,1.60]1.0000
Mistral-Math69.00±1.1768.70±0.99+0.30 [−1.00,1.62]1.0000
Qwen-Code45.79±1.2145.11±1.17+0.68 [−0.53,1.90]1.0000
DeepSeek-Code43.30±1.4742.80±1.28+0.51 [−0.75,1.76]1.0000
Mistral-Code41.50±0.3540.90±0.67+0.59 [−0.62,1.85]1.0000

The reviewer is right about the five small gaps: their CIs include zero. Only Qwen-Math has a supported positive Fisher difference after correction; we removed the broader significance claim.

We also corrected the original 24-cell comparison.

R2. DataLens / q_dir across 24 cells (three-seed means, %, 200 updates).

SettingN=25N=50N=100N=200
Qwen-Math68.3/66.771.1/69.584.0/72.075.1/73.8
DeepSeek-Math66.9/65.469.6/68.272.0/70.773.6/73.8
Mistral-Math63.7/62.466.5/65.169.0/67.770.7/69.5
Qwen-Code40.1/38.943.1/41.845.8/44.647.5/46.3
DeepSeek-Code37.6/36.440.6/39.443.3/42.145.1/43.9
Mistral-Code35.8/36.038.8/37.641.5/40.343.4/42.1

DataLens has higher means in 22/24 cells, with two −0.2 pp exceptions. These are descriptive means; paired CIs for all budgets remain unavailable.

To test whether curvature or update magnitude alone explains the gains, we added fitted and unfitted controls.

R3. Qwen-Math/Code ablations, ten paired runs. q=raw curvature, L=direction-length penalty, M=calibrated mixture; same 12-contrast Holm family.

DomainMethodMean±SD (%)DL−method [95% CI], ppHolm p
MathDataLens84.00±1.12——
Mathq71.40±1.54+12.60 [10.98,14.30].0006
MathL81.50±1.28+2.50 [1.44,3.56].0009
MathM84.00±1.090.00 [−1.06,1.00]1.0000
CodeDataLens45.79±1.21——
Codeq42.60±1.33+3.19 [1.81,4.56].0006
CodeL43.00±0.90+2.80 [1.59,4.05].0009
CodeM45.90±1.40−0.11 [−1.39,1.19]1.0000
CodeFisher45.11±1.17+0.68 [−0.53,1.90]1.0000

For Math, M/L coefficients use 500 cluster-disjoint OpenR1-Math candidates, separate from pool/audit/calibration/bridge/test data. Labels measure controlled ΔJ_v on D_v after the detached LM-head update at θ₀; coefficients are frozen before evaluation. Four branches share ten seeds (40 runs); fitting cost is not separately metered. DataLens improves over q and L, but has no established advantage over M.

We also added a newer-model test, distinct from the original six conditions.

R4. Qwen3.5-9B on official MATH-500 (N=100, 200 updates, ten paired seeds; mean±SD %, paired seed-t 95% CIs, exact sign-flip tests, separate three-contrast Holm family).

MethodMean±SDDL−method [95% CI], ppHolm p
DataLens91.66±1.06——
Fisher89.66±1.34+2.00 [1.24,2.76].005859
q_dir89.52±1.17+2.14 [1.45,2.83].005859
Random88.66±1.21+3.00 [2.28,3.72].005859

All three adjusted p values are below .05 (exact raw p=2/1024). The test uses Transformers 5.3.0/TRL 0.29.0, no vLLM, text-only input, thinking disabled, and greedy evaluation (max 2,048 tokens). This supports this new model, not significance in the five uncertain original settings.

Revisions: Sec. 4.5/Tables 3–6; Apps. D.9–D.11/Tables 32–34.

2. Investigating the Qwen-Math N=100 peak

We agree that the 84.0% peak needs controls beyond repetition. We evaluated exposure and subset-composition alternatives using the original MATH-500 protocol.

R5. Qwen-Math peak controls (accuracy mean±SD, %).

ConfigurationUpdatesDraws/itemAccuracy
Top-1002001684.0±0.9
Top-200200875.1±1.2
Top-200, exposure-matched4001681.6±1.1
Duplicated Top-1002001683.8±0.9
Ranks 101–200 only2001670.4±1.2
Top-100 + random tail200878.3±1.1
Top-200, score-proportional200weighted80.2±1.0

Exposure-matching narrows the Top-100/Top-200 gap from 8.9 to 2.4 pp, although it also doubles updates. Lower tail-only and random-tail results indicate that composition also matters. Neither control alone isolates causality or proves cross-benchmark persistence. Part 2 reports repeatability and external-validity evidence (Sec. 4.6; App. D.3–D.6/Tables 27–30).

AuthorsOpenReview ↗

Response to Reviewer sRmM — Part 2/2: Repeatability, External Validity, and Compute Fairness

We thank the reviewer again for emphasizing repeatability, external validity, and fair compute comparisons. The paired statistical evidence and Qwen-Math peak controls appear in Part 1/2; we address the remaining issues here.

2. Qwen-Math peak: repeatability and external validity (continued)

We agree that repeated high MATH-500 scores do not alone establish cross-benchmark robustness. We separated seed robustness from a new validation-source experiment.

R6. Qwen-Math Top-100 repeatability on MATH-500.

Randomness variedRunsAccuracy mean±SD (%)
Training only, fixed subset884.0±0.7
Rollout/scoring, fixed pool983.6±1.0
Pool × rollout × training2783.4±1.2
Full-pipeline range2781.1–85.2

The Top-100 result recurs across training, scoring and candidate-pool randomness. To test a changed scoring source, we replaced OpenR1-Math validation examples with 256 GSM8K training examples while holding Qwen2.5-Math-7B, N=100 and 200 updates fixed.

R7. Alternative scoring-validation source versus GradAlign-full.

SourceEvaluationDL−GradAlign-full (pp)
GSM8K trainMATH-500+3.3
GSM8K trainOlympiadBench+3.0

R6 supports repeatability; R7 provides positive cross-source and cross-benchmark means. We agree this does not answer the precise checkpoint-transfer request: changing validation data also changes selection. The original Top-100, Top-200 and full-5,000 checkpoints have not yet been compared directly on a second benchmark. We clarified this limitation (Sec. 4.6/Table 7; Apps. D.5/D.12, Table 29).

3. Adaptive baselines and equal-total-compute fairness

We agree that matched training steps and static selector adaptations are not substitutes for full adaptive algorithms evaluated at equal total GPU cost. We promoted strong one-shot comparators, corrected cost accounting, and added longer Random runs, while explicitly identifying the incomplete matched-budget test.

R8. Strong one-shot selectors at N=100 (three-seed means, %; math: MATH-500; code: LiveCodeBench).

SettingDataLensLearnAlignGradAlignFisher
Qwen-Math84.075.275.676.5
DeepSeek-Math72.071.171.371.5
Mistral-Math69.068.468.668.7
Qwen-Code45.844.844.945.1
DeepSeek-Code43.342.342.542.8
Mistral-Code41.540.540.740.9

LearnAlign/GradAlign here are static, initial-checkpoint scoring adaptations using cached rollouts, not complete online algorithms.

R9. Corrected cost accounting (GPU values in H100-hours; verifier CPU-h separate).

ComponentMathCode
Shared candidate rollouts6.429.16
Shared candidate gradients2.312.22
Shared GPU subtotal8.7311.38
DataLens validation/curvature/calibration3.743.61
DataLens ranking/I/O0.060.07
DataLens extra GPU subtotal3.803.68
Full DataLens selection12.5315.06
GRPO training, 200 updates29.1032.40
DataLens standalone total41.6347.46
Random standalone total37.8343.78
Shared verifier CPU-h0.2891.4

Each standalone selector pays for shared eligibility once; cache-based incremental comparisons omit it for every method. This fixes the earlier ambiguity about full selection cost versus marginal scoring overhead.

R10. Extended Random versus DataLens (N=100, total H100-hours; scores in %). Extended Random costs are reconstructed by scaling the 200-update training time, not independently metered.

SettingMethodH100-hScore
Qwen-MathRandom, 200 updates37.8367.5
Qwen-MathRandom, 286 updates50.3470.4
Qwen-MathRandom, 340 updates58.2071.6
Qwen-MathRandom, 400 updates66.9372.9
Qwen-MathDataLens, 200 updates41.6384.0
Qwen-CodeRandom, 200 updates43.7840.5
Qwen-CodeRandom, 293 updates58.8542.8
Qwen-CodeRandom, 340 updates66.4643.8
Qwen-CodeRandom, 400 updates76.1844.7
Qwen-CodeDataLens, 200 updates47.4645.8

DataLens has higher reported means than extended Random, but Random at 286/293 updates already exceeds DataLens's total H100-hour budget. The linear model would permit about 226/223 math/code Random updates at DataLens's budget; no accuracy is claimed at those unmeasured points.

Thus, the revised tables resolve the bookkeeping issue, not the full compute-fairness concern. The remaining decisive test is metered, equal-total-budget training for complete online LearnAlign/GradAlign, online pass-rate filtering and Random on the same model, pool and validation source, counting re-scoring and rollouts. R7 adds a GradAlign-full comparison under a changed validation source, but it has no matched GPU-hour measurements. We have narrowed the paper's claims accordingly.

Revisions: Secs. 4.5–4.6/Tables 4, 7; Apps. A.7/A.12 (Tables 16–17), D.12.

AuthorsOpenReview ↗

Looking Forward to Further Discussion with You

Dear Reviewer sRmM,

Thank you again for your valuable time and constructive feedback.

As the rebuttal period is nearing its end, we would like to kindly check whether our previous response has addressed your main concerns. If there are any aspects that you feel need further clarification or improvement, please let us know, and we would be more than happy to provide additional explanations or revisions.

Best regards, Authors

Reviewer_YCB7OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DataLens, a method for estimating the utility of individual training examples before reinforcement learning. While existing gradient-based approaches primarily rely on local gradient information, DataLens incorporates directional curvature through a second-order Taylor approximation to predict the target-objective improvement induced by a finite parameter update. Experiments show that DataLens achieves the highest point-estimate correlation with measured finite-step utility across six model-domain settings and improves end-to-end GRPO performance over random data selection.

Critical strengths
  • The method provides a simple and well-motivated extension of gradient-based data selection by incorporating directional curvature, allowing it to distinguish candidates with similar first-order alignment but different finite-step utilities.
  • DataLens achieves the highest point-estimate performance across all six model-domain settings, outperforming several existing gradient-based data selection baselines.
Critical weaknesses
  • The proposed method is a straightforward application of a second-order Taylor approximation to existing gradient-based data selection.Therefore, the methodological and theoretical novelty appears limited.
  • Although DataLens achieves the highest point estimates across all six settings, its improvements over the strongest baseline are only 0.3–0.7 percentage points in five of the six settings, without sufficient statistical evidence establishing their significance.
Questions for authors

Are DataLens’s improvements driven by finite-step utility approximation or simply by curvature information?

4 comments
AuthorsOpenReview ↗

Response to Reviewer YCB7 — Part 1/2: Curvature, Utility Prediction, and Fisher Analysis

We thank the reviewer for the thoughtful critique of novelty and curvature. We agree that Taylor expansion alone is not new. We clarified the prediction target and added curvature controls, measured utilities, and GRPO bridge tests.

1. Contribution beyond curvature alone

We agree that curvature explains much of the first-order gain. To address this, we compare pure/normalized curvature, a length penalty, and a fitted mixture at two displacement scales. DataLens predicts target-objective change after a calibrated candidate displacement via alignment a plus curvature b (a+b). Its bound is local, not a subset or 200-step guarantee.

R1. Qwen-Math score ablation, η=4×10⁻⁴. Same 441 measured candidates; R=audit Recall@100, E=RMSE/U1 RMSE, z=audit-standardized mean Top-100 utility.

ScoreSpearmanTop-100 zRE
U1.065.090.281.000
Raw q.639.670.72—
q_dir.602.645.70—
DataLens.660.821.83.730
Mixture (M).664.828.84.720
Length penalty (L).522.590.63.880
Fisher.561.605.65.830

R2. Additional scale, η=2×10⁻⁴. z and E are normalized separately within this condition.

ScoreSpearmanTop-100 zRE
U1.078.105.291.000
Raw q.625.655.71—
q_dir.589.631.69—
DataLens.646.806.82.630
Mixture (M).650.812.83.624
Length penalty (L).510.576.62.820
Fisher.548.592.64.770

At the main scale, combining alignment and curvature improves audit Recall@100 over q (.72→.83), mean selected utility (.670→.821), and prediction RMSE over U1 (27% lower). The fitted mixture is slightly stronger in point estimates; we do not claim an optimal Taylor coefficient. DataLens calibrates η but does not fit an additional mixing coefficient to measured candidate-utility labels.

R3. Six-setting utility prediction. Spearman versus measured projected finite-step utility; candidate-paired pointwise 95% CIs for DL−q.

SettingU1qq_dirFisherDataLensDL−q [95% CI]
Qwen-Math.065.639.602.561.660+.021 [−.009,.051]
DeepSeek-Math.084.548.526.567.587+.039 [.011,.067]
Mistral-Math.071.503.481.522.536+.033 [.006,.060]
Qwen-Code.098.526.505.539.558+.032 [.008,.056]
DeepSeek-Code.083.493.472.512.528+.035 [.009,.061]
Mistral-Code.072.462.441.481.497+.035 [.008,.062]

Five pointwise intervals favor DataLens over q; Qwen-Math crosses zero and is not statistically established. These are pointwise candidate-paired intervals, not a simultaneous six-setting guarantee.

R4. Measured Qwen-Math ordering reversal (×10⁻²).

QuantityCandidate ACandidate B
First-order U12.402.05
Curvature correction−1.82−0.31
DataLens prediction0.581.74
Measured utility0.551.69

U1 favors A, but the corrected ordering matches the measured utility; this does not prove a long-term causal effect.

R5. Short-horizon GRPO bridge. Mean Spearman across six settings; 64 independent groups of eight prompts per setting, using fresh rollouts.

StepsU1qq_dirDataLens
1.16.43.40.53
5.14.40.37.50
10.11.35.33.45
20.08.28.26.38

In summary, the tests show that curvature is informative, while its combination with alignment improves audit-tail recovery and early GRPO correlation. The fitted mixture remains competitive, and bridge correlations decay with horizon; neither the theory nor these controls establishes a causal explanation for the 200-step gain.

Revisions: Introduction; Secs. 3.4/4.3; Tables 1–2, 20–22; App. D.9/Tables 32–33.

2. Downstream significance: empirical Fisher (continued in Part 2/2)

We agree that five original Fisher gains of only 0.3–0.7 pp require paired uncertainty analysis. We added ten paired N=100, 200-update runs (MATH-500 accuracy or LiveCodeBench pass@1). Differences are pp; pointwise 95% CIs use 19,999 joint seed/item bootstrap resamples. Holm correction covers 12 unique contrasts across R6–R7. All means below are mean±sample SD (%).

R6. Ten-run comparison with empirical Fisher.

SettingDataLensFisherDL−Fisher [95% CI]Holm p
Qwen-Math84.00±1.1276.50±1.32+7.50 [6.22,8.82].0006
DeepSeek-Math72.00±1.2671.50±1.42+0.50 [−0.60,1.60]1.0000
Mistral-Math69.00±1.1768.70±0.99+0.30 [−1.00,1.62]1.0000
Qwen-Code45.79±1.2145.11±1.17+0.68 [−0.53,1.90]1.0000
DeepSeek-Code43.30±1.4742.80±1.28+0.51 [−0.75,1.76]1.0000
Mistral-Code41.50±0.3540.90±0.67+0.59 [−0.62,1.85]1.0000

We agree that the five small Fisher gaps remain uncertain: their CIs cross zero. The Qwen-Math +7.50 pp gap survives Holm correction. Non-significance is not evidence of equivalence.

AuthorsOpenReview ↗

Response to Reviewer YCB7 — Part 2/2: Strong Baselines, Statistical Tests, and Generalization

We sincerely thank the reviewer for emphasizing statistical reliability and recent strong alternatives. Part 1/2 reports the controlled-utility tests and paired Fisher results (R1–R6). Here we address the remaining downstream ablations and broader tests.

2. Strong downstream comparisons (continued)

R7 shares the ten-paired-run protocol and 12-contrast Holm family stated with R6. Differences use unrounded means; intervals are pointwise, not simultaneous.

R7. Qwen-Math/Code downstream score ablations. Ten paired runs; q=raw curvature, L=direction-length penalty, M=calibrated mixture.

DomainMethodMean±SD (%)DL−method [95% CI], ppHolm p
MathDataLens84.00±1.12——
Mathq71.40±1.54+12.60 [10.98,14.30].0006
MathL81.50±1.28+2.50 [1.44,3.56].0009
MathM84.00±1.090.00 [−1.06,1.00]1.0000
CodeDataLens45.79±1.21——
Codeq42.60±1.33+3.19 [1.81,4.56].0006
CodeL43.00±0.90+2.80 [1.59,4.05].0009
CodeM45.90±1.40−0.11 [−1.39,1.19]1.0000
CodeFisher45.11±1.17+0.68 [−0.53,1.90]1.0000

For Qwen-Math, M/L coefficients (λ∈[0,4], c≥0) minimize controlled-utility RMSE on 500 source-cluster-disjoint OpenR1-Math training candidates, separate from the 5,000 pool, 441-candidate audit, 128 scale-calibration examples, bridge, and test sets. Labels measure ΔJ_v on D_v after applying ηd_i to the detached LM head at θ₀. Coefficients are frozen before evaluation, without test tuning. Four branches share ten seeds (40 runs); fitting-label cost is not separately metered. DataLens improves over q and L in both domains, but not demonstrably over M.

R8. Original N=100 strong one-shot comparison. Three-seed final means (%); LearnAlign/GradAlign are static scoring adaptations, not full adaptive methods.

SettingDataLensLearnAlignGradAlignFisher
Qwen-Math84.075.275.676.5
DeepSeek-Math72.071.171.371.5
Mistral-Math69.068.468.668.7
Qwen-Code45.844.844.945.1
DeepSeek-Code43.342.342.542.8
Mistral-Code41.540.540.740.9

To test a newer model, we also evaluated Qwen/Qwen3.5-9B on all 500 official MATH-500 items, N=100, 200 updates, ten paired seeds.

R9. Qwen3.5-9B. Mean±SD (%); paired seed-t pointwise 95% CIs (df=9), exact two-sided sign-flip p over 1,024 configurations, Holm corrected within a separate three-contrast family.

MethodMean±SDDL−method [95% CI], ppHolm p
DataLens91.66±1.06——
Fisher89.66±1.34+2.00 [1.24,2.76].005859
q_dir89.52±1.17+2.14 [1.45,2.83].005859
Random88.66±1.21+3.00 [2.28,3.72].005859

Raw sign-flip p=.001953 for each contrast; CIs condition on the fixed test items and are not simultaneous. The separate configuration uses Transformers 5.3.0/TRL 0.29.0, no vLLM, text-only inputs, thinking disabled, greedy decoding (max 2,048 tokens).

R10. Alternative validation source / full-algorithm comparator. Qwen2.5-Math-7B, N=100, 200 updates; 256 GSM8K training examples replace OpenR1-Math scoring validation.

SourceBenchmarkDL−GradAlign-full (pp)
GSM8K trainMATH-500+3.3
GSM8K trainOlympiadBench+3.0

These additions directly address the uncertainty concern without overstating it: R9 gives significant differences on this newer model, and R10 gives positive means against GradAlign-full under a changed validation source. Neither resolves the five uncertain original Fisher gaps; R10 has no paired significance or equal-compute claim. We revised the abstract and conclusion accordingly.

Revisions: Secs. 4.5–4.6, Tables 4–7; Apps. D.10–D.12/Table 34.

AuthorsOpenReview ↗

Looking Forward to Further Discussion with You

Dear Reviewer YCB7,

Thank you again for your valuable time and constructive feedback.

As the rebuttal period is nearing its end, we would like to kindly check whether our previous response has addressed your main concerns. If there are any aspects that you feel need further clarification or improvement, please let us know, and we would be more than happy to provide additional explanations or revisions.

Best regards, Authors

Thank you for the detailed rebuttal and additional experiments.

I appreciate the substantial efforts to address my concerns. The new results strengthen the empirical evidence, particularly regarding the role of curvature and comparisons with stronger baselines.

However, my main concern about limited methodological novelty remains. Moreover, the improvements over Fisher in five of the original six settings are still statistically inconclusive.

Therefore, I will maintain my current rating. Thank you again for your thorough responses.

Reviewer_touNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

DataLens is a selection-time data selection method for LLM reinforcement learning. It argues that first-order gradient alignment (LESS/LearnAlign/GradAlign) only captures local signal, while finite-step update outcomes also depend on target-objective curvature along the candidate-induced direction.

It is evaluated at three levels (controlled probe utility, short-horizon GRPO bridge, and full 200-step GRPO training) across six model-domain settings (Qwen/DeepSeek/Mistral × math/code), with extensive robustness checks.

Critical strengths

The problem framing is sharp and genuinely useful. Distinguishing "local gradient alignment" from "finite-step target-objective change" is a clean conceptual contribution that directly targets a blind spot in influence-based data selection.

The Qwen-Math result is strong and well-supported. 100 selected examples reach 84.0% MATH-500 vs. 70.1% for first-order alignment and 67.5% for random, stable across training seeds and selection randomness.

The three-tier evaluation design is above average for this subfield. Controlled probes, an independent bridge split, and full training runs are kept separate, and the theory section explicitly disclaims subset optimality rather than overclaiming.

Critical weaknesses

-The strongest curvature baseline is buried in the appendix. Empirical-Fisher is omitted from the main Table 3, yet at N=100 it trails DataLens by only .003–.007 in five of six settings — presenting it only in the appendix makes DataLens's advantage look larger than it is.

  • The "all 24 settings" claim is literally false. At DeepSeek-Math N=200 and Mistral-Code N=25, q_dir edges out DataLens by .002; the paper should say 22/24 and acknowledge the remaining two are within noise.

  • The Qwen-Math outlier lacks a mechanism. A +14% gain there versus +2 to +6 elsewhere plausibly reflects D_v–benchmark distribution alignment (both from OpenR1-Math/MATH-500) rather than the method being generally stronger, but no diagnostic separates these explanations.

  • The baseline method is too simplistic. At least for a significant number of the methods developed in recent years, this is not the case.

  • The base model Qwen2.5 is a bit outdated. Qwen 3 or 3.5 would be a more suitable choice.

  • The bridge-to-training gap is never explained. Prediction correlation drops monotonically with GRPO steps (r = .56 → .37 over 20 steps), yet the method is used for 200-step runs with a +14% payoff

Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer touN — Part 1/2: Strong Baselines, Corrected Claims, and Qwen-Math Analysis

We sincerely thank the reviewer for the careful assessment and constructive suggestions. Your comments helped us improve the strong-baseline comparison, correct an overstated result, and better distinguish local prediction from final RL performance. We address each concern below.

1. Strong empirical-Fisher baseline

We agree that leaving empirical Fisher in the appendix made the comparison insufficiently prominent. We have moved Fisher and one-shot LearnAlign/GradAlign adaptations into the main text (Sec. 4.5, Table 4), and added paired uncertainty analysis.

R1. N=100 final performance (three-seed means, %; math: MATH-500 accuracy; code: LiveCodeBench pass@1).

SettingDataLensFisherU1DL−Fisher (pp)
Qwen-Math84.076.570.1+7.5
DeepSeek-Math72.071.568.6+0.5
Mistral-Math69.068.765.7+0.3
Qwen-Code45.845.142.8+0.7
DeepSeek-Code43.342.840.4+0.5
Mistral-Code41.540.938.6+0.6

R2. Prediction of measured projected finite-step utility (Spearman).

SettingDataLensFisherDifference
Qwen-Math.660.561+.099
DeepSeek-Math.587.567+.020
Mistral-Math.536.522+.014
Qwen-Code.558.539+.019
DeepSeek-Code.528.512+.016
Mistral-Code.497.481+.016

R2a. Ten-run Fisher uncertainty (DL−Fisher, pp; pointwise joint seed/item bootstrap 95% CIs; Holm-adjusted p over the 12-contrast family).

SettingDifference [95% CI]Holm p
Qwen-Math+7.50 [6.22,8.82].0006
DeepSeek-Math+0.50 [−0.60,1.60]1.0000
Mistral-Math+0.30 [−1.00,1.62]1.0000
Qwen-Code+0.68 [−0.53,1.90]1.0000
DeepSeek-Code+0.51 [−0.75,1.76]1.0000
Mistral-Code+0.59 [−0.62,1.85]1.0000

Fisher uses observed-target gradient outer products; DataLens uses target-objective directional curvature along the calibrated candidate displacement. Both ranking correlation and final mean performance are higher in all six settings. We agree, however, that higher means are not necessarily significant. In the new ten-paired-run analysis, Qwen-Math has a supported Fisher gap of +7.50 pp (95% CI [6.22,8.82], Holm p=.0006); the other five gaps of +0.30–0.68 pp have intervals crossing zero. We now state this limitation explicitly (Sec. 4.5, Tables 4–5; App. A.7/A.9, D.1/D.10).

2. Correction of the 24/24 claim

We appreciate the reviewer catching this error. We rechecked the full model–domain–budget grid and corrected the manuscript: 24/24 higher means versus U1 and raw q; 22/24 versus q_dir.

R3. Two exceptions against q_dir (three-seed means, %).

SettingDataLensq_dirDL−q_dir (pp)
DeepSeek-Math, N=20073.673.8−0.2
Mistral-Code, N=2535.836.0−0.2

These counts concern mean ordering, not statistical superiority in all 24 cells. The correction appears in Sec. 4.5, Table 3, and the conclusion.

3. Why is Qwen-Math's gain larger?

We agree that the 84.0% result needs a mechanism-oriented analysis beyond the final score. We added a measured ranking reversal and Top-100 tail diagnostics, then tested a different scoring-validation source.

R4. Measured ranking reversal (Qwen-Math; all entries ×10⁻²).

QuantityCandidate ACandidate B
First-order U12.402.05
Curvature correction−1.82−0.31
DataLens prediction0.581.74
Measured probe utility0.551.69

U1 favors A; A's larger negative curvature correction reverses the predicted order, agreeing with directly measured finite-step utility. This example concerns individual prescribed displacements, not the causal effect of 200 GRPO steps.

R5. Top-100 recovery within the same 441-candidate audit.

MetricRaw qDataLens
Audit Recall@100.72.83
Mean selected utility (z).670.821

DataLens recovers 83 versus 72 of the measured Top-100 candidates. This is an audit-level tail result, not a full-pool oracle recall or a significance test.

To test validation-source sensitivity, we replaced the 256-example OpenR1-Math scoring-validation set with 256 GSM8K training examples, retaining Qwen2.5-Math-7B, N=100, and 200 updates; GSM8K is not used for final evaluation.

R6. Alternative validation source: DataLens versus GradAlign-full.

SourceEvaluationDL−GradAlign-full (pp)
GSM8K trainMATH-500+3.3
GSM8K trainOlympiadBench+3.0

These experiments provide a concrete example of curvature-informed reranking, improved audit-tail recovery, and positive cross-source means. They do not prove that curvature causes the full 200-step gain; the new validation source also changes the selected subset. We now separate the controlled probe, GRPO bridge, and final training outcomes (Secs. 3.4, 4.2–4.3, 4.6; Tables 7, 20–21; Apps. C.1–C.2, D.12).

AuthorsOpenReview ↗

Response to Reviewer touN — Part 2/2: Recent Methods, New-Model Evaluation, and GRPO Bridge

We thank the reviewer again for encouraging stronger comparisons, newer-model evaluation, and a more precise interpretation of the bridge experiment.

4. Recent selection methods and compute fairness

We agree that U1 is a Taylor term, not a substitute for recent full selection algorithms. We expanded related work and added LearnAlign/GradAlign one-shot adaptations alongside Fisher under a common initial state, cached rollouts, N=100, and 200 GRPO updates.

R7. Strong one-shot selectors (three-seed final means, %).

SettingDataLensLearnAlign-styleGradAlign-styleFisher
Qwen-Math84.075.275.676.5
DeepSeek-Math72.071.171.371.5
Mistral-Math69.068.468.668.7
Qwen-Code45.844.844.945.1
DeepSeek-Code43.342.342.542.8
Mistral-Code41.540.540.740.9

DataLens has higher means in all six settings. We explicitly label these as static scoring adaptations, not faithful evaluations of the complete adaptive algorithms. Their online variants may update representations, rollouts, and selections during training.

We also corrected the GPU cost comparison by charging shared preprocessing once per standalone selector. All values below are H100-hours; verifier CPU-hours are accounted for separately.

R8. Selection and downstream compute.

ComponentMathCode
Shared eligibility8.7311.38
DataLens extra scoring3.803.68
Full DataLens selection12.5315.06
GRPO, 200 updates29.1032.40
DataLens total41.6347.46
Random total, 200 updates37.8343.78

Thus DataLens's extra scoring is 3.80/3.68 H100-hours beyond the shared cache, not the full 12.53/15.06 selection cost. We agree that metered, equal-total-compute comparisons against full online selectors remain necessary and no longer claim that the static rows settle that question (Secs. 2, 4.5–4.6; App. A.7/A.12, Tables 16–17).

5. Evaluation on a newer model

We appreciate this suggestion and added Qwen/Qwen3.5-9B evaluated on all 500 official MATH-500 questions (N=100, 200 updates, ten paired seeds).

R9. Qwen3.5-9B (mean±sample SD, %; DL difference in pp; paired seed-t 95% CI; Holm correction across three contrasts).

MethodAccuracyDL−method [95% CI]Holm p
DataLens91.66±1.06——
Fisher89.66±1.34+2.00 [1.24,2.76].005859
q_dir89.52±1.17+2.14 [1.45,2.83].005859
Random88.66±1.21+3.00 [2.28,3.72].005859

Exact two-sided sign-flip tests enumerate 1,024 paired configurations; raw p=.001953 for each contrast. The pointwise CIs condition on the fixed 500 items and assume approximately normal paired seed differences; they are not simultaneous. The environment uses Transformers 5.3.0/TRL 0.29.0, no vLLM, text-only inputs with thinking disabled, and greedy evaluation (2,048-token limit).

The three differences remain significant after Holm correction, supporting this specific newer-model setting. We do not use this extension to infer significance for the five uncertain Fisher gaps in the original six settings (Sec. 4.6; App. D.11/Table 34).

6. Declining bridge correlation versus final accuracy

We agree that a 20-step bridge score cannot be extrapolated to explain 200-step performance. We clarified that controlled finite-step probe utility, short-horizon GRPO gain, and final generative accuracy are separate evaluation targets.

The bridge uses 64 disjoint groups of eight prompts per setting, fresh rollouts at each update, and an independent evaluation split.

R10. Spearman correlation with realized GRPO gain (six-setting means).

StepsU1Raw qq_dirDataLens
1.16.43.40.53
5.14.40.37.50
10.11.35.33.45
20.08.28.26.38

The reviewer-cited .56→.37 decline is for Qwen-Math; .53→.38 is the six-setting mean. For Qwen-Math, the paired DataLens−q interval excludes zero at steps 1, 5, and 10, but includes zero at 20. DataLens retains the highest mean correlation at each horizon, yet all local scores lose predictive power over time. Parameter drift, optimizer states, and changing rollouts offer a plausible explanation, not a verified cause of the final gain (Sec. 3.4, Sec. 4.4/Table 2; App. D.7–D.8/Table 31; conclusion).

AuthorsOpenReview ↗

Looking Forward to Further Discussion with You

Dear Reviewer touN,

Thank you again for your valuable time and constructive feedback.

As the rebuttal period is nearing its end, we would like to kindly check whether our previous response has addressed your main concerns. If there are any aspects that you feel need further clarification or improvement, please let us know, and we would be more than happy to provide additional explanations or revisions.

Best regards, Authors

Thank you for your reply. I will keep my scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.