acceptodds
Under review as a conference paper at ICLR 2027

OptiSelect: How does the Optimizer Shape Data Curriculum?

Abstract

Online data selection has demonstrated substantial efficiency gains for LLM pretraining by training on the most valuable candidates within each batch. Since a candidate's value is realized through its effective model update, principled selection should account for the optimizer step, which reshapes the raw gradient before it updates model parameters. We formalize this optimizer-aware selection paradigm as OptiSelect and present the first systematic study of how the optimizer shapes data selection. Our theory establishes a selection gain principle in which the advantage of online selection is governed by the discriminability of the optimizer-induced utility scores. We prove that sign-based and polar-tangential preconditioner of \Lion and \Muon would suffer from a discriminability collapse which caps attainable gains from OptiSelect, whereas diagonal-adaptive optimizers such as \AdamW and \Sophia admit strictly better upper bounds. The proposed principle also quantitatively derives the optimal candidate oversampling ratio. Pretraining experiments on 124M and 720M models conform with our theoretical analysis and show that \AdamW's diagonal-adaptive scoring geometry remains the strongest scoring geometry even under \Muon optimizer. We further demonstrate that OptiSelect maintains the benefits under modern rephrasing, which applies in modern data processing pipeline. Our findings provide theoretical foundations and practical guidance for co-designing optimizers and data selection in LLM pretraining.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
data, LLM, optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kyCEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the optimizer used for scoring affects online data selection during pretraining. It builds on OPUS and compares several scoring rules and training optimizers on 124M and 720M models. One main finding is that AdamW scoring works better than Muon scoring even when the model is trained with D-Muon. The paper also provides theoretical analyses intended to explain these differences.

Critical strengths

None

Critical weaknesses
  • I am not convinced that the experiments establish an advantage of one scoring geometry over another. Different scoring rules change both the direction and the magnitude of the transformed gradients. In Eq. (1), multiplying these gradients by (c) multiplies the alignment term by (c), but the redundancy penalty by (c^2). The magnitude of the scores also affects how concentrated the Boltzmann sampling is. Therefore, using the same temperature and the same scoring formula does not necessarily give different scorers comparable selection behavior. This leaves a basic question unanswered: does AdamW scoring identify better samples, or does it happen to give a more suitable balance between alignment, redundancy, and selection strength? The learning-rate sweeps reported in the paper do not resolve this question.
  • I also do not see a sufficiently direct connection between the theory and the implemented method. For Muon, the radial-collapse result concerns the derivative of the polar map, but the experiments use a right-preconditioned approximation. The paper itself states that the radial-collapse guarantee does not apply to this approximation. For Lion, there is a similar distinction between the exact sign map being analyzed and the linear surrogate being used. These results may be mathematically informative, but they do not yet explain why the implemented scorers perform as reported.
  • The additional contribution over OPUS and the practical value for pretraining are not clear enough to me. OPUS already studies optimizer-aware online data selection with AdamW and Muon. Adding more optimizers broadens the comparison, but I do not find the broader comparison alone sufficient, especially when the explanation of the results remains unclear. The AdamW-scoring/Muon-training finding could be a meaningful contribution, but it needs the controls discussed above.
  • The computational cost is another major concern. With an oversampling ratio of two, the method performs forward–backward computation on twice as many tokens as it uses for updates. Why is this preferable to simply training on those tokens? The paper's own results show that, at equal read tokens, the method falls behind standard training at both model scales. I appreciate that the authors report this clearly, but it leaves the practical motivation unresolved. The results show that selection can reduce the number of applied tokens; they do not show that it makes pretraining more efficient. P.S. The experiments are also limited to 124M and 720M models, with one seed per run. I would be cautious about drawing conclusions for larger-scale pretraining from this evidence. PPS: I also question whether the gains shown here establish a benefit for general-purpose pretraining. The selection is guided by examples from five downstream benchmarks, including their correct answers, so improvements on those tasks are not particularly surprising. Even with separate evaluation examples, these tasks have already influenced which data the model trains on. The paper does evaluate four additional tasks, but this is still a narrow basis for concluding that the resulting model is a better general-purpose base model. I would like to understand what capabilities are gained or lost through this targeted selection, and whether the overall tradeoff is favorable under the same compute budget.
Questions for authors
  1. In the cross-scorer experiments, how are score magnitude, sampling temperature, and redundancy strength controlled? Does AdamW scoring still outperform Muon scoring if the redundancy term is removed and deterministic top-B selection is used? This would help separate sample ranking from score-scale effects.

  2. What evidence shows that the collapse mechanisms analyzed in the theory explain the behavior of the implemented scorers? In particular, the implemented Muon scorer does not have the radial-collapse property used in the theoretical argument.

  3. Could the approximation "H_F \approx I" affect the comparison between scoring rules? Different transformed gradients may interact differently with the curvature that this approximation removes. I am not claiming that this necessarily disadvantages Muon, but I would like to see whether the ranking survives removing the redundancy term or using a more accurate curvature estimate in a small-scale test.

7 comments
AuthorsOpenReview ↗

Rebuttal (1/3)

We thank the reviewer for the careful assessment and for the close reading of our theory. We respond to each concern below.

W1 & Q1. Derivation of the redundancy term, and why the learning rate is the only hyperparameter

The cc and c2c^2 you point to are not coefficients that we tune. They are the scalings of the first- and second-order terms of the expansion of the marginal utility of a candidate. The revision states this expansion with an explicit remainder as Proposition 6 (Appendix H.3). In the derived utility the learning rate ηt\eta_t is the only hyperparameter: it alone sets the balance between the two terms.

Proposition 6 (Marginal utility: alignment and redundancy). Let F:Rd→RF: \mathbb{R}^d \to \mathbb{R} be twice continuously differentiable with ∥∇2F∥≤M\|\nabla^2 F\| \le M and an LL-Lipschitz Hessian, and write gV=∇F(θt)g_V = \nabla F(\theta_t) and HF=∇2F(θt)H_F = \nabla^2 F(\theta_t). Fix a step tt, let B(t)B^{(t)} be the set of samples already selected at this step, and for a candidate zz write its step contribution Δz=ηt Ot(gz)\Delta_z = \eta_t\, O_t(g_z). Put v=−∑z′∈B(t)Δz′v = -\sum_{z' \in B^{(t)}} \Delta_{z'}, so that the already-selected point is θsel=θt+v\theta_{\mathrm{sel}} = \theta_t + v. Then the marginal utility U(z,t)=F(θsel)−F(θsel−Δz)U(z,t) = F(\theta_{\mathrm{sel}}) - F(\theta_{\mathrm{sel}} - \Delta_z) satisfies

U(z,t)=⟨Δz,gV⟩−⟨Δz,HF∑z′∈B(t)Δz′⟩+RzU(z,t) = \langle \Delta_z, g_V \rangle - \langle \Delta_z, H_F \sum_{z' \in B^{(t)}} \Delta_{z'} \rangle + R_z, ∣Rz∣≤M2∥Δz∥2+L2∥v∥2∥Δz∥\quad |R_z| \le \tfrac{M}{2}\|\Delta_z\|^2 + \tfrac{L}{2}\|v\|^2 \|\Delta_z\|.

The first term is the alignment and the second is the redundancy, with NO extra hyperparameters. The proposition and full proof are provided in Appendix H.3.

Q1. Does AdamW scoring still outperform D-Muon scoring without the redundancy term and Boltzmann sampling?

Yes. We ran the ablation you propose (Appendix H.2): the redundancy term is removed and selection is deterministic top-B, so that selection depends only on the alignment with the validation-proxy gradient. Final downstream loss at 124M with D-Muon as the training optimizer (lower is better):

scoring geometryBoltzmann sampling, with redundancy (default)deterministic top-B, no redundancydifference
D-Muon's own3.47073.4529−0.0178
AdamW's3.41053.3864−0.0241
AdamW's − D-Muon's own−0.0602−0.0666

Two takeaways (Figure 14, Table 13):

  1. With the redundancy term removed and deterministic selection, AdamW's scoring geometry still beats D-Muon's own, by 0.067 nats against 0.060 with both components on.
  2. Removing the redundancy term and the Boltzmann sampling has only a small effect on the final loss. The ablated selector ends with 0.018 and 0.024 nats lower downstream losses in the two geometries, which slightly impact the performance.
AuthorsOpenReview ↗

Rebuttal (2/3)

W3. Our contribution beyond OPUS, on the theoretical and the empirical side

We would like to clarify that our paper does not propose a new scoring rule. It studies the question that OPUS [1] leaves open: how does an optimizer's preconditioner affect online data selection? Building on the online-selection paradigm of OPUS, our contributions are both theoretical and empirical.

Theoretical contributions

  1. A selection-gain principle (Theorem 1). The per-step advantage of selection over uniform sampling is Gt=cq⋅σUG_t= c_q \cdot \sigma_U. It is set by the spread (the discriminability) of the optimizer-induced utility scores, times a constant that depends only on the retained fraction. OPUS gives no account of when, or by how much, selection should help.
  2. Optimizer-specific predictions (Theorems 2–4). A sign-based operator cannot amplify the score spread beyond that of SGD (GREATS [2]), and the exact sign score collapses under momentum dominance (Theorem 2). A diagonal-adaptive operator, as in AdamW and AdEMAMix, can amplify the benefit of data selection (Theorem 3). Muon's polar operator sees only the tangential part of a candidate and discards the radial part (Theorem 4). Together these predict which optimizers benefit from optimizer-aware selection before any training run is launched.
  3. The optimal candidate multiplier (Lemma 1). We derive the candidate multiplier that maximizes the gain per streamed token, =2, and verify it in LLM experiments. It is a critical hyperparameter of online data selection, yet in GREATS [2] and OPUS [1] it is set heuristically, without further justification.

Empirical contributions

  1. A benchmark across optimizers and scales. We compare standard training and OptiSelect for five optimizers at 124M and 720M, and for AdamW and D-Muon at 1.5B in the revision; OPUS evaluates AdamW and Muon. The benefit of selection turns out to depend strongly on the optimizer (Table 1, Figure 2).
  2. The scoring geometry decoupled from the training optimizer (§5.2, Figure 4). OPUS pairs each optimizer with its own preconditioner. We show empirically that the matched geometry is not necessarily the best one: training with D-Muon but scoring in AdamW's geometry beats scoring in D-Muon's own at 124M, and at 1.5B in both downstream loss and benchmark accuracy (revision, Appendix H.1, Table 12).
  3. Validation of the candidate multiplier (§5.4, Figure 7). Across values of , the measured selection gain follows the predicted cq⋅σUc_q \cdot \sigma_U, and the gain per streamed token peaks at λ=2\lambda=2.
  4. Interaction with data rephrasing (§5.3). The selection benefit persists when half or all of the corpus is rephrased, so selection and rephrasing are complementary rather than substitutes.

W2 & Q2. What our theory does and does not predict

We agree with the reviewer that our collapse theory does not fully explain the behaviour of the implemented surrogates for Lion and D-Muon. We use the surrogates mainly for efficiency, because they avoid per-document operations, but the approximation costs accuracy in the utility scores. For the diagonal-adaptive optimizers AdamW and AdEMAMix, where the implemented operator is the one the theory analyses, Theorem 3 predicts that optimizer-aware selection outperforms selection on raw gradients, and this agrees with our results (Figure 16, Table 15). Sophia is the exception: the theorem places it with AdamW, but raw gradients are ahead there, and we report it as such in Appendix H.4.

AuthorsOpenReview ↗

Rebuttal (3/3)

W4 (I). Performance at matched read tokens

We agree that at our experimental scales (124M and 720M models, trained on 4.2B and 14B tokens), OptiSelect wins at equal applied tokens (those used in optimizer steps) but not at equal read tokens, which is what determines the actual compute budget. The gap is smaller at the larger scales, however: with AdamW, OptiSelect reaches the final downstream loss of standard training with 1.43× fewer applied tokens at 124M, 1.84× fewer at 720M and 1.80× fewer at 1.5B, where 2× would break even on read tokens at λ=2\lambda=2.

In addition, OPUS [1] presents evidence at a larger model scale and token budget. According to their Figure 1, early in training (before 10–15B tokens) online selection does not yet make up for reading twice the tokens, while late in training (160–200B tokens) it approaches an 8× acceleration over standard training.

W4 (II). Larger models and robustness across seeds

1. Larger models. The revision adds experiments at 1.5B (Appendix H.1). With AdamW, OptiSelect outperforms standard training in both downstream loss (Figure 13) and benchmark accuracy (Table 11). With D-Muon, the picture matches our evidence at 124M: the benefit of selecting in D-Muon's own geometry disappears by the end of training and turns slightly negative (the solid green curve in Figure 13). In contrast, selecting in AdamW's geometry outperforms both the matched geometry and standard training, in downstream loss and in accuracy (Table 12).

1.5B modelfinal downstream lossvs. standardavg. accuracy, 10 tasks (%)vs. standard
AdamW, standard2.829556.72
AdamW + OptiSelect2.7743−0.05559.24+2.52
D-Muon, standard2.811159.82
D-Muon + OptiSelect, D-Muon's own geometry2.8692+0.05858.54−1.28
D-Muon + OptiSelect, AdamW's geometry2.7921−0.01961.50+1.69

2. Multiple random seeds. We repeated the 124M AdamW experiment with three random seeds (Figure 15, Table 14). The result is robust across seeds: the gain is 0.114 ± 0.004 nats.

Downstream probe loss at 128k steps:

seed 0seed 1seed 2mean ± s.d.
standard3.35993.36543.36803.3644 ± 0.0042
OptiSelect3.24143.25213.25803.2505 ± 0.0085
gain0.11850.11330.11000.1140 ± 0.0043

W4 (III). What do the benchmark-guided experiments establish about transfer?

(1) No evaluations are leaked from proxy: The five proxy task families enter data selection through their prompts and correct completions. Evaluating on separate examples from those families therefore measures generalization to the targeted tasks.

(2) Broad capabilities and benchmarks: As requested by the reviewer, we evaluate on 5 more benchmarks on our 1.5B model. The results are shown in the paper revision (Table 11 and 12) and also presented below.

With AdamW, selection raises the ten-task mean from 56.72% to 59.24% at equal applied tokens. Among the five tasks outside the proxy, BoolQ and COPA improve by 10.45 and 11.00 percentage points, WinoGrande and OpenBookQA by 1.42 and 2.20, while LAMBADA declines by 1.35. With D-Muon, the outcome depends on the scoring geometry: selecting in D-Muon's own geometry lowers the ten-task mean from 59.82% to 58.54%, while selecting in AdamW's geometry raises it to 61.50%.

taskmetricAdamW, standardAdamW + OptiSelect
PIQA73.7273.07 (−0.65)
HellaSwag51.9552.90 (+0.95)
ARC-e61.2061.85 (+0.65)
ARC-c32.5933.45 (+0.85)
SciQ80.5080.20 (−0.30)
WinoGrande56.9158.33 (+1.42)
OBQA36.8039.00 (+2.20)
LAMBADA56.2054.85 (−1.35)
BoolQ48.3558.80 (+10.45)
COPA69.0080.00 (+11.00)
Average56.7259.24 (+2.52)
taskD-Muon, standardD-Muon + OptiSelect, D-Muon's own geometryD-Muon + OptiSelect, AdamW's geometry
PIQA73.6773.72 (+0.05)74.54 (+0.87)
HellaSwag54.0053.00 (−1.00)54.30 (+0.30)
ARC-e63.9062.25 (−1.65)66.00 (+2.10)
ARC-c34.2235.32 (+1.11)38.65 (+4.44)
SciQ84.9082.50 (−2.40)85.00 (+0.10)
WinoGrande60.5457.30 (−3.24)59.04 (−1.50)
OBQA36.0037.20 (+1.20)38.20 (+2.20)
LAMBADA59.3555.10 (−4.25)59.60 (+0.25)
BoolQ57.6055.00 (−2.60)60.70 (+3.10)
COPA74.0074.00 (+0.00)79.00 (+5.00)
Average59.8258.54 (−1.28)61.50 (+1.69)

We hope our answer and additional results can resolve your concerns. If you have any further questions, we are happy to address!

[1] OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

[2] GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration

Reviewer_9rZaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OptiSelect, the first systematic framework investigating optimizer-aware online data selection. They construct a selection gain principle: the benefit of online data selection depends on how well an optimizer's utility scores discriminate between different samples. They theoretically showed that optimizers like Lion and Muon suffer from a discriminability collapse that limits the selection gains, while diagonal-adaptive optimizers, like AdamW and Sophia, give higher performance limits.

Critical strengths

Experiments include pretrain of gpt small and large on fineweb, and report both the val loss/loss curve and the downstream benchmark results.

Critical weaknesses
  1. The paper used the same utility score as in the OPUS paper. Preconditioner matrix for different optimizer ( adamw and muon) are already provided by OPUS. The paper provide preconditioner for some new optimizer, but AdamW and Muon are currently the most widely used oprimizers. Therefore, I find little contribution of the paper.
  2. Even though used the same framework as OPUS, the author did not compare OptiSelect with OPUS. Besides, I think there are some other close related works like GREATS can be compared.

Wang, Jiachen T., Tong Wu, Dawn Song, Prateek Mittal, and Ruoxi Jia. "Greats: Online selection of high-quality data for llm training in every iteration." Advances in Neural Information Processing Systems 37 (2024): 131197-131223.

Questions for authors

In line 180, the author mentioned that utility scores follows a Gaussian distribution. Why is this true? Is it verified by experiment or theory? I suspect this does not hold, and therefore the theorem 1 and 2 are undermined.

5 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for their effort in understanding our paper and executable suggestions for our revision. We respond to each question as follows.

W1: Contribution of this paper

We agree on the starting point that we did adopt OPUS's utility score, its proxy construction and its sketching, and we cite OPUS at each reuse. However, we would like to clarify that our paper does not propose a new scoring rule but studies the question OPUS leaves open: How various optimizer’s preconditioner affect the online data selection procedure. Build upon the online data selection paradigm from OPUS [1], the contributions sit in both theoretical and empirical sides:

Theoretical contributions

  1. A selection-gain principle (Theorem 1). The per-step advantage of selection over uniform sampling is Gt=cq⋅σUG_t = c_q \cdot \sigma_U. It is set by the spread (the discriminability) of the optimizer-induced utility scores, times a constant that depends only on the retained fraction. OPUS gives no account of when, or by how much, selection should help.
  2. Optimizer-specific predictions (Theorems 2–4). A sign-based operator cannot amplify the score spread beyond SGD (GREATS [2]), where the exact sign score collapses under momentum dominance (Theorem 2). A diagonal-adaptive operator such as AdamW and Ademamix can amplify the benefits from data selection (Theorem 3). Muon's polar operator sees only the tangential part of a candidate and discards the radial part (Theorem 4). These predict which optimizers benefit from optimizer-aware selection before any training runs get launched.
  3. The optimal candidate multiplier (Lemma 1). We theoretically derive the optimal candidate multiplier λ⋆=2\lambda^{\star}=2 for the optimal gain-per-stream-token, and the results are verified on empirical LLM experiments. It is a critical hyperparameter in online data selection, but in GREATS [2] and OPUS [1] this ratio is set by heuristics with no further explanation.

Empirical contributions

  1. A benchmark across optimizers and scales. We compare standard training and OptiSelect for five optimizers at 124M and 720M, and for AdamW and D-Muon at 1.5B in the revision; OPUS evaluates AdamW and Muon. The benefit of selection turns out to depend strongly on the optimizer (Table 1, Figure 1).
  2. The scoring geometry decoupled from the training optimizer (§5.2, Figure 4). OPUS pairs each optimizer with its own preconditioner. Whereas we empirically show that the matched geometry is not necessarily the best one: training with D-Muon but scoring in AdamW's geometry beats scoring in D-Muon's own at 124M and at 1.5B (revision, Appendix H.1, Table 12) in both downstream loss and accuracies.
  3. Validation of the candidate multiplier (§5.4, Figure 7). Across various λ\lambda values, the measured selection gain follows the predicted cq⋅σUc_q\cdot \sigma_U, and the gain per streamed token peaks at λ = 2.
  4. Interaction with data rephrasing (§5.3). The selection benefit persists when half or all of the corpus is rephrased, so selection and rephrasing are complementary rather than substitutes.
AuthorsOpenReview ↗

Rebuttal (2/2)

W2 & Q1: Comparison with GREATS and what the results tell us

We add GREATS as one more baseline to all five optimizers and provide the results in the revision (Appendix H.2, Table 15, Figure 16). We clarify that GREATS compute data utility scores with raw training gradient and no optimizer-awareness, which is equivalent to the SGD geometry in section 5.3 . Our theory states each optimizer's discriminability relative to this SGD geometry, so it predicts the sign of "OptiSelect − GREATS" for every optimizer. We present the final downstream loss in the table below:

optimizerstandardOptiSelectGREATSsel − grttheory predicts
AdamW3.35993.24143.2868−0.046< 0 ✓
AdEMAMix3.40933.27443.3130−0.039< 0 ✓
Sophia3.48563.45803.3942+0.064< 0 ✗
Lion3.41693.39933.3644+0.035≥ 0 ✓
D-Muon3.45403.47073.4238+0.047≥ 0 ✓

Four out of five results are predicted from our theory, while only the result on Sophia is inconsistent with our prediction. Our hypothesis is that Sophia's diagonal is a stochastic curvature estimate, much noisier than a second moment and clipped at a floor. We provide more detailed, optimizer-wise analysis in Appendix H.2 .

Q2: Empirical measured utility score distribution and Gaussian assumption

We provide the empirically measured utility score distribution and superquantile cqc_q across various candidate multipliers λ\lambda in Appendix H.5, Figure 17 and Table 16. The empirical distributions are close to a Gaussian distribution. Also, intuitively, we expect the Web-scraped data contains a large portion of normal-quality data, while only including a few high-quality and low-quality documents. More statistics and explanations are provided in Appendix H.5.

c^q\hat{c}_q at λ\lambda = 2Gaussian (calculated)
AdamW + OptiSelect0.665 ± 0.0360.798
D-Muon, own geometry0.744 ± 0.0280.798
D-Muon, AdamW geometry0.744 ± 0.0410.798

We hope our answer and additional results can resolve your concerns. If you have any further questions, we are happy to address!

[1] OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

[2] GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration

AuthorsOpenReview ↗

Reminder of responses

Dear Reviewer, since the discussion period would end in less than two days, we gracefully remind you to read through our responses and revisions. We hope our answer and additional results can resolve your concerns and you can consider to raise your scores. If you have any further questions, we are happy to address!

Sincerely, Authors

Reviewer_DSbmOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Online data selection aims to select the most valuable candidates within each training batch to update the model parameters, leading to significant efficiency gains for LLM pre-training. This paper investigates the optimizer-aware data selection paradigm and studies how 5 different optimizers guide data selection. This paper introduces a selection-gain principle and shows that sign collapse bounds Lion's gain and radial collapse bounds D-Muon's, while AdamW and Sophia retain a strictly larger score spread. In addition, AdamW’s scoring is shown to achieve the best performance with the Muon optimizer.

Critical strengths
  • This paper offers a systematic study of how optimization approaches affect online data selection.
  • This paper proves some important points: Lion's gain is bounded by a sign collapse, and D-Muon's gain is bounded by a radial collapse, while AdamW and Sophia retain a strictly larger score spread.
  • This paper shows that AdamW is the best-scoring method under the Muon optimizer.
Critical weaknesses
  • The computational cost increases with the size of the validation dataset VV.
  • At λ=2\lambda = 2, each selection step executes a complete forward–backward pass on 2 ∣B(t)∣|B^{(t)}| tokens. When fairly compared to standard training with matched 2 ∣B(t)∣|B^{(t)}| tokens, OptiSelect is worse in both downstream loss and accuracy across 5 optimizers.
  • The authors pre-train 124M and 720M models with 1 seed, so results may vary across seeds or fail to hold for larger models.
Questions for authors
  1. Does the method require computing the validation gradient proxy at every selection step? If the size of the validation dataset is large, does that computation need more than 1 forward-backward pass?

  2. Approximating the Hessian matrix with an identity matrix will lose information. Have you tried other approximation methods?

  3. This is a minor point. What is S(t)S^{(t)} in lines 134 and 143? Is that a typo for B(t)B^{(t)}?

3 comments
AuthorsOpenReview ↗

Rebuttal (1/2)

We thank the reviewer for their constructive feedback and the effort spent on both our theory and empirical verifications. We provide the answers to your questions as follows:

W1 & Q1: Computation cost on the validation proxy

We agree with the reviewer that the computation cost to get the cost of computing the validation-proxy gradient grows with the size of the proxy set. In our implementation, the proxy set is a fixed set of 4096 densely packed 512-token sequences (≈\approx 2.1M tokens). Also, the proxy gradient is not recomputed at every selection step: it is computed once, cached, and refreshed every Δ=1000\Delta = 1000 steps, which amortizes the overhead. Appendix E.5 provides the ablation on the refresh interval at 124M scale on AdamW optimizer. Refreshing every 1,000 steps instead of every 500 halves the number of proxy passes with little cost in downstream loss (3.7178 against 3.7165)

W2: Weak performance at matched read tokens

We admit that at our experimental scales (124M/720M models with 15B tokens), OptiSelect only wins at equal applied tokens (used in optimizer step) but not at equal read tokens, which is what determines the actual compute budget. However, we show that the gap narrows with scale: with AdamW, OptiSelect reaches the final downstream loss of standard training with 1.43× fewer applied tokens at 124M and 1.84× fewer at 720M, approaching the equal read tokens baseline (λ=2×\lambda=2\times applied tokens). Besides, OPUS [1] presented empirical evidence at larger model scale and token budgets: according to Figure 1 in OPUS [1], in early stage training (before 10-15B tokens), the online selection cannot catch with half of read tokens, while in late-stage training (160-200B tokens) online selection approaches 8x acceleration in accuracy compared to standard training.

We would also like to clarify the contributions of our work. We take the online-selection paradigm of OPUS as given and ask the question that follows from it: how does the optimizer shape the selection, and in which geometry should candidates be scored? We provide theoretical and empirical evidence on how different optimizers change the selection pattern (Theorems 2-4) and how to choose the candidate multiplier (Section 5.4, Lemma 1), which was previously a heuristically tuned hyperparameter. Empirically, we find that the optimizer’s own geometry is not necessarily the best one for selection: when training with D-Muon, scoring candidates in AdamW’s geometry can outperform scoring in D-Muon’s own (Section 5.2, Figure 4). All of these comparisons are between selection arms that read the same number of tokens, so they do not depend on the read-token overhead.

W3: Larger-scale model and multiple seeds experiments

Larger model scale experiments: The revision adds experiments at 1.5B (Appendix H.1). With AdamW, OptiSelect outperforms standard training in both downstream loss (Figure 13) and benchmark accuracy (Table 11). With Muon, the picture matches our evidence at 124M: the benefit of selecting in D-Muon’s own geometry disappears by the end of training and turns slightly negative (the solid green curve in Figure 13). In contrast, selection in AdamW’s geometry outperforms both the matched geometry and standard training, in downstream loss and in accuracy (Table 12).

1.5B modelfinal downstream loss ↓\downarrowvs. standardavg. accuracy, 10 tasks (%) ↑\uparrowvs. standard
AdamW, standard2.829556.72
AdamW + OptiSelect2.7743−0.05559.24+2.52
D-Muon, standard2.811159.82
D-Muon + OptiSelect, D-Muon's own geometry2.8692+0.05858.54−1.28
D-Muon + OptiSelect, AdamW's geometry2.7921−0.01961.50+1.69

Multiple random seeds: We repeat the experiments at 124M with AdamW across 3 random seeds and report the loss curves in Figure 15 and Table 14. The results are robust across seeds. We also present the statistics in the table below. Downstream probe loss at 128k:

seed 0seed 1seed 2mean ± s.d.
standard3.35993.36543.36803.3644 ± 0.0042
OptiSelect3.24143.25213.25803.2505 ± 0.0085
gain0.11850.11330.11000.1140 ± 0.0043
AuthorsOpenReview ↗

Rebuttal (2/2)

Q2: Hessian Approximation

We thank the reviewer for this insightful question! We approximate the Hessian with identity following OPUS [1] since it prevent further computation overheads from recursive approximation (e.g. LiSSA [2], HyperINF [3]). We find that using the true curvature can be tractable in principle: each Hessian-vector product HFGtH_F G_t costs roughly one additional forward–backward pass on the proxy. It would, however, require substantial changes to our infrastructure, and we could not complete it within the rebuttal period. We leave it as a direction for future work. We empirically verify that the approximation does not drive our conclusions. It enters the score only through the redundancy term, while removing the redundancy term entirely leads to slight impact on the training curves (Appendix H.2, Figure 14, Table 13).

Q3: Typo correction

We thank the reviewer for catching this. We have corrected both S(t)S^{(t)} to B(t)B^{(t)} in our revised version.

We hope our answer and additional results can resolve your concerns. If you have any further questions, we are happy to address!


[1] OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration

[2] Second-Order Stochastic Optimization for Machine Learning in Linear Time

[3] HyperINF: Unleashing the HyperPower of the Schulz's Method for Data Influence Estimation

Thank you for providing your additional experimental results and clarifications. Most of my concerns have been sufficiently addressed; therefore, I maintain my rating leaning towards acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Paper Revision

We thank all three reviewers for their careful reading and constructive comments. We have uploaded a revised paper with new ablations and results. We provide the roadmap of new results in Appendix H as follows:

what is newwhere in the revision
Experiments at 1.5B, with AdamW and D-Muon, trained for the full 114k steps: downstream loss curves and accuracy on 10 benchmark tasks§H.1, Figure 13, Table 11 (AdamW), Table 12 (D-Muon)
Ablation of the selector: redundancy term removed and deterministic top-B selection, in both scoring geometries§H.2, Figure 14, Table 13
Seed spread at 124M: downstream loss over 3 seeds§H.2, Table 14, Figure 15
Derivation of the redundancy term with an explicit remainder§H.3, Proposition 6
Comparison with GREATS (the SGD scoring geometry) for all five optimizers at 124M§H.4, Table 15, Figure 16
Measured distribution of the influence scores against the Gaussian assumption of Theorem 1§H.5, Figure 17, Table 16
Notation: S^(t) corrected to B^(t)§3.2

Section, theorem, table and figure numbers in our responses refer to the revised paper. Each response below is self-contained.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.