- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how the optimizer used for scoring affects online data selection during pretraining. It builds on OPUS and compares several scoring rules and training optimizers on 124M and 720M models. One main finding is that AdamW scoring works better than Muon scoring even when the model is trained with D-Muon. The paper also provides theoretical analyses intended to explain these differences.
Critical strengths
None
Critical weaknesses
- I am not convinced that the experiments establish an advantage of one scoring geometry over another. Different scoring rules change both the direction and the magnitude of the transformed gradients. In Eq. (1), multiplying these gradients by (c) multiplies the alignment term by (c), but the redundancy penalty by (c^2). The magnitude of the scores also affects how concentrated the Boltzmann sampling is. Therefore, using the same temperature and the same scoring formula does not necessarily give different scorers comparable selection behavior. This leaves a basic question unanswered: does AdamW scoring identify better samples, or does it happen to give a more suitable balance between alignment, redundancy, and selection strength? The learning-rate sweeps reported in the paper do not resolve this question.
- I also do not see a sufficiently direct connection between the theory and the implemented method. For Muon, the radial-collapse result concerns the derivative of the polar map, but the experiments use a right-preconditioned approximation. The paper itself states that the radial-collapse guarantee does not apply to this approximation. For Lion, there is a similar distinction between the exact sign map being analyzed and the linear surrogate being used. These results may be mathematically informative, but they do not yet explain why the implemented scorers perform as reported.
- The additional contribution over OPUS and the practical value for pretraining are not clear enough to me. OPUS already studies optimizer-aware online data selection with AdamW and Muon. Adding more optimizers broadens the comparison, but I do not find the broader comparison alone sufficient, especially when the explanation of the results remains unclear. The AdamW-scoring/Muon-training finding could be a meaningful contribution, but it needs the controls discussed above.
- The computational cost is another major concern. With an oversampling ratio of two, the method performs forward–backward computation on twice as many tokens as it uses for updates. Why is this preferable to simply training on those tokens? The paper's own results show that, at equal read tokens, the method falls behind standard training at both model scales. I appreciate that the authors report this clearly, but it leaves the practical motivation unresolved. The results show that selection can reduce the number of applied tokens; they do not show that it makes pretraining more efficient. P.S. The experiments are also limited to 124M and 720M models, with one seed per run. I would be cautious about drawing conclusions for larger-scale pretraining from this evidence. PPS: I also question whether the gains shown here establish a benefit for general-purpose pretraining. The selection is guided by examples from five downstream benchmarks, including their correct answers, so improvements on those tasks are not particularly surprising. Even with separate evaluation examples, these tasks have already influenced which data the model trains on. The paper does evaluate four additional tasks, but this is still a narrow basis for concluding that the resulting model is a better general-purpose base model. I would like to understand what capabilities are gained or lost through this targeted selection, and whether the overall tradeoff is favorable under the same compute budget.
Questions for authors
-
In the cross-scorer experiments, how are score magnitude, sampling temperature, and redundancy strength controlled? Does AdamW scoring still outperform Muon scoring if the redundancy term is removed and deterministic top-B selection is used? This would help separate sample ranking from score-scale effects.
-
What evidence shows that the collapse mechanisms analyzed in the theory explain the behavior of the implemented scorers? In particular, the implemented Muon scorer does not have the radial-collapse property used in the theoretical argument.
-
Could the approximation "H_F \approx I" affect the comparison between scoring rules? Different transformed gradients may interact differently with the curvature that this approximation removes. I am not claiming that this necessarily disadvantages Muon, but I would like to see whether the ranking survives removing the redundancy term or using a more accurate curvature estimate in a small-scale test.
7 comments
Rebuttal (2/3)
W3. Our contribution beyond OPUS, on the theoretical and the empirical side
We would like to clarify that our paper does not propose a new scoring rule. It studies the question that OPUS [1] leaves open: how does an optimizer's preconditioner affect online data selection? Building on the online-selection paradigm of OPUS, our contributions are both theoretical and empirical.
Theoretical contributions
- A selection-gain principle (Theorem 1). The per-step advantage of selection over uniform sampling is . It is set by the spread (the discriminability) of the optimizer-induced utility scores, times a constant that depends only on the retained fraction. OPUS gives no account of when, or by how much, selection should help.
- Optimizer-specific predictions (Theorems 2–4). A sign-based operator cannot amplify the score spread beyond that of SGD (GREATS [2]), and the exact sign score collapses under momentum dominance (Theorem 2). A diagonal-adaptive operator, as in AdamW and AdEMAMix, can amplify the benefit of data selection (Theorem 3). Muon's polar operator sees only the tangential part of a candidate and discards the radial part (Theorem 4). Together these predict which optimizers benefit from optimizer-aware selection before any training run is launched.
- The optimal candidate multiplier (Lemma 1). We derive the candidate multiplier that maximizes the gain per streamed token, =2, and verify it in LLM experiments. It is a critical hyperparameter of online data selection, yet in GREATS [2] and OPUS [1] it is set heuristically, without further justification.
Empirical contributions
- A benchmark across optimizers and scales. We compare standard training and OptiSelect for five optimizers at 124M and 720M, and for AdamW and D-Muon at 1.5B in the revision; OPUS evaluates AdamW and Muon. The benefit of selection turns out to depend strongly on the optimizer (Table 1, Figure 2).
- The scoring geometry decoupled from the training optimizer (§5.2, Figure 4). OPUS pairs each optimizer with its own preconditioner. We show empirically that the matched geometry is not necessarily the best one: training with D-Muon but scoring in AdamW's geometry beats scoring in D-Muon's own at 124M, and at 1.5B in both downstream loss and benchmark accuracy (revision, Appendix H.1, Table 12).
- Validation of the candidate multiplier (§5.4, Figure 7). Across values of , the measured selection gain follows the predicted , and the gain per streamed token peaks at .
- Interaction with data rephrasing (§5.3). The selection benefit persists when half or all of the corpus is rephrased, so selection and rephrasing are complementary rather than substitutes.
W2 & Q2. What our theory does and does not predict
We agree with the reviewer that our collapse theory does not fully explain the behaviour of the implemented surrogates for Lion and D-Muon. We use the surrogates mainly for efficiency, because they avoid per-document operations, but the approximation costs accuracy in the utility scores. For the diagonal-adaptive optimizers AdamW and AdEMAMix, where the implemented operator is the one the theory analyses, Theorem 3 predicts that optimizer-aware selection outperforms selection on raw gradients, and this agrees with our results (Figure 16, Table 15). Sophia is the exception: the theorem places it with AdamW, but raw gradients are ahead there, and we report it as such in Appendix H.4.
Rebuttal (3/3)
W4 (I). Performance at matched read tokens
We agree that at our experimental scales (124M and 720M models, trained on 4.2B and 14B tokens), OptiSelect wins at equal applied tokens (those used in optimizer steps) but not at equal read tokens, which is what determines the actual compute budget. The gap is smaller at the larger scales, however: with AdamW, OptiSelect reaches the final downstream loss of standard training with 1.43× fewer applied tokens at 124M, 1.84× fewer at 720M and 1.80× fewer at 1.5B, where 2× would break even on read tokens at .
In addition, OPUS [1] presents evidence at a larger model scale and token budget. According to their Figure 1, early in training (before 10–15B tokens) online selection does not yet make up for reading twice the tokens, while late in training (160–200B tokens) it approaches an 8× acceleration over standard training.
W4 (II). Larger models and robustness across seeds
1. Larger models. The revision adds experiments at 1.5B (Appendix H.1). With AdamW, OptiSelect outperforms standard training in both downstream loss (Figure 13) and benchmark accuracy (Table 11). With D-Muon, the picture matches our evidence at 124M: the benefit of selecting in D-Muon's own geometry disappears by the end of training and turns slightly negative (the solid green curve in Figure 13). In contrast, selecting in AdamW's geometry outperforms both the matched geometry and standard training, in downstream loss and in accuracy (Table 12).
| 1.5B model | final downstream loss | vs. standard | avg. accuracy, 10 tasks (%) | vs. standard |
|---|---|---|---|---|
| AdamW, standard | 2.8295 | 56.72 | ||
| AdamW + OptiSelect | 2.7743 | −0.055 | 59.24 | +2.52 |
| D-Muon, standard | 2.8111 | 59.82 | ||
| D-Muon + OptiSelect, D-Muon's own geometry | 2.8692 | +0.058 | 58.54 | −1.28 |
| D-Muon + OptiSelect, AdamW's geometry | 2.7921 | −0.019 | 61.50 | +1.69 |
2. Multiple random seeds. We repeated the 124M AdamW experiment with three random seeds (Figure 15, Table 14). The result is robust across seeds: the gain is 0.114 ± 0.004 nats.
Downstream probe loss at 128k steps:
| seed 0 | seed 1 | seed 2 | mean ± s.d. | |
|---|---|---|---|---|
| standard | 3.3599 | 3.3654 | 3.3680 | 3.3644 ± 0.0042 |
| OptiSelect | 3.2414 | 3.2521 | 3.2580 | 3.2505 ± 0.0085 |
| gain | 0.1185 | 0.1133 | 0.1100 | 0.1140 ± 0.0043 |
W4 (III). What do the benchmark-guided experiments establish about transfer?
(1) No evaluations are leaked from proxy: The five proxy task families enter data selection through their prompts and correct completions. Evaluating on separate examples from those families therefore measures generalization to the targeted tasks.
(2) Broad capabilities and benchmarks: As requested by the reviewer, we evaluate on 5 more benchmarks on our 1.5B model. The results are shown in the paper revision (Table 11 and 12) and also presented below.
With AdamW, selection raises the ten-task mean from 56.72% to 59.24% at equal applied tokens. Among the five tasks outside the proxy, BoolQ and COPA improve by 10.45 and 11.00 percentage points, WinoGrande and OpenBookQA by 1.42 and 2.20, while LAMBADA declines by 1.35. With D-Muon, the outcome depends on the scoring geometry: selecting in D-Muon's own geometry lowers the ten-task mean from 59.82% to 58.54%, while selecting in AdamW's geometry raises it to 61.50%.
| task | metric | AdamW, standard | AdamW + OptiSelect |
|---|---|---|---|
| PIQA | 73.72 | 73.07 (−0.65) | |
| HellaSwag | 51.95 | 52.90 (+0.95) | |
| ARC-e | 61.20 | 61.85 (+0.65) | |
| ARC-c | 32.59 | 33.45 (+0.85) | |
| SciQ | 80.50 | 80.20 (−0.30) | |
| WinoGrande | 56.91 | 58.33 (+1.42) | |
| OBQA | 36.80 | 39.00 (+2.20) | |
| LAMBADA | 56.20 | 54.85 (−1.35) | |
| BoolQ | 48.35 | 58.80 (+10.45) | |
| COPA | 69.00 | 80.00 (+11.00) | |
| Average | 56.72 | 59.24 (+2.52) |
| task | D-Muon, standard | D-Muon + OptiSelect, D-Muon's own geometry | D-Muon + OptiSelect, AdamW's geometry |
|---|---|---|---|
| PIQA | 73.67 | 73.72 (+0.05) | 74.54 (+0.87) |
| HellaSwag | 54.00 | 53.00 (−1.00) | 54.30 (+0.30) |
| ARC-e | 63.90 | 62.25 (−1.65) | 66.00 (+2.10) |
| ARC-c | 34.22 | 35.32 (+1.11) | 38.65 (+4.44) |
| SciQ | 84.90 | 82.50 (−2.40) | 85.00 (+0.10) |
| WinoGrande | 60.54 | 57.30 (−3.24) | 59.04 (−1.50) |
| OBQA | 36.00 | 37.20 (+1.20) | 38.20 (+2.20) |
| LAMBADA | 59.35 | 55.10 (−4.25) | 59.60 (+0.25) |
| BoolQ | 57.60 | 55.00 (−2.60) | 60.70 (+3.10) |
| COPA | 74.00 | 74.00 (+0.00) | 79.00 (+5.00) |
| Average | 59.82 | 58.54 (−1.28) | 61.50 (+1.69) |
We hope our answer and additional results can resolve your concerns. If you have any further questions, we are happy to address!
[1] OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration
[2] GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration
Rebuttal (1/3)
We thank the reviewer for the careful assessment and for the close reading of our theory. We respond to each concern below.
W1 & Q1. Derivation of the redundancy term, and why the learning rate is the only hyperparameter
The c and c2 you point to are not coefficients that we tune. They are the scalings of the first- and second-order terms of the expansion of the marginal utility of a candidate. The revision states this expansion with an explicit remainder as Proposition 6 (Appendix H.3). In the derived utility the learning rate ηt is the only hyperparameter: it alone sets the balance between the two terms.
Proposition 6 (Marginal utility: alignment and redundancy). Let F:Rd→R be twice continuously differentiable with ∥∇2F∥≤M and an L-Lipschitz Hessian, and write gV=∇F(θt) and HF=∇2F(θt). Fix a step t, let B(t) be the set of samples already selected at this step, and for a candidate z write its step contribution Δz=ηtOt(gz). Put v=−∑z′∈B(t)Δz′, so that the already-selected point is θsel=θt+v. Then the marginal utility U(z,t)=F(θsel)−F(θsel−Δz) satisfies
The first term is the alignment and the second is the redundancy, with NO extra hyperparameters. The proposition and full proof are provided in Appendix H.3.
Q1. Does AdamW scoring still outperform D-Muon scoring without the redundancy term and Boltzmann sampling?
Yes. We ran the ablation you propose (Appendix H.2): the redundancy term is removed and selection is deterministic top-B, so that selection depends only on the alignment with the validation-proxy gradient. Final downstream loss at 124M with D-Muon as the training optimizer (lower is better):
Two takeaways (Figure 14, Table 13):