- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper distinguishes between advancement from data curation methods, which improve a model's attainable performance, and acceleration, which reduces the training budget required to reach a given performance. The authors develop an analytical treatment using a teacher–student perceptron model, introduce a train–target loss discrepancy (the Δ-test), and provide experiments on language-model data mixture and selection. Overall, the paper's findings are unsurprising, the proposed test is weak, and it offers no meaningful additional insight.
Critical strengths
- The empirical evaluation includes several established data mixture and selection methods across different model scales.
- Distinguishing persistent performance improvements from finite-budget improvements is a useful consideration when evaluating data curation methods.
Critical weaknesses
1 Limited originality and a largely conceptual contribution.
The distinction between improving the attainable optimum and accelerating convergence is elementary in optimisation. The main theoretical statement, Proposition 1, establishes that if the baseline training objective coincides with the target objective and already attains a global minimum, no alternative allocation can improve the target optimum. This follows essentially by definition from the assumptions. The work feels closer to a position paper advocating a particular interpretation of existing results than a substantive technical contribution.
2 The Δ-test is insufficiently justified.
The theoretical argument concerns whether population training and target objectives coincide. However, the Δ-test measures differences in loss values at a single checkpoint. Two objectives can have identical loss values at a checkpoint while having different gradients and optima, or different losses while sharing the same optimum. Furthermore, Eq. (1) takes absolute values before aggregation, potentially reporting a positive discrepancy even when aggregate training and target losses coincide. If I understand correctly, the model is optimised with respect to aggregate loss, thus the Δ-test and training use slightly different definitions.
3 Unnecessarily flowery writing and overstated significance.
The manuscript repeatedly presents familiar optimisation observations as fundamental discoveries. Examples include the claim that a “single principle governs” data mixture and selection (lines 54–61), that the framework “cleanly explains recent contradictory findings” (lines 65–70), and that Proposition 1 establishes a “fundamental algebraic consequence” (lines 225–235). The rhetorical framing is considerably stronger than the theoretical and empirical evidence warrants, and the writing borders on unscientific and unfit for the venue.
4 Questionable practical significance.
Acceleration is treated as secondary to advancement, despite reducing training compute being a central motivation for data curation. Furthermore, the Δ-test is evaluated at 50% of the training budget, making its characterisation as a lightweight early diagnostic questionable. Evaluations at exponentially spaced percentages of the training budget (e.g., 1/16th, 1/8th, 1/4th, 1/2) would be preferable to understand how close to the end of training one must be for the test to be useful.
Questions for authors
- What is the nontrivial theoretical contribution of Proposition 1 beyond its assumptions?
- How can loss differences at a single checkpoint diagnose differences between attainable optima?
- Why does Eq. (1) take absolute values before aggregation? I may have missed if there was a strong motivation.
4 comments
Response to Reviewer wgaV (2/2)
Q3: Absolute values before aggregation. This is a deliberate design to prevent cancellation. Training minimizes the aggregate loss for fixed domain weights, whereas curation asks whether changing these weights helps, which depends on each domain separately. If the baseline over-serves one domain (e.g., through repetition) and under-serves another, their signed discrepancies can cancel in the aggregate even though reweighting would help. When the target has no counterpart in the training pool (held-out domain in the exogenous scenario and selection), there is a single group and reduces to the aggregate form, consistent with the training objective.
W3: Scope and strength of the claims
We have revised the passages the reviewer quoted and similar phrasing throughout. The three Propositions are now called Observations, the "fundamental algebraic consequence" phrasing is removed, and the "-test" is renamed the -diagnostic to reflect its role as an operational signal rather than a statistical test. All revised text is marked in blue.
W4: Practical value of acceleration and diagnostic cost.
Value of acceleration. We do not intend to present acceleration as secondary. To avoid this impression, we have revised the wording throughout. Our point is that the two effects should be measured differently. Advancement improves the endpoint and persists as the budget grows. Acceleration reduces the samples needed to reach a given performance, so its advantage in a fixed-budget comparison shrinks with the budget: in the no-gap mixture setting, mixtures that lead Uniform early see their advantage approach zero with training.
Early evaluation. Following the reviewer's suggestion, we evaluate at , , , and of the mixture budget (new Table 7, five seeds):
| Budget | 1/20 | 1/10 | 3/20 | 1/2 |
|---|---|---|---|---|
| No gap (null) | 0.026 ± 0.003 | 0.003 ± 0.001 | 0.013 ± 0.001 | 0.076 ± 0.001 |
| Endogenous | 0.041 ± 0.003 | 0.102 ± 0.002 | 0.158 ± 0.002 | 0.649 ± 0.013 |
| 5 | 155 | 147 | 537 | |
| Exogenous | 0.448 ± 0.012 | 0.465 ± 0.002 | 0.458 ± 0.004 | 0.468 ± 0.002 |
| 137 | 726 | 452 | 367 |
The exogenous gap is clear and stable from of the budget. The endogenous gap grows as small domains start being repeated and is clearly separated from the null by of the budget. Thus, need not be measured near the end of training: early diagnostics suffice to predict whether curation can advance, though the exact point may depend on the task.
We hope these revisions and new results address the reviewer's concerns, and we would appreciate it if the reviewer would re-evaluate the paper in light of them.
Thank you for the rebuttal. I believe the framing of the work is now much improved, as is the strength of the submission. However, due to the overall purely diagnostic nature of the work and the still unsatisfactory quality of the writing, I will refrain from recommending acceptance. However, I do increase my score from 1 to 2.
Thank you for the re-reading, for raising the score, and for the candid assessment.
On "purely diagnostic": we see it as a decision that should be made before any search is launched, and getting it wrong is expensive in LLM training practice. It answers whether curation can raise the endpoint at all, or only accelerate. If the latter, whether the gains are worth pursuing depends on the budget: they may shrink as the budget grows, while the cost of running the search does not. settles this from about 1/10 of a baseline run in our experiments.
We understand this is a different kind of contribution from a new method and respect the reviewer's weighting. We remain committed to further improving the writing. Any specific suggestions would be very welcome.
Response to Reviewer wgaV (1/2)
We thank the reviewer for the candid assessment. Several points led to concrete improvements, and we address each below.
What the paper studies. We believe there might be a misunderstanding here. The paper concerns data scaling rather than optimization: it studies how curation gains change with the sample budget under controlled baseline–target alignment, not how fast an optimizer converges to a given optimum. In our closed-form analysis (Section 3), at every budget the model is trained to convergence, so none of these gains comes from faster optimization. The question is whether a gain observed at a finite budget persists as the budget grows (advancement) or vanishes (acceleration), and how this depends on the train–target gap.
Role of Proposition 1. We agree that Proposition 1 is mathematically simple, and we have toned down its presentation (now Observation 1). Nevertheless, our central question is whether a curation gain reflects advancement or acceleration, and Proposition 1 gives only the negative half of the answer: without a gap, curation cannot advance. The nontrivial results lie beyond this. In a teacher–student model whose full performance-vs.-budget curve can be computed exactly (Sec. 3), we show that (i) without a gap, fixed-budget gains can be pure acceleration, vanishing as the budget grows (Fig. 1(b)); and (ii) with a gap, keeping only part of the data yields better model than full retention (Fig. 1(f)). Both predictions hold in LLM training (Fig. 2; Table 4).
Although mathematically simple, Proposition 1 is easily overlooked in practice, in two opposite directions. (i) Transient gains, e.g., from upweighting "hard" domains, can be mistaken for advancement: in our no-gap mixture experiments, early leads over Uniform vanish with training (Table 1, Fig. 2). (ii) The strength of uniform mixtures and random selection [3, 4] can be read as evidence against curation in general, yet the same methods yield large gains once a gap exists (Table 4).
W2 & Q2: Loss values versus optima. Δ does not compare optima. It tests the premise of Proposition 1, that the distribution the baseline actually trains on matches the target. If it does, the two losses are the same function of the parameters and agree at every checkpoint, so an appreciably nonzero Δ at any checkpoint, converged or not, shows that the premise fails. Since the training loss is measured on seen data, we calibrate against the small no-gap null (ζ>3). The two cases raised are handled as follows.
Beyond detecting a gap, Δ also tracks its size: the attainable gain grows with Δ in both solvable models (Fig. 4) and with the corrected Δ in a new target-interpolation experiment (Spearman ρ=1; new Table 5, Sec. 5.2; see also our response to Reviewer NTjY, W2, Target interpolation).
References:
[3] M. Chen et al. Aioli: A Unified Optimization Framework for Language Model Data Mixing. ICLR 2025.
[4] T. Xia et al. Rethinking Data Selection at Scale: Random Selection is Almost All You Need. EMNLP 2025.
(continued in the next comment)