acceptodds
Under review as a conference paper at ICLR 2027

Advancement or Acceleration? How Data Curation Improves LLM Training

Abstract

While data curation is widely credited with improving large language model (LLM) training, a basic question remains unresolved: does curation expand what a model can ultimately learn, or merely reach the same solution faster? In this paper, we formalize this distinction between *advancement* (improving the attainable performance) and *acceleration* (reaching the same performance with less data). Through a closed-form analysis and real-world LLM experiments, we show that curation can advance a model only when a train–validation gap is present. Without such a gap, it can only accelerate, and no allocation beats the uncurated baseline. This reframes advancement as a property of the curation *task* rather than the curation method. Under a common gap-centric view of data mixture and selection, we operationalize this condition as the -test (), an observable indicator computable prior to expensive curation. Across extensive mixture and selection experiments, the -test successfully predicts when curation helps and reveals that many reported gains are merely acceleration. Ultimately, our findings show that the true ceiling of data curation is dictated by validation construction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data curation, data mixture, data selection, advancement vs. acceleration, train–validation gap

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wgaVOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper distinguishes between advancement from data curation methods, which improve a model's attainable performance, and acceleration, which reduces the training budget required to reach a given performance. The authors develop an analytical treatment using a teacher–student perceptron model, introduce a train–target loss discrepancy (the Δ-test), and provide experiments on language-model data mixture and selection. Overall, the paper's findings are unsurprising, the proposed test is weak, and it offers no meaningful additional insight.

Critical strengths
  • The empirical evaluation includes several established data mixture and selection methods across different model scales.
  • Distinguishing persistent performance improvements from finite-budget improvements is a useful consideration when evaluating data curation methods.
Critical weaknesses

1 Limited originality and a largely conceptual contribution.

The distinction between improving the attainable optimum and accelerating convergence is elementary in optimisation. The main theoretical statement, Proposition 1, establishes that if the baseline training objective coincides with the target objective and already attains a global minimum, no alternative allocation can improve the target optimum. This follows essentially by definition from the assumptions. The work feels closer to a position paper advocating a particular interpretation of existing results than a substantive technical contribution.

2 The Δ-test is insufficiently justified.

The theoretical argument concerns whether population training and target objectives coincide. However, the Δ-test measures differences in loss values at a single checkpoint. Two objectives can have identical loss values at a checkpoint while having different gradients and optima, or different losses while sharing the same optimum. Furthermore, Eq. (1) takes absolute values before aggregation, potentially reporting a positive discrepancy even when aggregate training and target losses coincide. If I understand correctly, the model is optimised with respect to aggregate loss, thus the Δ-test and training use slightly different definitions.

3 Unnecessarily flowery writing and overstated significance.

The manuscript repeatedly presents familiar optimisation observations as fundamental discoveries. Examples include the claim that a “single principle governs” data mixture and selection (lines 54–61), that the framework “cleanly explains recent contradictory findings” (lines 65–70), and that Proposition 1 establishes a “fundamental algebraic consequence” (lines 225–235). The rhetorical framing is considerably stronger than the theoretical and empirical evidence warrants, and the writing borders on unscientific and unfit for the venue.

4 Questionable practical significance.

Acceleration is treated as secondary to advancement, despite reducing training compute being a central motivation for data curation. Furthermore, the Δ-test is evaluated at 50% of the training budget, making its characterisation as a lightweight early diagnostic questionable. Evaluations at exponentially spaced percentages of the training budget (e.g., 1/16th, 1/8th, 1/4th, 1/2) would be preferable to understand how close to the end of training one must be for the test to be useful.

Questions for authors
  1. What is the nontrivial theoretical contribution of Proposition 1 beyond its assumptions?
  2. How can loss differences at a single checkpoint diagnose differences between attainable optima?
  3. Why does Eq. (1) take absolute values before aggregation? I may have missed if there was a strong motivation.
4 comments
AuthorsOpenReview ↗

Response to Reviewer wgaV (1/2)

We thank the reviewer for the candid assessment. Several points led to concrete improvements, and we address each below.

W1 & Q1: Contribution and distinction from optimization convergence.

What the paper studies. We believe there might be a misunderstanding here. The paper concerns data scaling rather than optimization: it studies how curation gains change with the sample budget under controlled baseline–target alignment, not how fast an optimizer converges to a given optimum. In our closed-form analysis (Section 3), at every budget the model is trained to convergence, so none of these gains comes from faster optimization. The question is whether a gain observed at a finite budget persists as the budget grows (advancement) or vanishes (acceleration), and how this depends on the train–target gap.

Role of Proposition 1. We agree that Proposition 1 is mathematically simple, and we have toned down its presentation (now Observation 1). Nevertheless, our central question is whether a curation gain reflects advancement or acceleration, and Proposition 1 gives only the negative half of the answer: without a gap, curation cannot advance. The nontrivial results lie beyond this. In a teacher–student model whose full performance-vs.-budget curve can be computed exactly (Sec. 3), we show that (i) without a gap, fixed-budget gains can be pure acceleration, vanishing as the budget grows (Fig. 1(b)); and (ii) with a gap, keeping only part of the data yields better model than full retention (Fig. 1(f)). Both predictions hold in LLM training (Fig. 2; Table 4).

Although mathematically simple, Proposition 1 is easily overlooked in practice, in two opposite directions. (i) Transient gains, e.g., from upweighting "hard" domains, can be mistaken for advancement: in our no-gap mixture experiments, early leads over Uniform vanish with training (Table 1, Fig. 2). (ii) The strength of uniform mixtures and random selection [3, 4] can be read as evidence against curation in general, yet the same methods yield large gains once a gap exists (Table 4).

W2 & Q2-3: Interpretation and aggregation of the Δ\Delta diagnostic

W2 & Q2: Loss values versus optima. Δ\Delta does not compare optima. It tests the premise of Proposition 1, that the distribution the baseline actually trains on matches the target. If it does, the two losses are the same function of the parameters and agree at every checkpoint, so an appreciably nonzero Δ\Delta at any checkpoint, converged or not, shows that the premise fails. Since the training loss is measured on seen data, we calibrate against the small no-gap null (ζ>3\zeta>3). The two cases raised are handled as follows.

  • Missed gaps. Different distributions may yield equal losses at one checkpoint, but are unlikely to do so at all of them; the diagnosis is consistent from 1/201/20 to 1/21/2 of the budget (new Table 7; see W4). We agree that Δ≈0\Delta\approx0 is not a certificate of no gap. The original App. A.4 already notes that distinct distributions can yield similar losses. The revision now also states this in Section 4.
  • Spurious gaps. We thank the reviewer for this important point. A target that is intrinsically harder or easier to predict shifts its loss without any correctable mismatch. We now subtract the discrepancy of a reference model not trained on the baseline's distribution (new Eq. (2)), which removes this entropy difference. Please refer to our response to Reviewer NTjY, W2, point 2 for details.

Beyond detecting a gap, Δ\Delta also tracks its size: the attainable gain grows with Δ\Delta in both solvable models (Fig. 4) and with the corrected Δ\Delta in a new target-interpolation experiment (Spearman ρ=1\rho=1; new Table 5, Sec. 5.2; see also our response to Reviewer NTjY, W2, Target interpolation).

References:

[3] M. Chen et al. Aioli: A Unified Optimization Framework for Language Model Data Mixing. ICLR 2025.

[4] T. Xia et al. Rethinking Data Selection at Scale: Random Selection is Almost All You Need. EMNLP 2025.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer wgaV (2/2)

Q3: Absolute values before aggregation. This is a deliberate design to prevent cancellation. Training minimizes the aggregate loss for fixed domain weights, whereas curation asks whether changing these weights helps, which depends on each domain separately. If the baseline over-serves one domain (e.g., through repetition) and under-serves another, their signed discrepancies can cancel in the aggregate even though reweighting would help. When the target has no counterpart in the training pool (held-out domain in the exogenous scenario and selection), there is a single group and Δ\Delta reduces to the aggregate form, consistent with the training objective.

W3: Scope and strength of the claims

We have revised the passages the reviewer quoted and similar phrasing throughout. The three Propositions are now called Observations, the "fundamental algebraic consequence" phrasing is removed, and the "Δ\Delta-test" is renamed the Δ\Delta-diagnostic to reflect its role as an operational signal rather than a statistical test. All revised text is marked in blue.

W4: Practical value of acceleration and diagnostic cost.

Value of acceleration. We do not intend to present acceleration as secondary. To avoid this impression, we have revised the wording throughout. Our point is that the two effects should be measured differently. Advancement improves the endpoint and persists as the budget grows. Acceleration reduces the samples needed to reach a given performance, so its advantage in a fixed-budget comparison shrinks with the budget: in the no-gap mixture setting, mixtures that lead Uniform early see their advantage approach zero with training.

Early evaluation. Following the reviewer's suggestion, we evaluate Δ\Delta at 1/201/20, 1/101/10, 3/203/20, and 1/21/2 of the mixture budget (new Table 7, five seeds):

Budget1/201/103/201/2
No gap (null)0.026 ± 0.0030.003 ± 0.0010.013 ± 0.0010.076 ± 0.001
Endogenous0.041 ± 0.0030.102 ± 0.0020.158 ± 0.0020.649 ± 0.013
  ζ\zeta5155147537
Exogenous0.448 ± 0.0120.465 ± 0.0020.458 ± 0.0040.468 ± 0.002
  ζ\zeta137726452367

The exogenous gap is clear and stable from 1/201/20 of the budget. The endogenous gap grows as small domains start being repeated and is clearly separated from the null by 1/101/10 of the budget. Thus, Δ\Delta need not be measured near the end of training: early diagnostics suffice to predict whether curation can advance, though the exact point may depend on the task.

We hope these revisions and new results address the reviewer's concerns, and we would appreciate it if the reviewer would re-evaluate the paper in light of them.

Thank you for the rebuttal. I believe the framing of the work is now much improved, as is the strength of the submission. However, due to the overall purely diagnostic nature of the work and the still unsatisfactory quality of the writing, I will refrain from recommending acceptance. However, I do increase my score from 1 to 2.

Thank you for the re-reading, for raising the score, and for the candid assessment.

On "purely diagnostic": we see it as a decision that should be made before any search is launched, and getting it wrong is expensive in LLM training practice. It answers whether curation can raise the endpoint at all, or only accelerate. If the latter, whether the gains are worth pursuing depends on the budget: they may shrink as the budget grows, while the cost of running the search does not. Δ\Delta settles this from about 1/10 of a baseline run in our experiments.

We understand this is a different kind of contribution from a new method and respect the reviewer's weighting. We remain committed to further improving the writing. Any specific suggestions would be very welcome.

Reviewer_NTjYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper explores the importance and efficacy of data curation methods for both domain mixing and sample selection. The authors investigate whether such methods can genuinely improve the final attainable performance of an LLM (advancement) or reduce the training data/steps required to reach the same performance (acceleration). The authors (via a theoretical lens and empirical results) argue that to improve the model’s attainable performance, there needs to be a correctable mismatch between the baseline training data and the target evaluation distribution. If the training and evaluation distributions do not have any mismatch, data curation at best yields temporary acceleration, with any gains diminishing as training progresses. Finally, they introduce a diagnostic called the Δ\Delta-test (computed from an uncurated baseline run to compare the baseline model’s seen-data training loss with its target loss) that evaluates whether data curation can yield advancement (persistent gains) before committing to expensive curation.

Critical strengths

1- The paper tackles an important and practical question: separating final performance gains from training speedups. This question can help to explain conflicting findings in recent data curation literature.

2- The authors clearly unify data mixture and sample selection under one shared framework, showing how the mismatch between the training and target distributions governs when curation can improve final performance.

3- The experimental evaluation is thorough, testing a wide range of mixture and selection baselines under different setups.

Critical weaknesses

1- The main issue with this paper is that it over-formalizes basic and intuitive ML principles: The paper's main claim (that reweighting or pruning data cannot improve the final converged model when the training and test distributions already match) is a standard property of Empirical Risk Minimization.

2- There are some issues with using the Δ\Delta-test as a standalone metric, as it simply measures the discrepancy between baseline training loss (on seen data) and target validation loss:

  • There is no clear threshold for "appreciably nonzero".
  • Raw cross-entropy loss depends heavily on how predictable or repetitive a text dataset is; two datasets can have very different loss values without having a meaningful mismatch.
  • It relies on having a well-defined target set available upfront.
  • It still requires training an uncurated baseline model first.
Questions for authors

Please check out the critical weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer NTjY (1/2)

We sincerely appreciate your recognition of our technical contributions. We address your concerns below.

W1: Presentation and scope of Proposition 1

We agree that Proposition 1 follows directly from population risk minimization, and we have toned down its presentation (now Observation 1). It is, however, one component of a broader picture rather than the main claim.

Analytical scope: Proposition 1 covers only the unlimited-data, no gap case. The central question of the paper, whether curation improves the attainable optimum or only reaches the same performance faster, is about how curation gains depend on the sample budget. We address it with a sample-budget-dependent account of target generalization under curation, built from two complementary parts.

  • The teacher–student solvable model (Sec. 3, App. A.1–A.2) gives full scaling curves at finite budgets, with and without a gap.
  • Proposition 1 covers one corner of this picture, the large-budget limit without a gap. This follows from the general bi-level structure alone, so this conclusion does not rest on the teacher–student setting.

Practical relevance: simple in mathematics, but easily obscured in practice. Mathematically, the limiting statement (Proposition 1) is simple, but it is easily obscured in practice.

  • [3] and [4] find that uniform mixtures and random selection are hard to beat, or even "almost all you need". This is puzzling given the many works that report clear gains from mixture and selection. Proposition 1 helps explain the puzzle: in both studies, the target distribution effectively matches the training data, so no allocation can improve the endpoint. Making this boundary explicit tells practitioners when to stop refining the allocation and instead examine the train–target gap.
  • A second common practice is to upweight domains that appear hard and downweight easy ones, expecting stronger capabilities. Gains observed at a limited budget seem to confirm this heuristic. Yet when the target matches the training distribution, Proposition 1 implies that no such reweighting can improve the endpoint, and the teacher–student solvable model shows that the early gain vanishes as the budget grows (Fig. 1(b)).

W2 point 1-2: Threshold and predictive power of Δ\Delta

We agree with both points and have revised Section 4 accordingly.

Threshold. Thanks for the suggestion. We call Δ\Delta appreciably nonzero when ζ=(Δ−μnull)/σnull>3\zeta=(\Delta-\mu_{\mathrm{null}})/\sigma_{\mathrm{null}}>3, the conventional three-standard-deviation criterion. Here μnull\mu_{\mathrm{null}} and σnull\sigma_{\mathrm{null}} are the mean and standard deviation of Δ\Delta over 5 baseline seeds in the no-gap setting of the same experiment family, whose target is held-out data from the training distribution. Calibrating against this null is needed because a measured Δ\Delta is never exactly zero and its scale depends on the model, tokenizer, and data. The tables below report Δ\Delta for each experiment family.

FamilySettingΔ\Deltaζ\zeta
MixtureNo gap (null)0.076 (0.073)–
Endogenous0.649 (0.641)≈540
Exogenous0.468 (1.258)≈370
SelectionNo gap (null)0.016 (0.016)–
Shifted target0.158 (0.552)≈140

Values outside parentheses are after the entropy correction, as discussed in the next point, which removes the difference in intrinsic difficulty (entropy) between the training and target data. Values in parentheses are the uncorrected ones in the original paper. In both selection and mixture, every gap setting, measured at half the budget, exceeds the null with ζ≥140\zeta \geq 140, i.e., at least 140 standard deviations above the null mean, far above the threshold of 3.

References:

[3] M. Chen et al. Aioli: A Unified Optimization Framework for Language Model Data Mixing. ICLR 2025.

[4] T. Xia et al. Rethinking Data Selection at Scale: Random Selection is Almost All You Need. EMNLP 2025.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer NTjY (2/2)

Entropy confound and predictive power of Δ\Delta. We thank the reviewer for this important point. Raw cross-entropy also reflects how intrinsically predictable a dataset is. Hence, for an unpaired target, the raw discrepancy between the training and target losses mixes the gap with an entropy difference that no curation can change, and can overstate or understate the gap. We therefore add an entropy correction to Δ\Delta (new Eq. (2)):

Δ=∑j∣dj(θbaseline)−dj(θref)⏟new∣dj(θ)=w^jLtarj(θ)−∑k∈SjwkLtraink(θ),\Delta=\sum_{j}\Big| d_j(\theta_{\mathrm{baseline}}) \underbrace{- d_j(\theta_{\mathrm{ref}})}_{\text{new}} \Big| \qquad d_j(\theta)=\hat w_j \mathcal L _{\mathrm{tar}}^{j}(\theta)-\sum _{k\in S_j} w_k \mathcal L _{\mathrm{train}}^{k}(\theta),

where djd_j is the signed discrepancy between target component jj and its matched training domains SjS_j. The reference model θref\theta_{\mathrm{ref}} captures the intrinsic difficulty of the data but has not been trained on the baseline's training data, so subtracting its discrepancy removes the entropy difference. Without the reference term, this is the original Δ\Delta: Sj=jS_j=\\{j\\} gives the domain-wise form for mixture, and K=J=1K=J=1 gives the aggregated scalar form for selection. The reference model is only evaluated, never trained. In our experiments, the reference model used are the public pretrained Pythia-160M, which shares our tokenizer, for mixture, and the base TinyLlama model for selection.

FamilySettingCorrected Δ\Delta
MixtureNo gap0.076
Endogenous0.649
Exogenous0.468
SelectionNo gap0.016
Shifted target0.158

In the paired settings (no gap, endogenous), the reference discrepancy is close to zero and the correction leaves Δ\Delta essentially unchanged. The correction matters for unpaired targets, for the held-out CommonCrawl target in mixture (exogenous), about 63% of the raw 1.26 reflects CommonCrawl being harder to predict. The corrected 0.47 still lies far above the null (ζ≈370\zeta\approx370). In selection, the correction recovers a positive gap of 0.158 (ζ≈140\zeta\approx140).

Target interpolation (new). To test whether Δ\Delta tracks the size of the attainable gain, and not only its presence, we mix a fraction pp of the reasoning target (MetaMath–Mammoth) into the OpenWebMath target and rerun BLADE for each pp (new Table 5, Sec. 5.2).

pp00.20.40.60.81.0
Corrected Δ\Delta0.0160.0410.0690.1010.1290.158
ζ\zeta–255385113142
BLADE gain over Base-CT+0.3+9.6+10.4+12.0+12.3+14.1

As the target shifts, the corrected Δ\Delta increases monotonically and the BLADE gain increases with it (Spearman ρ=1\rho=1), showing that Δ\Delta tracks the attainable gain. Even at p=0.2p=0.2, Δ\Delta lies well above the no-gap null (ζ≈25\zeta\approx25).

W2 point 3-4: Target requirements and diagnostic cost

Target set requirements. Yes, our analysis requires a target set. Without one, there is no reference against which an improvement can be defined. The target is therefore required by the question itself, not by our method.

In practice, a target is actually more often present than one might expect, either explicitly or implicitly.

  • Target-aware methods use target data directly (e.g., DoGE, TANDEM, RHO-1, BLADE), and domain-adaptive continued pre-training (math, code, medicine) typically has a development set of the target domain.
  • Seemingly target-agnostic pipelines encode the target in a model trained on target data: CCNet filters CommonCrawl by the perplexity of a KenLM trained on Wikipedia, and GPT-3 and DCLM use quality classifiers whose positive examples are WebText/Wikipedia/books and OpenHermes/ELI5, respectively. Since these reference corpora are themselves the de facto target, they can serve directly as the target sample for Δ\Delta. When only the model is available, a classifier's odds could give importance weights that estimate the target loss on the training pool.

Requiring a target therefore usually adds no burden beyond what curation already assumes. Δ\Delta makes this target explicit and uses it only for loss evaluation.

Baseline run. Yes, Δ\Delta requires a baseline run, but this run can be very cheap: it can be both short and small.

  • Short. Δ\Delta already separates gap from no-gap settings from 1/201/20 of the training budget, clearly by 1/101/10, and remains stable afterwards (new Table 7, details in our response to Reviewer wgaV, W4).
  • Small. The diagnosis at 160M transfers to 1.1B (Table 3), so a small proxy model baseline run suffices.

Beyond this run, computing Δ\Delta involves only evaluation. The entropy correction adds one forward pass of a public model and no training. A short run of a small model is thus sufficient for this Δ\Delta-diagnostic.

We hope the clarifications and new results address the reviewer's concerns.

Reviewer_NTjYOpenReview ↗

Rebuttal

I wanted to thank the authors for their comprehensive response, they have answered all my questions and I will keep my score.

Thank you for the careful reading and for confirming that all questions are addressed. We are grateful for the detailed comments, which materially improved the revision.

Reviewer_mAHZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes that data curation may improve the best solution the model can attain (advancement), or it may merely reduce the data needed to reach the same solution (acceleration). To verify it, the authors introduce the ∆-test, a discrepancy between target and baseline training losses, as a lightweight diagnostic of advancement potential. Main experiments are performed on two tasks: SlimPajama data mixture and TinyLlama/OpenWebMath data selection, which compare no-gap, shifted-target, and limited data gap settings.

Critical strengths

The paper presents an interesting perspective by categorizing the effects of data curation into two types: advancement and acceleration. It further supports this perspective through both theoretical analysis and empirical evaluation, validating the proposed Δ-test on data mixture and data selection tasks. Overall, the paper is clearly written, and the experiments are comprehensive.

Critical weaknesses

The experiment currently focuses primarily on data selection and data mixture. Since practical data curation may also involve operations such as reweighting and reordering, could the authors consider extending the analysis to additional curation modules? Evaluating these settings would help further assess the validity and broader applicability of the paper’s central hypothesis.

Questions for authors

See Weaknesses part.

1 comment
AuthorsOpenReview ↗

Response to Reviewer mAHZ

We thank the reviewer for recognizing that our work is interesting, theory-backed, and comprehensively evaluated. We address your concerns below.

W1: Extending the analysis to re-weighting and re-ordering

Thanks for this helpful suggestion. Our analysis concerns different aggregate training distributions and thus covers reweighting. Reordering concerns different optimization trajectories under the same aggregate training distribution, beyond our research scope.

Reweighting. Our bi-level formulation (Eq. (48) and Eq. (53)) is itself a reweighting formulation. Mixture optimizes domain weights, and selection is analyzed over continuous per-example weights w∈[0,1]P\boldsymbol{w} \in [0,1]^P, with hard selection as the special case w∈0,1P\boldsymbol{w} \in \\{0,1\\}^P. The no-gap result (Observation 3 (Proposition 3 in the original version)) holds for any nonnegative weights, so continuous loss reweighting falls within our analysis: without a correctable gap, no reweighting improves the attainable endpoint, and with one, reweighting can advance only insofar as it corrects the gap. For mixture, sampling-based and loss-weighted implementations further induce the same limiting objective (App. A.3, Eq. (47)). We now state this explicitly after Observation 3.

Reordering. Analytically, reordering marks the scope of our analysis. Both our solvable models and bi-level optimization formulation (Eq. (48) and Eq. (53)) abstract away training dynamics and curriculum order, as discussed in Limitation (2) (App. A.7). Whether reordering can guide optimization toward better solutions [1] is therefore beyond our analysis.

Empirically, our selection experiments are connected to re-ordering. RHO-1 and BLADE involve a dynamic element: within each large batch, they keep the sub-batch most useful to the current model, so the data are actually partly re-ordered. Without a gap, neither consistently outperforms Random or Base-CT (Table 4, upper). With a gap, both advance substantially (Table 4, bottom). Within our experiments, this dynamic element does not change the gap-centric pattern. More broadly, the advancement/acceleration distinction applies to curricula as an evaluation principle: a gain should be checked for persistence across budgets rather than read from a single budget. This is consistent with [2], who find that curricula help mainly under limited training budgets or noisy data. Nevertheless, these results are preliminary. A complete analysis of reordering is beyond the scope of this paper, and we leave it to future work.

We hope these clarifications address the reviewer's concerns, and we are happy to answer any further questions.

References:

[1] Y. Bengio et al. Curriculum Learning. ICML 2009.

[2] X. Wu et al. When Do Curricula Work? ICLR 2021.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers for their thoughtful feedback. We are encouraged that the reviewers found that the paper:

  • tackles "an important and practical question" that "can help to explain conflicting findings" (NTjY, also mAHZ, wgaV);
  • "clearly unifies data mixture and sample selection under one shared framework" (NTjY);
  • offers a "thorough" and "comprehensive" evaluation (NTjY, mAHZ, wgaV).

The main concerns are (1) the presentation and role of Proposition 1 (NTjY, wgaV), and (2) the reliability of Δ\Delta as a diagnostic (NTjY, wgaV). We address each below. (We also respond individually to Reviewer mAHZ's suggestion on reweighting and reordering.)

1. Presentation and role of Proposition 1.

We agree that Proposition 1 is mathematically simple and have toned down its presentation (now Observation 1). Its role is clearest in the context of the question the paper addresses: when a curated dataset outperforms the baseline at a fixed budget, is the gain acceleration, which vanishes as the budget grows, or advancement, which persists?

The paper answers this question in three steps. (i) Proposition 1 covers the limiting case: without a train–target gap, no curation can improve the endpoint. (ii) Real training runs at finite budgets, so we analyze a simplified teacher–student model in which the full performance-vs.-budget curve can be computed exactly under any curation (Sec. 3). It shows that without a gap, curation can still yield sizable gains at small budgets, but these gains shrink as the budget grows. With a gap, including one created only by repetition of small domains, correcting the gap yields gains that persist. (iii) Δ\Delta estimates the gap from a baseline run, before any curation is attempted. Proposition 1 thus only fixes the limiting case in the no-gap regime. Distinguishing acceleration from advancement at practical budgets relies on (ii) and (iii).

Although simple, the result of Proposition 1 is easily overlooked in practice. Upweighting harder domains, for example, can improve performance at a limited budget without improving the endpoint, and such gains can be mistaken for advancement. Conversely, the strong performance of uniform mixtures and random selection in the no-gap scenario can be read as evidence against curation in general. Our analysis reconciles the two observations: the former is acceleration, and the latter reflects the absence of a gap.

For details, see our responses to Reviewer NTjY (W1) and Reviewer wgaV (W1 & Q1).

2. Reliability of Δ\Delta as a diagnostic.

Δ\Delta measures the train–target gap, and thus indicates whether curation can advance or only accelerate. We thank the reviewers for their detailed comments, which have directly improved it.

  • A threshold for detecting a gap. The original paper did not specify when Δ\Delta counts as nonzero. We now calibrate it against a no-gap null: the spread of Δ\Delta across baseline seeds in the matching no-gap setting. A gap is declared only when Δ\Delta exceeds the null mean by more than three standard deviations. All our original gap settings, measured at half the budget, exceed it by at least 140 standard deviations.
  • Correcting for intrinsic entropy. Δ\Delta compares losses on two different datasets, so a target that is intrinsically harder or easier to predict can shift Δ\Delta even when there is no gap that curation could correct. We now subtract the same discrepancy measured by a reference model not trained on the baseline's distribution (new Eq. (2)), so that the effect of intrinsic entropy is removed. After this correction, the Δ\Delta in every gap setting still lies far above the null.
  • Early diagnosis. Δ\Delta was originally measured at half the training budget, which may limit its use as an advance diagnostic. New evaluations at earlier checkpoints show that, in our settings, about 1/101/10 of the budget suffices, and the diagnosis does not change afterwards (new Table 7).

To further examine whether Δ\Delta predicts how much curation can gain, we construct a setting in which the gap varies continuously: the selection target is a mixture whose fraction pp of reasoning data increases from 00 to 11, moving the target gradually away from the training distribution. As pp grows, Δ\Delta and the gain of target-aware selection over the baseline increase together (Spearman ρ=1\rho=1; new Table 5, Sec. 5.2), so a larger Δ\Delta indicates more room for advancement. Δ\Delta remains a one-directional signal: a near-zero Δ\Delta does not prove the absence of a gap, as we now state in Sec. 4. For details, see our responses to Reviewer NTjY (W2) and Reviewer wgaV (W2, W4, Q2–3).

All revisions are marked in blue in the updated version. We hope these revisions address the concerns, and we are happy to answer further questions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.