acceptodds
Under review as a conference paper at ICLR 2027

When Does Training Order Buy Compute? Exact Boundaries and Robust Gain Guarantees

Abstract

When does changing data weights outperform the best fixed mixture, rather than merely a poorly chosen baseline? We give an exact answer for shared-optimum quadratic learning. For two mirror sources at angle , with orthogonal initial-error mass , balanced mixing is optimal over all measurable schedules at budget exactly when . Below the boundary, two mixed phases are enough to improve performance; sufficiently close to it they attain the global gain ceiling exactly. We then distinguish exactness from robustness. Aligned target anisotropy shifts the exact threshold, but arbitrarily small cross moments or unequal source strengths can destroy exact static optimality. A completed-square bound nevertheless limits the reoptimized gain quadratically in the cross moment, while a uniform perturbation bound covers general changes. Finite-budget plots show where candidate schedules approach the all-schedule ceiling and where a gap remains. Exact Gaussian stochastic gradient descent (SGD) experiments are indexed by distance from the boundary and control source composition through stratified batches and rapid alternation. A uniform SGD-to-flow bound guarantees convergence, not the observed near-boundary gain rate. The results identify both the value and the fragility of training order, without claiming a universal neural-network curriculum.

Primary area:
learning theory
Keywords:
curriculum learning, training order, optimal data mixtures, quadratic learning, stochastic gradient descent

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_grwoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper discusses some scenarios where changing the data mixture and training data order improves training effectiveness, and produce some analysis outlining why this happens.

Critical strengths

The paper covers some scenarios where the order of training data can or cannot bring benefits for neural network training, under the simple quadratic learning set up with gaussian noise.

Critical weaknesses

The paper has some serious presentation issues that make it difficult to decipher the meaning of the sentences. It uses arcane phrases throughout the paper; some phrases are mentioned once and not used again. For example:

  1. "This comparison is stronger than matching the curriculum’s total source exposure or comparing an order with its reversal." (Source exposure isn;t mentioned anywhere, what does it mean?)
  2. "Reoptimized perturbed curricula are compared with all-weight static brackets, and finite-budget gains are plotted against a global ceiling that includes the energy bound." (What is energy bound?)
  3. Entire section 4 where a few theorems are presented without ANY context.

There are many examples of these in the entire paper.

Questions for authors

I think the paper covers some scenarios where the order of training data can or cannot bring benefits for neural network training, under the simple quadratic learning set up with gaussian noise. What concrete takeaway can we get from this piece of work?

3 comments
AuthorsOpenReview ↗

Response to Reviewer grwo: presentation and takeaways

Thank you for the concrete examples. You are right that the submitted text used terms before defining them and gave Section 4 too little context. We rewrote Sections 1 and 4 of the revised PDF and did the same across the main text, replacing once-used phrases such as "posterior radius" with plain wording and defining technical terms where they first appear or in a new glossary (Appendix O, p. 35).

"Source exposure." Exposure is the total training work spent on a source, defined in words in Section 1 (p. 1) and by the formula P=∫0Tp(t)dtP=\int_0^T p(t)dt in Section 3 (p. 3), where TT is the training budget and p(t)p(t) the share of work on source A at time tt. In the example of Section 1, a schedule trains on source B for the first quarter of the budget and then on A. Its reverse order and the fixed mixture with the same exposure end with 1.50 and 1.25 times its loss. Yet it ends with 1.19 times the loss of the balanced (50/50) mixture, which no schedule beats there, so beating both controls does not show that order helps.

"Energy bound." The energy bound says that no schedule can shrink the error faster than the largest curvature of the loss (its largest Hessian eigenvalue) allows. Section 4 now defines it before Eq. (9), where it caps every schedule's gain (p. 5). Section 1 now also explains the other two quoted terms (p. 2): the "static bracket" gives upper and lower bounds on the best fixed mixture's loss, and the "ceiling" bounds the gain of every schedule.

Section 4. Section 4 now opens with "The results in words" (p. 4). Both sources reduce the error quickly along one direction and slowly along the perpendicular one, and every schedule obeys two floors: one on the product of the remaining fast and slow errors, and one on the slow error alone. Balanced mixing attains both. The loss is the sum of the two errors, and a sum with a fixed product is smallest when its terms are equal, so a schedule gains by equalizing them. If balanced training already ends with at least as much slow as fast error, equalizing would push the slow error below its floor, so no schedule beats balance; this is exactly the condition of Theorem 2. The new Figure 1(a) maps where order helps (p. 5).

Concrete takeaways. Section 7 now ends with takeaways (p. 10). In these quadratic models, a training order fixed in advance saves at most a constant factor in compute and never improves a power-law scaling exponent, and we make no claim about neural networks.

  • A curriculum should be compared with the best fixed mixture at the same budget, not only with exposure-matched or reversed-order controls.
  • With two equally strong sources and an initial error aligned with their bisector, order stops helping under gradient flow beyond a finite budget (the boundary of Theorem 2) whenever some initial error lies in the slowly learned direction. Near this boundary the benefit is second order: about 0.13% when the initial slow-direction error is 10% below its boundary value.
  • In two dimensions, if one source is stronger and the initial error is aligned with the bisector but has unequal masses along the fast and slow directions, a two-phase schedule with the same exposure beats the best fixed mixture whenever that mixture uses both sources (Corollary 25). The loss gain can be large, but the compute gain is modest: at the boundary of Theorem 2, the best fixed mixture matches the best schedule found with a median of only 6% more training when one source is twice as strong, and 15% when it is four times as strong (Appendix P.4).
  • Loss ratios are not compute savings. At 45° with no initial slow error and budget 4, the best fixed mixture's loss is 1.89 times that of a simple rule that trains on one source and then the other (Section 3), yet it needs only 1.17 times the rule's work to reach a small target loss.
  • Feedback from observed per-source losses is a different resource from order and can help where no schedule fixed in advance can (Appendix R).

Please tell us if any passage is still hard to follow.

Sorry, I still find the passage hard to follow. And with the incoherence in writing, it is difficult to pinpoint the contribution and novelty of the paper. I would like to keep my score.

AuthorsOpenReview ↗

Response to Reviewer grwo (follow-up)

Thank you for your reply. We are sorry the writing is still hard to follow, so here is the contribution in plain terms.

Contribution. We ask when changing the data mixture during training beats the best fixed mixture at the same budget. For two equally strong data sources that share the same best solution in a quadratic learning problem, Theorem 2 gives an exact answer: balanced mixing is optimal over all schedules exactly when balanced training already ends with at least as much error in the slowly learned direction as in the fast one. Otherwise, a two-phase schedule beats every fixed mixture.

Novelty. Related control-based work gives necessary conditions that an optimal schedule must satisfy (Section 2). We instead give an exact comparison between all schedules and the best fixed mixture.

Section 4 opens with "The results in words" (page 4), which is the quickest way into the results.

Reviewer_MhoKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper demonstrates when fixed mixture schedule dominates in a toy 2D setting with common minimizer, equal learning generator trace (of G_A and G_B) and an error 2nd moment that's diagonal in the basis where G_A and G_B share diagonal terms.

Critical strengths
  • This is a well motivated and timely problem, as data mixture optimization is critical in applications like LLM pretraining
  • The claim that fixed mixture schedule is optimal is interesting, tho in a toy setting.
Critical weaknesses
  • W1: It appears that the main claim requires S to be diagonal in the basis where G_A and G_B share diagonal terms, and requires G_A and G_B share the magnitude of the diagonal terms (as shown in proposition 9). This seems commonly not the case in empirical study, where certain data sources (e.g. wikipedia) can be easier to learn than others (e.g. website), which motivates scheduling work like DoReMi or adaptive work like Online Data Mixing [1]
  • There are quite a few notation/writing issues (e.g. reuse of BB, tt, ⋅\cdot vs. ′' for derivatives, or terms introduced before their definitions). The main paper as written is rather hard to follow.

[1] Albalak et al., Efficient Online Data Mixing For Language Model Pre-Training, https://arxiv.org/abs/2312.02406

Questions for authors
  • Is W1's characterization of the main assumptions required incorrect?
  • How would the main claim change or scale for n-dimensional optimization?
  • Is it important that schedule is fixed ahead of time? Does this result apply to adaptive schedules where errors observed at each steps impacts the scheduling choices.
4 comments
AuthorsOpenReview ↗

Response to Reviewer MhoK (1/2): assumptions, unequal sources, notation

Thank you for the careful reading.

Q1 and W1: the assumptions. Your characterization is correct for the exact if-and-only-if boundary, with two refinements (the first two points below). We also show that in two dimensions, when the best fixed mixture uses both sources, exact optimality needs aligned SS except at isolated values of the slow-direction mass ε\varepsilon (Section 5, pp. 6 and 7, and Appendix P, p. 36).

  • In two dimensions, GAG_A and GBG_B share a diagonal in some orthonormal basis exactly when their traces are equal (Lemma 22).
  • Mirror symmetry is not needed. It suffices that some mixture strictly between the two sources is diagonal in that basis, that SS is diagonal there, and that the target matrix QQ is the identity (Proposition 23).
  • In any dimension, exact optimality generically forces a compatibility condition on SS. Suppose the best fixed mixture uses both sources and the eigenvalue differences of its generator G⋆G_\star are nonzero and pairwise distinct, which is automatic in two dimensions when its eigenvalues differ. In an eigenbasis of G⋆G_\star, fixed mixing at budget TT can then be optimal over all schedules only if (GA−GB)ij(Se−TG⋆Q)ji=0(G_A-G_B)_{ij}(Se^{-TG_\star}Q)_{ji}=0 for all i≠ji\ne j (Proposition 24). Otherwise a two-phase schedule with the same exposure (total work on each source) is strictly better.
  • In two dimensions this has two consequences (Corollary 25). With equal traces and SS rotated away from the shared-diagonal basis, a mixture that uses both sources is exactly optimal for at most one value of ε\varepsilon. With unequal strengths, SS diagonal in the bisector basis and ε≠1\varepsilon\ne1, a two-phase schedule with the same exposure beats the best fixed mixture at every budget whenever that mixture uses both sources. This strengthens the unequal-strength part of Proposition 9.

Two caveats apply: at ε=1\varepsilon=1 no two-phase schedule helps, and Proposition 24 and Corollary 25 say nothing when the best fixed mixture uses a single source.

Unequal strengths. Your intuition about easier and harder sources is right, but the compute gains are modest. We computed 288 exact cases at three angles and three budgets (Appendix P.4, p. 39). When one source is twice as strong and ε\varepsilon sits on the equal-strength boundary (r=1r=1), the best fixed mixture's loss exceeds that of the best schedule found by a median of 14.9%, whereas with equal strengths this excess is exactly zero. Yet the best fixed mixture reaches that schedule's loss with a median of only 1.06 times the budget. In 140 of 144 unequal-strength cases, the best schedule found shifts the stronger source toward the end.

DoReMi. DoReMi outputs a fixed mixture, so under the conditions of Corollary 25 some schedule fixed in advance beats even the best fixed mixture. However, our model assumes one shared optimum and cannot represent domains whose best predictors differ, the case that DoReMi's minimax objective addresses; Section 7 lists this limitation.

W2: notation. We fixed every clash you named. The budget is now TT everywhere, and AA and BB only label sources. The squared norm ∥Φv∥2\Vert\Phi v\Vert^2, which the submission called tt, is now ℓ\ell, so tt only means time. Dots denote time derivatives, primes denote derivatives in the mixture weight pp (or another scalar named where used), and ∂γ\partial_\gamma denotes a derivative in a perturbation amplitude γ\gamma (Appendix I, p. 29). The appendices now call this amplitude γ\gamma, as Section 4 does, so η\eta only denotes the SGD step size. We also removed further clashes, such as mm, which named both an eigenvalue bound (now μ\mu) and the batch size. Terms the submission used before defining them, such as the excess gain (gain minus one), are now defined at or before first use or in the glossary and notation table of Appendix O (p. 35).

AuthorsOpenReview ↗

Response to Reviewer MhoK (2/2): n dimensions and adaptive schedules

Q2: n dimensions. The boundary extends exactly to problems built from identical two-dimensional planes, and partly to planes that differ (Appendix Q, p. 40). Propositions 10 and 15 (the uniform perturbation and SGD-to-flow bounds) and the spectral result of Section 7 were already dimension-free; the last shows that scheduling cannot change the loss exponent when curvatures and initial errors decay as power laws.

  • Identical planes that share one schedule reduce exactly to the two-dimensional problem with summed masses (Proposition 27), so Theorem 2 holds in any such dimension with ε=∑jsv,j/∑jsu,j\varepsilon=\sum_j s_{v,j}/\sum_j s_{u,j}, where su,js_{u,j} and sv,js_{v,j} are the initial-error masses of plane jj along its fast and slow directions.
  • For planes that differ but share a decoupling mixture (diagonal in every plane's basis), that mixture is optimal if every plane ends training under it with at least as much slow as fast error, and it can be improved if no plane does (Proposition 28). Other cases depend on all planes jointly.
  • We computed exact cases at ε=0.01\varepsilon=0.01 with up to 32 dimensions (Appendix Q, p. 41). The largest excess gain found (the best fixed mixture's loss over the schedule's, minus one) falls from 8.96% for one plane to 1.37% for 16 planes with equal angles and curvatures scaled by 1/j1/j that share one schedule, but only to 4.73% with a separate schedule for each plane. These gains come from a search, so they are lower bounds.

For general nn-dimensional sources we give only Propositions 10 and 24 and the energy and determinant lower bounds; an exact boundary there remains open.

Q3: adaptive schedules. Yes, it matters: Theorem 2 covers only schedules fixed in advance and does not bound adaptive ones (Appendix R, p. 42, and Section 7). In our model the slow component of the initial error has a random coefficient ZZ with mean 0 and variance 1, and the initial per-source losses reveal ZZ (Proposition 29). With exact losses, the best feedback policy reads them once and then commits to a schedule tailored to that error. We tested 36 cases in which no schedule fixed in advance beats balanced mixing (Appendix R, p. 42).

  • The best fixed mixture's risk is 1.38 to 51.2 times that of this read-once policy for Z=±1Z=\pm1, and 1.26 to 17.2 times for Gaussian ZZ.
  • For a greedy rule that trains on the source with the larger current loss, this ratio is 1.38 to 16.0 for Z=±1Z=\pm1.
  • Choosing a constant mixture after reading the losses already gives a ratio of 1.38 to 26.2 for Z=±1Z=\pm1. Adding order on top of this mixture multiplies this ratio by at most 1.18 at budgets up to 2, but by 1.38 to 2.97 at budget 4.

The greedy rule is a noise-free, short-sighted analogue of the per-domain loss reward in Online Data Mixing (Albalak et al., 2023, now cited), but we do not model noisy rewards or exploration. Stage-wise curricula, RegMix-type mixtures and DoReMi's averaged weights are fixed in advance, so in our model they lie in the class that Theorem 2 covers. Loss-reactive methods such as Online Data Mixing do not.

I thank the authors for further clarifications.

In the short time remaining, I cannot claim to understand the clarifications fully. That said, it remains unclear to me how the particular setting in which the main results hold could be a common case in practice. Considering that, I intend to keep my original score.

AuthorsOpenReview ↗

Response to Reviewer MhoK (follow-up)

Thank you for engaging with our clarifications in the short time available.

You are right that the exact setting in which data order gives no benefit is special, and the paper does not claim that it is common. It serves as an exactly solvable reference point, and Proposition 24 and Corollary 25 show that it is fragile: when the best fixed mixture uses both sources, a misaligned initial error or unequal source strengths almost always make some schedule strictly better.

The conclusions meant for practice concern how much the order of the data can help, and they hold beyond this setting. For a small misalignment, the gain of any schedule over the best fixed mixture is at most quadratic in the size of the misalignment (Section 5). Starting from the no-benefit setting with equal source strengths, if one source is twice as strong, the best fixed mixture reaches the best schedule's loss with a median of only 1.06 times the budget (Appendix P.4). In power-law block models, no schedule fixed in advance changes the scaling exponent of the loss (Section 7).

We therefore recommend comparing any curriculum with the best fixed mixture at the same budget.

Reviewer_bZVVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper theoretically analyses the impact of dataset mixture schedules in training based on a simple quadratic model.

Critical strengths

Curriculum learning and dataset mixture schedules are relevant areas of study, especially in the context of modern LLM training. Developing a theoretical model can be a useful approach to help understand when such schedules are usefull.

Critical weaknesses

The writing in this article is extremely dense, missing context and full of technical terms that were never introduced, making it impossible for me to understand. It seems extremely likely to me that large parts of this paper are LLM-generated without proper review.

Questions for authors
1 comment
AuthorsOpenReview ↗

Response to Reviewer bZVV: presentation and AI use

Thank you for the review. You are right that the submitted text was dense and used technical terms before introducing them, and the other two reviewers made the same point. The revised PDF rewrites the presentation without changing any of the original results.

What changed.

  • The abstract and introduction (pp. 1 and 2) now state the question and the main result in plain words, with a worked example.
  • Terms are defined where they first appear, and Appendix O (pp. 34 and 35) adds a glossary and a notation table.
  • Section 4 opens with "The results in words" (p. 4), which explains the main theorem before it is stated, and Sections 5 and 6 open by saying what they test.
  • Figure and table captions now say what each plot or table shows, and Section 7 ends with numbered takeaways (p. 10).
  • We removed notation clashes, such as one letter used for both the training budget and a data source.

What the paper shows. In a quadratic model where all data sources share one optimum, we determine exactly when changing the data mixture during training beats the best fixed mixture. With two equally strong sources and an initial error aligned with their bisector, no schedule beats the best fixed mixture once the training budget passes an explicit threshold, provided some initial error lies in the slowly learned direction. Just below that threshold the benefit is tiny: about 0.13% when the initial slow-direction error is 10% below its boundary value. With unequal sources the loss gain can be large, but the compute saving is modest. At the equal-strength boundary, the best fixed mixture matches the best schedule found with a median of only 6% more training when one source is twice as strong.

On AI use. As the paper's AI use statement says, we used generative AI tools to draft sections of the paper, to polish the writing and to edit code, and we reviewed all AI-assisted work and take responsibility for it. The results can be checked independently of the prose. The main results have full proofs in the appendices, and a table at the start of the appendices (p. 14) lists where each one is proved. The SGD experiments compute expected risks exactly rather than estimating them from sampled runs (Section 6).

Please tell us if any passage is still hard to follow.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of the revision

We thank the three reviewers. All three found the paper hard to read, and we agree. The revised PDF rewrites the abstract and introduction in plain terms, defines terms at first use or in a new glossary, explains each main result in words, and ends Section 7 with numbered takeaways. New Appendices P to S answer the reviewers' questions.

The main text still fits in 10 pages, and all original numbering is unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.