- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper discusses some scenarios where changing the data mixture and training data order improves training effectiveness, and produce some analysis outlining why this happens.
Critical strengths
The paper covers some scenarios where the order of training data can or cannot bring benefits for neural network training, under the simple quadratic learning set up with gaussian noise.
Critical weaknesses
The paper has some serious presentation issues that make it difficult to decipher the meaning of the sentences. It uses arcane phrases throughout the paper; some phrases are mentioned once and not used again. For example:
- "This comparison is stronger than matching the curriculum’s total source exposure or comparing an order with its reversal." (Source exposure isn;t mentioned anywhere, what does it mean?)
- "Reoptimized perturbed curricula are compared with all-weight static brackets, and finite-budget gains are plotted against a global ceiling that includes the energy bound." (What is energy bound?)
- Entire section 4 where a few theorems are presented without ANY context.
There are many examples of these in the entire paper.
Questions for authors
I think the paper covers some scenarios where the order of training data can or cannot bring benefits for neural network training, under the simple quadratic learning set up with gaussian noise. What concrete takeaway can we get from this piece of work?
3 comments
Sorry, I still find the passage hard to follow. And with the incoherence in writing, it is difficult to pinpoint the contribution and novelty of the paper. I would like to keep my score.
Response to Reviewer grwo (follow-up)
Thank you for your reply. We are sorry the writing is still hard to follow, so here is the contribution in plain terms.
Contribution. We ask when changing the data mixture during training beats the best fixed mixture at the same budget. For two equally strong data sources that share the same best solution in a quadratic learning problem, Theorem 2 gives an exact answer: balanced mixing is optimal over all schedules exactly when balanced training already ends with at least as much error in the slowly learned direction as in the fast one. Otherwise, a two-phase schedule beats every fixed mixture.
Novelty. Related control-based work gives necessary conditions that an optimal schedule must satisfy (Section 2). We instead give an exact comparison between all schedules and the best fixed mixture.
Section 4 opens with "The results in words" (page 4), which is the quickest way into the results.
Response to Reviewer grwo: presentation and takeaways
Thank you for the concrete examples. You are right that the submitted text used terms before defining them and gave Section 4 too little context. We rewrote Sections 1 and 4 of the revised PDF and did the same across the main text, replacing once-used phrases such as "posterior radius" with plain wording and defining technical terms where they first appear or in a new glossary (Appendix O, p. 35).
"Source exposure." Exposure is the total training work spent on a source, defined in words in Section 1 (p. 1) and by the formula P=∫0Tp(t)dt in Section 3 (p. 3), where T is the training budget and p(t) the share of work on source A at time t. In the example of Section 1, a schedule trains on source B for the first quarter of the budget and then on A. Its reverse order and the fixed mixture with the same exposure end with 1.50 and 1.25 times its loss. Yet it ends with 1.19 times the loss of the balanced (50/50) mixture, which no schedule beats there, so beating both controls does not show that order helps.
"Energy bound." The energy bound says that no schedule can shrink the error faster than the largest curvature of the loss (its largest Hessian eigenvalue) allows. Section 4 now defines it before Eq. (9), where it caps every schedule's gain (p. 5). Section 1 now also explains the other two quoted terms (p. 2): the "static bracket" gives upper and lower bounds on the best fixed mixture's loss, and the "ceiling" bounds the gain of every schedule.
Section 4. Section 4 now opens with "The results in words" (p. 4). Both sources reduce the error quickly along one direction and slowly along the perpendicular one, and every schedule obeys two floors: one on the product of the remaining fast and slow errors, and one on the slow error alone. Balanced mixing attains both. The loss is the sum of the two errors, and a sum with a fixed product is smallest when its terms are equal, so a schedule gains by equalizing them. If balanced training already ends with at least as much slow as fast error, equalizing would push the slow error below its floor, so no schedule beats balance; this is exactly the condition of Theorem 2. The new Figure 1(a) maps where order helps (p. 5).
Concrete takeaways. Section 7 now ends with takeaways (p. 10). In these quadratic models, a training order fixed in advance saves at most a constant factor in compute and never improves a power-law scaling exponent, and we make no claim about neural networks.
Please tell us if any passage is still hard to follow.