- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies curriculum effects on grokking by separating example order, exposure, and support. Across six algorithmic tasks at two model scales, global difficulty sorting delays or prevents grokking within the training budget despite matched exposure, while sorting within full-data epochs largely removes the effect on the two tasks tested. The results suggest sustained restriction to an unrepresentative data subset as the main explanation.
Critical strengths
- The research question and gap are clear, and the paper's structure is logical.
Critical weaknesses
- The sampling schemes are specified, but support and confinement duration need a clearer formal definition, including the time window over which support is measured. The growing-pool schedules use linear pacing, leaving it unclear whether the conclusions hold under non-linear pacing or discrete schedules that expand the pool in stages [1].
- The experiments use six tasks at two scales, and the prefix-matched control covers only two of them, which limits the evidence that fine-grained order has little effect on grokking. Each task uses a single difficulty metric, so sensitivity to alternative metrics on the same task remains unclear. Appendix J.4 tests different difficulty signals for QA, but those experiments are not exposure-controlled.
[1]. Wang, X., Chen, Y. and Zhu, W., 2022. A survey on curriculum learning, IEEE transactions on pattern analysis and machine intelligence.
Questions for authors
- How are support and confinement duration defined over time, and is there evidence that the conclusions hold under non-linear pacing or discrete schedules that expand the pool in stages?
- For the same task, do the results in Tab 2 and 3 hold when a different difficulty metric changes the example order?
6 comments
Thank you again for your review. As the discussion period closes on Nov 18, we wanted to check whether the four additions you asked for address your concerns: the windowed definitions of support and confinement duration, the non-linear and staged pacing runs, the matched-prefix control on all six tasks, and the alternative difficulty metrics. We would be glad to answer any further questions. If the revision addresses your concerns, we would be grateful if you would consider updating your score.
Thank you for your response and additional work, which address several of my concerns through clearer support definitions and expanded pacing, control, and difficulty-metric experiments. However, the revision does not yet isolate support from training history and forgetting, establish a grokking-specific explanation, and resolve the remaining statistical and theoretical issues. So, I keep my score.
Thank you. All four requests are now in the paper.
Definitions of support and confinement duration over time, including the window (§2, Box 1).
Non-linear and staged pacing (§5.8, Appendix I). We ran square-root, quadratic, exponential and staged pacing on mod-97 addition, GCD and S5, three seeds each.
Predictions made in advance. Before these runs we predicted the mod-97 transition times from the linear-pacing runs, at threshold 0.7. The assumption was that the pool fraction at the transition does not depend on the pacing law.
Staged schedules cross shortly after the first stage whose pool reaches that fraction, in 6 of 8 schedules.
Pool fraction at 0.7. It varies little across laws on mod-97 addition and GCD. Appendix I also gives it at 0.9.
Where the account fails.
So a growing pool's history adds to its size. We report this as a limitation of the simple support account (Limitation 8).
The matched-prefix control beyond two tasks (§5.2, Table 2). All six tasks, 5 seeds, evaluation every 25 steps.
Alternative difficulty metrics on the same task, re-running Tables 1 and 2 (§5.8). We repeat the global and the epoch sort with one to three alternative structured keys per task, plus a key that permutes the difficulty values across examples. For S5 these include input-side keys.
Do these address your concerns? We would be glad to add further variants.