- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how training-data ordering affects interatomic potentials while holding the training pool, sample exposure, initialization and within-model optimization settings fixed. It compares five orderings across four architectures on OC20, with a larger-budget extension. The main finding is that several intuitive curricula worsen force prediction, while occasional gains depend on the model, budget and prediction objective.
This is a useful controlled empirical study. However, its contribution currently remains a regime-specific observation rather than a sufficiently general explanation of when ordering helps or hurts. Stronger mechanistic analysis and broader training-regime validation would substantially improve the paper.
Critical strengths
The matched-support design addresses a real ambiguity: curriculum gains can otherwise reflect changes in data selection or repeated exposure. Experiments cover four architectures and four ID/OOD validation splits. Complete-split evaluation strengthens the evidence beyond the initial validation subsets. Reporting energy and force separately exposes regressions hidden by a single metric. The negative results provide a useful caution against assuming that intuitive curricula improve training.
Critical weaknesses
The practical scope is narrow. Every example is presented once, starting from random initialization. The experiments establish ordering effects in this regime, but do not show whether they persist under multi-epoch training, convergence-oriented budgets or fine-tuning. A common shuffled continuation would help distinguish persistent effects from transient endpoint bias.
The tested difficulty measure limits the conclusion. Reference-force RMS measures label amplitude rather than learning difficulty. Failure of ascending and descending force-magnitude orders does not establish the limitations of teacher-error, uncertainty-based or adaptive curricula.
Mechanistic insight remains limited. Reordering examples also changes minibatch composition and gradient statistics. The study does not separate these effects from inter-batch order or explain the observed energy–force responses. Moreover, three architectures predict forces directly, limiting interpretation in terms of energy-value versus energy-gradient accuracy.
The empirical contribution needs stronger generalization. The study demonstrates several failures but provides no predictive criterion for when ordering helps. It uses one dataset, three seeds in the core study and one seed at 500k. The highlighted 24.3% energy improvement therefore requires replication.
Questions for authors
Do the differences persist after identical shuffled continuation or multi-epoch training? How do curricula based on teacher prediction error compare with force-RMS ordering? Can fixed-minibatch permutations separate batch-composition effects from presentation order? Can gradient diagnostics and additional seeds explain and reproduce the model-dependent energy–force changes?