- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper investigates per-example weight-perturbation sensitivity as a curriculum difficulty score, using random filter-normalized directions and per-example gradient ascent. It analyzes ranking stability, correlations with existing scores and human difficulty, and curriculum performance on CIFAR-100. It also relates ranking changes at larger radii to weight-quantization degradation. The proposed curriculum improves over random ordering but remains below standard training.
Critical strengths
- The stability analysis separates variation across perturbation directions from variation across reference models. This is useful for judging whether a difficulty ordering reflects a reproducible property of the data.
- The paper uses a random curriculum with matched pacing and includes standard-training baselines. It reports the negative performance results openly, which helps assess the practical value of the score.
Critical weaknesses
- The score does not justify the interpretation as sample-level minima sharpness. An aggregate-loss minimum need not minimize each example's loss, so gradient-guided increases may be dominated by first-order effects. Equation (4) maximizes an averaged directional profile, not the loss over a neighborhood, and can yield negative values. Ranking changes also do not establish local convexity or chaos. Matching overall perturbation norms cannot establish a common quantization-collapse mechanism when directions and layerwise structure differ. These issues affect the paper's central explanation.
- The curriculum evidence does not establish a reliable benefit beyond loss. The best sharpness result is 78.40%, versus 78.36% for CELoss, with only one seed. The eligible pool stops at 90%, so ordering is confounded with exclusion of the hardest examples and unequal exposure. Validation uses one dataset and pacing schedule, and all curricula use SAM. Without repeated full-coverage experiments, matched optimizer comparisons, and scoring-cost measurements, neither a practical advantage nor a sufficiently supported negative conclusion has been established.
Questions for authors
- Section 3: What does each score measure in a local loss expansion? Can you separate gradient effects from curvature and justify Equation (4) as a sharpness proxy?
- Sections 4–5.3 and Appendix B: Why does the curriculum stop at 90% coverage? Do the results hold with full coverage, multiple seeds, and a held-out test evaluation?
- Sections 5.1–5.3: Does sharpness add predictive or curriculum value after controlling for loss and gradient norm? Please include the full scoring cost in this comparison.
- Section 5.4: What direct evidence supports local convexity, chaos, or a shared quantization threshold? Do the results survive comparisons with matched parameter sets and perturbation norms?
We thank the reviewer for the careful review. We agree with several points, have narrowed our claims accordingly, and correct a misdescription of our pacing schedule.
Q1: What the scores measure. We agree that θ⋆ minimizes the aggregate loss, not each Le, so ge=∇Le(θ⋆)=0. Expanding the direction-averaged profile of Eq. 4 gives αge⊤dˉ+2Dα2∑jdj⊤Hedj. The first-order term has zero mean and shrinks as 1/D; the second is a Hutchinson-type estimate of a filter-normalized Hessian trace. Eq. 4 is thus an average-case sharpness (Andriushchenko et al., 2023) plus a first-order residual, not a worst-case ϵ-sharpness; negative values reflect negative averaged curvature, possible because θ⋆ is not a minimum of Le. The gradient-guided score is first-order dominated (each step adds ≈Δ∥Mge∥), as are Keskar et al.'s ϵ-sharpness and SAM's inner step. We added this expansion to Section 3, dropped "minima", and list the unseparated terms as a limitation.
Q4: Convexity, chaos, quantization. We agree that ranking changes do not establish convexity or chaos and replaced that wording. Our direct evidence is how the score scales with the radius (App. Figs. 8-9): on average 155 at k=0.25, 3.3×106 at k=1, 4.8×1013 at k=2. A quadratic model predicts a factor of 16 from k=0.25 to 1; we observe ≈2×104, whereas from k=0.1 to 0.25 the deviation is mild (×22 vs. ×6.25). The second-order description thus fails in the same interval in which the orderings decorrelate, so we now speak of a near-quadratic and a strongly non-linear regime. For quantization, matched global norms indeed do not establish a shared mechanism: per-channel quantization is scaled per channel like our filter-normalized directions, but it is deterministic, uniform and, at b=2, rounds most weights to zero. We did not match per-layer norms and now claim only that accuracy and the ordering break down at a similar relative perturbation.
Q2: Pacing, seeds, evaluation.
Q3: Cost. On the WideResNet used for the curricula, sharpness agrees with the established orderings at only ρ=0.25-0.52, yet the curricula perform alike; on EfficientNet-B0 it ranks examples much like converged loss (ρ=0.73). We therefore claim no value beyond loss; a gradient-norm control is listed as a limitation. Cost beyond the shared reference training, in passes over the training set: CELoss/ProbabilityScore 1 forward pass; PredictionDepth 17 truncated passes plus k-NN fits; random sharpness D⋅∣Ak∣=50⋅3=150 forward passes (D directions, 3 coefficients each); CumulativeAccuracy/FirstIteration 200 (one per checkpoint); gradient-guided up to T=20 forward+backward steps with a weight copy per example (memory ∝ batch size × #parameters). Sharpness is thus not cheaper than the baselines and brings no gain over CELoss. Added to Section 4, together with a Limitations paragraph.