acceptodds
Under review as a conference paper at ICLR 2027

Investigating Sample-Level Sharpness as a Curriculum Scoring Function

Abstract

Curriculum learning presents training examples to a model in order of increas- ing difficulty, with sample-level difficulty estimations typically relying on metrics that are implicitly deduced from the model’s loss landscape. Yet, the connection to minima sharpness, a key characteristic of the local loss landscape, which is often investigated in the context of generalisation on a model or dataset level, has barely been made in the literature. In this work, we introduce per-sample sharpness as a difficulty estimation that measures how quickly the loss on a single example rises under random, filter-normalized weight perturbations of a trained model, along with a Gradient-Guided variant that perturbs along the example’s own loss gra- dient. We evaluate both on CIFAR-10, CIFAR-100, and the Minimum Viewing Time dataset, comparing sharpness against five established scoring functions and human difficulty labels.We find that (i) sharper minima on a sample-level min- ima correlate with higher difficulty according to other difficulty metrics, imply- ing a related but distinct notion of difficulty, which, however, does not correlate with human perception, (ii) random effects w. r. t. to underlying filter-normalized directions can be mitigated through ensembles, which also increase correlation with gradient-guided sharpness, (iii) sharpness-based curriculum learning outper- forms a random-order baseline but not standard training (iv) under post-training weight quantization, model accuracy collapses at the same perturbation magnitude at which the Sharpness Score ordering itself becomes unreliable, consistent with a transition from a locally convex to a chaotic region of the loss landscape. We share our code on https://anonymous.4open.science/r/curriculum-learning-7302/

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Sharpness, Curriculum Learning, SAM, Sample Difficulty

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_uJKcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates per-example weight-perturbation sensitivity as a curriculum difficulty score, using random filter-normalized directions and per-example gradient ascent. It analyzes ranking stability, correlations with existing scores and human difficulty, and curriculum performance on CIFAR-100. It also relates ranking changes at larger radii to weight-quantization degradation. The proposed curriculum improves over random ordering but remains below standard training.

Critical strengths
  1. The stability analysis separates variation across perturbation directions from variation across reference models. This is useful for judging whether a difficulty ordering reflects a reproducible property of the data.
  2. The paper uses a random curriculum with matched pacing and includes standard-training baselines. It reports the negative performance results openly, which helps assess the practical value of the score.
Critical weaknesses
  1. The score does not justify the interpretation as sample-level minima sharpness. An aggregate-loss minimum need not minimize each example's loss, so gradient-guided increases may be dominated by first-order effects. Equation (4) maximizes an averaged directional profile, not the loss over a neighborhood, and can yield negative values. Ranking changes also do not establish local convexity or chaos. Matching overall perturbation norms cannot establish a common quantization-collapse mechanism when directions and layerwise structure differ. These issues affect the paper's central explanation.
  2. The curriculum evidence does not establish a reliable benefit beyond loss. The best sharpness result is 78.40%, versus 78.36% for CELoss, with only one seed. The eligible pool stops at 90%, so ordering is confounded with exclusion of the hardest examples and unequal exposure. Validation uses one dataset and pacing schedule, and all curricula use SAM. Without repeated full-coverage experiments, matched optimizer comparisons, and scoring-cost measurements, neither a practical advantage nor a sufficiently supported negative conclusion has been established.
Questions for authors
  1. Section 3: What does each score measure in a local loss expansion? Can you separate gradient effects from curvature and justify Equation (4) as a sharpness proxy?
  2. Sections 4–5.3 and Appendix B: Why does the curriculum stop at 90% coverage? Do the results hold with full coverage, multiple seeds, and a held-out test evaluation?
  3. Sections 5.1–5.3: Does sharpness add predictive or curriculum value after controlling for loss and gradient norm? Please include the full scoring cost in this comparison.
  4. Section 5.4: What direct evidence supports local convexity, chaos, or a shared quantization threshold? Do the results survive comparisons with matched parameter sets and perturbation norms?
1 comment

We thank the reviewer for the careful review. We agree with several points, have narrowed our claims accordingly, and correct a misdescription of our pacing schedule.

Q1: What the scores measure. We agree that θ⋆\theta^\star minimizes the aggregate loss, not each Le\mathcal{L}_e, so ge=∇Le(θ⋆)≠0g_e=\nabla\mathcal{L}_e(\theta^\star)\neq0. Expanding the direction-averaged profile of Eq. 4 gives α ge⊤dˉ+α22D∑jdj⊤Hedj\alpha\,g_e^\top\bar d+\frac{\alpha^2}{2D}\sum_j d_j^\top H_e d_j. The first-order term has zero mean and shrinks as 1/D1/\sqrt{D}; the second is a Hutchinson-type estimate of a filter-normalized Hessian trace. Eq. 4 is thus an average-case sharpness (Andriushchenko et al., 2023) plus a first-order residual, not a worst-case ϵ\epsilon-sharpness; negative values reflect negative averaged curvature, possible because θ⋆\theta^\star is not a minimum of Le\mathcal{L}_e. The gradient-guided score is first-order dominated (each step adds ≈Δ∥Mge∥\approx\Delta\|Mg_e\|), as are Keskar et al.'s ϵ\epsilon-sharpness and SAM's inner step. We added this expansion to Section 3, dropped "minima", and list the unseparated terms as a limitation.

Q4: Convexity, chaos, quantization. We agree that ranking changes do not establish convexity or chaos and replaced that wording. Our direct evidence is how the score scales with the radius (App. Figs. 8-9): on average 155155 at k=0.25k=0.25, 3.3×1063.3\times10^6 at k=1k=1, 4.8×10134.8\times10^{13} at k=2k=2. A quadratic model predicts a factor of 1616 from k=0.25k=0.25 to 11; we observe ≈2×104\approx2\times10^4, whereas from k=0.1k=0.1 to 0.250.25 the deviation is mild (×22\times22 vs. ×6.25\times6.25). The second-order description thus fails in the same interval in which the orderings decorrelate, so we now speak of a near-quadratic and a strongly non-linear regime. For quantization, matched global norms indeed do not establish a shared mechanism: per-channel quantization is scaled per channel like our filter-normalized directions, but it is deterministic, uniform and, at b=2b=2, rounds most weights to zero. We did not match per-layer norms and now claim only that accuracy and the ordering break down at a similar relative perturbation.

Q2: Pacing, seeds, evaluation.

  • No 90% cap. Appendix B misdescribed the schedule. In the pacing of Wu et al. (2021) we use, the pool reaches the full training set at 90% of the iterations, so the last ≈20\approx20 epochs use all examples for every curriculum. Nothing is excluded.
  • Exposure. In expectation, the easiest 10% are drawn ≈481\approx481 times, the median example ≈159\approx159 and the hardest ≈20\approx20 (standard training: 200200). This profile is identical for every ordering, including the Random Curriculum, so curricula differ only in which examples get which exposure; this is why the Random Curriculum is our primary baseline. Added to App. B.
  • Seeds. We agree: all runs are single-seed, and 78.4078.40 vs. 78.3678.36 (CELoss) is noise; we claim no advantage over CELoss. Five of six radii lie above the Random Curriculum (77.7477.74-78.4078.40 vs. 77.4377.43), which we now present as an indication only.
  • Evaluation. We used no validation split: all runs report best-epoch accuracy on the CIFAR-100 test set, now stated explicitly.
  • Optimizer. All curricula use SAM, so we removed the claim about an ordering-optimizer interaction. The single dataset and pacing schedule are listed as limitations.

Q3: Cost. On the WideResNet used for the curricula, sharpness agrees with the established orderings at only ρ=0.25\rho=0.25-0.520.52, yet the curricula perform alike; on EfficientNet-B0 it ranks examples much like converged loss (ρ=0.73\rho=0.73). We therefore claim no value beyond loss; a gradient-norm control is listed as a limitation. Cost beyond the shared reference training, in passes over the training set: CELoss/ProbabilityScore 11 forward pass; PredictionDepth 1717 truncated passes plus kk-NN fits; random sharpness D⋅∣Ak∣=50⋅3=150D\cdot|\mathcal{A}_k|=50\cdot3=150 forward passes (DD directions, 33 coefficients each); CumulativeAccuracy/FirstIteration 200200 (one per checkpoint); gradient-guided up to T=20T=20 forward+backward steps with a weight copy per example (memory ∝\propto batch size ×\times #parameters). Sharpness is thus not cheaper than the baselines and brings no gain over CELoss. Added to Section 4, together with a Limitations paragraph.

Reviewer_PacaOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes per-sample sharpness as a scoring function for curriculum learning. The sharpness score measures how much the loss of each training sample increases when the weights of a trained reference model are perturbed along random filter-normalized directions, while a gradient-guided variant perturbs the weights along the sample's own loss gradient. The authors analyze the stability of the score across random directions and models, as well as its correlation with the converged loss, five existing scoring functions, and human difficulty labels on CIFAR-10H and MVT. In curriculum learning with WideResNet-28-4 on CIFAR-100, the sharpness score outperforms a random curriculum at most radii, but it underperforms most existing scoring functions and standard training without a curriculum.

Critical strengths
  1. The paper is generally well-written.

  2. On CIFAR-100 with EfficientNet-B0, the authors carefully separate the variance induced by random directions, which ensembling mitigates (Spearman correlation increases from 0.513 to 0.916 at D = 25, L309–312), from the variance across ten independently trained models (0.550 with 50 directions each, L315–317), which ensembling inherently cannot remove.

Critical weaknesses
  1. The biggest concern is the empirical effectiveness of the proposed scoring function. In Table 1, the best accuracy achieved by the sharpness score (78.40% at k = 0.25) is lower than that of most existing scoring functions (78.64% to 78.89%), and every curriculum is at least 2.5pp below standard training with SAM (81.46%).

  2. All runs use a single seed (L377), and the radius k is chosen from six values on the same set. The authors also argue that "a few tenths of a point should not be over-interpreted" since they only used a single seed (L377). The authors should have reported results across multiple seeds instead of using single-seed variance to dismiss these differences.

  3. The scope of the curriculum experiments is narrow. Although the abstract mentions CIFAR-10, CIFAR-100, and MVT, the curriculum experiments only use CIFAR-100 with WideResNet-28-4 (L717), while the other datasets are only used for the correlation analysis.

[Minor]

  1. The paper is currently using the ICLR 2026 style.

  2. In L21, labels.We find -> labels. We find

  3. In L24, w. r. t. to -> w.r.t.

  4. In L104, acomparison -> a comparison

  5. In L254, a trained the reference models reference models -> the trained reference models

  6. In L477, other established scoring function -> other established scoring functions

  7. In L483, a more chaotic regions loss landscape -> more chaotic regions of the loss landscape

  8. British and American spellings are mixed, e.g., generalisation (L15, L81), optimisation (L47), hypothesise (L81), and filter-normalised (L158), while generalize and normalized are used elsewhere. The authors should standardize on one spelling convention.

Questions for authors

Could you share the wall-clock time and GPU memory overhead to compute the Sharpness Score and the Gradient-Guided Sharpness Score, compared with training the baselines?

1 comment

W1: Effectiveness. We agree, and have revised the paper accordingly. The Sharpness Score curricula outperform neither the established scoring functions nor standard training, and we no longer claim any curriculum value beyond the converged loss (Sec. 5.3). We see the contribution in what the score reveals rather than in a better curriculum: (i) the variance across random directions can be separated from, and averaged out independently of, the variance across models; (ii) a single gradient direction recovers the 50-direction ordering (ρ=0.625\rho=0.625) better than independently trained models agree with each other (ρ=0.550\rho=0.550); (iii) sharpness ranks examples much like converged loss (ρ=0.73\rho=0.73), but only moderately like established scores and hardly like human difficulty; (iv) difficulty orderings change regime at about the relative weight perturbation at which quantization destroys accuracy. The ≥2.5\geq2.5pp gap to standard training holds for every curriculum, including the established scores, in line with Wu et al. (2021). It is confounded by unequal exposure under pacing (the hardest examples are drawn ≈20\approx20 times vs. 200200 in standard training), which we now quantify in App. B.

W2: Seeds and choice of kk. We agree that single-seed variance should not be used to dismiss differences, and we could not add seeds within the rebuttal period. We revised Sec. 5.3 so that the caveat applies in both directions: neither the 0.040.04pp advantage over CELoss nor the 0.240.24-0.490.49pp gap to the other scores is established, and the advantage over the Random Curriculum is presented as an indication only. On the choice of kk, we report all six radii rather than a selected one; k=0.25k=0.25 is the radius used in all other analyses and the one at which ensembles of directions agree best (App. Fig. 7), a criterion that uses neither labels nor test data. Without selecting any radius, the six sharpness curricula average 77.88%77.88\%, above the Random Curriculum (77.43%77.43\%) and below every established score (≥78.36%\geq78.36\%). We now also state that all runs report best-epoch accuracy on the CIFAR-100 test set, as we used no separate validation split.

W3: Scope. We agree. The curriculum experiments use CIFAR-100 with WideResNet-28-4 only; CIFAR-10, CIFAR-10H and MVT serve the analysis of the scores. The revised abstract states this explicitly, and the Limitations paragraph lists the single dataset, architecture and pacing schedule.

Minor. All fixed: the paper now uses the ICLR 2027 style, we corrected the typos in L21, L24, L104, L254, L477 and L483, and we use American spelling throughout, including the figures.

Q: Wall-clock time and memory. On a single 12 GB NVIDIA Titan GPU, training the WideResNet-28-4 SAM baseline took 12.312.3 h (excluding evaluation), a curriculum run 12.612.6 h. Beyond training its reference model, the random-direction score needs D⋅∣Ak∣=50⋅3=150D\cdot|\mathcal{A}_k|=50\cdot3=150 forward passes of EfficientNet-B0 (4.14.1M parameters) over the training set and no backward passes, so its memory is that of inference. For comparison, the SAM baseline performs 400400 forward and 400400 backward passes over the training set with a larger model (5.95.9M parameters). The gradient-guided score needs up to T=20T=20 forward and backward passes with per-example gradients (4040 for the adaptive variant in Table 1); at our batch size of 1616, the per-example copies of weights and gradients take about 0.50.5 GB on top of the activations. Among the baselines, CELoss and ProbabilityScore need 11 forward pass, CumulativeAccuracy and FirstIteration 200200, and PredictionDepth 1717 truncated passes plus kk-NN fits. Details are in App. B.

Reviewer_15QtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates per-sample sharpness as a difficulty score for curriculum learning. It introduces a random, filter-normalized perturbation score and a gradient-guided variant, analyzes their stability and relationships with existing difficulty measures, and evaluates the resulting curricula. The paper also explores a connection between perturbation radius and post-training quantization. Sharpness-based curricula outperform the random-curriculum baseline but do not outperform standard training or most existing scoring functions.

Critical strengths

The paper examines sensitivity to perturbation directions, reference-model initialization, perturbation radius, and training checkpoints. Separating variation due to random directions from variation across reference models is useful.

Critical weaknesses
  1. The contribution needs clearer positioning against prior work and broader difficulty baselines.
    Per-sample sharpness and sharpness-sensitive data selection have already been explored in methods such as ESAM and SharpBalance, although their estimators and objectives differ. The paper should identify the specific methodological or analytical advance beyond these approaches. Its difficulty comparisons also omit EL2N/GraNd and forgetting-event counts, despite their relevance to sample difficulty and data selection. FirstIteration does not substitute for a forgetting-count baseline.

  2. The geometric interpretation of the proposed scores remains insufficiently established.
    Equation 4 averages loss profiles across directions before selecting the larger endpoint value. Its relationship to worst-case sharpness within a neighborhood needs clarification. Furthermore, stationarity of the aggregate training loss does not imply stationarity of individual sample losses, so the gradient-guided score can reflect first-order sensitivity as well as curvature. Comparisons against gradient norm or independently measured curvature would help establish what additional information it captures. Comparisons between the two proposed methods are further complicated by their different meanings of the perturbation radius kk.

  3. The connection between static reference-model sharpness and useful curriculum ordering is underexplained.
    Sensitivity around the final reference-model parameters does not directly establish which examples are beneficial early in a newly initialized model’s training. Figures 11–12 show that orderings change during training, but do not evaluate the downstream effectiveness of curricula constructed from different checkpoints. Consequently, the choice of a fully trained reference model is not sufficiently justified by curriculum-performance evidence.

  4. The practical benefit is limited, and the main performance comparisons lack replication.
    The best sharpness-based accuracy is 78.40%, compared with 77.43% for random curriculum and 81.46% for standard SAM training. It also falls below four of the five existing scoring functions. Its advantage over CELoss is only 0.04 percentage points. Since these curriculum experiments use a single seed, the variability and reproducibility of these differences cannot be assessed. The separate analysis of ten reference models does not address this limitation. An analytical contribution remains possible, but Table 1 alone provides limited evidence of the proposed score’s added value.

  5. The PTQ experiment does not adequately support the stronger geometric interpretation.
    Quantization provides a reasonable example of weight perturbation, but model-level accuracy degradation does not directly validate a sample-level difficulty score. Quantization errors and random filter-normalized perturbations also differ in direction and distribution. Similar perturbation magnitudes at which rankings change and accuracy collapses are insufficient to establish a shared “convex-to-chaotic” transition. The paper would benefit from direct tests connecting sample-level sharpness to sample-level quantization damage.

  6. An apparent inconsistency between reported results needs clarification.
    Figure 20 shows validation accuracies reaching approximately 84–88%, whereas Table 1 reports curriculum accuracies around 77–79%. It is unclear whether these correspond to different experimental settings or runs. This discrepancy makes the curriculum results difficult to reconcile.

Questions for authors
  1. Could you also compare the proposed sharpness scores with EL2N under the same curriculum learning protocol?
  2. Do the proposed sharpness scores capture additional information about sample difficulty beyond loss and gradient norm?
  3. How does curriculum performance change when scores are computed from different training checkpoints?
  4. Could you report the mean and standard deviation across multiple random seeds for the main curriculum comparisons?
  5. How does the PTQ experiment support the interpretation of the proposed sharpness scores as measures of sample difficulty?
  6. How do the experiments underlying Figure 20 correspond to those in Table 1, and what explains their different validation-accuracy levels?
1 comment

We appreciate that the reviewer spent so much effort to provide detailed comments on our contribution.

W1 / Q1: Prior work and baselines. We thank the reviewer for pointing out ESAM and SharpBalance; these references are clearly relevant, and we now discuss them in Sec. 2. Both use per-sample sharpness inside the optimizer, measured at the current iterate: ESAM restricts the SAM update to the most sharpness-sensitive examples to reduce its cost, and SharpBalance makes each ensemble member flat on a sharpness-selected subset to keep the ensemble diverse. Our analysis differs in kind: we study per-sample sharpness as a static difficulty score of a trained model, i.e., its stability across directions, models and radii, its relation to established and human difficulty, and its value as a curriculum ordering. We agree that EL2N/GraNd and forgetting-event counts are relevant baselines and that FirstIteration does not substitute for forgetting counts. We could not add them under the same curriculum protocol within the rebuttal period; we now name EL2N and GraNd in Sec. 2 and list the missing baselines as a limitation.

W2 / Q2: Geometric interpretation. We agree that a converged model state does not imply convergence for every individual sample. The revised Sec. 3 makes explicit what each score measures: expanding the direction-averaged profile of Eq. 4 gives α ge⊤dˉ+α22D∑jdj⊤Hedj\alpha\,g_e^\top\bar d+\frac{\alpha^2}{2D}\sum_j d_j^\top H_e d_j, where the first-order term has zero mean and shrinks as 1/D1/\sqrt{D} and the second is a Hutchinson-type estimate of a filter-normalized Hessian trace. Eq. 4 is thus an average-case rather than worst-case sharpness, plus a small first-order residual, whereas the gradient-guided score is first-order dominated. Beyond loss, we find little additional information in our setting: sharpness ranks examples much like converged loss (ρ=0.73\rho=0.73) and adds no curriculum value beyond CELoss. We did not compute gradient norms or independent curvature estimates (see also our response to Nfv5) and list this as a limitation. Since kk is relative to the filter norms for one score and absolute for the other, we compare the two only through the rank correlation of their orderings.

W3 / Q3: Choice of model state. We agree that it would be interesting to explore sharpness orderings from different model states, and that our choice is not backed by curriculum-performance evidence, as we did not train curricula from earlier checkpoints. We chose the final model state because it is the most converged one: most examples are fit well, so their gradients, and hence the first-order effects above, are on average smallest (App. Fig. 16), while Figs. 11-12 show that orderings from earlier checkpoints differ considerably. The revised paper states this rationale and lists the checkpoint comparison as a limitation.

W4 / Q4: Limited practical effect. We understand the concern. As in our response to Nfv5, we see value in analyzing sample difficulty beyond performance gains, especially since related work often finds no overall benefit of curriculum learning over standard training on such benchmarks (Wu et al., 2021). We agree that Table 1 alone provides limited evidence of added value and revised Sec. 5.3 accordingly: with a single seed, neither the 0.040.04pp advantage over CELoss nor the gap to the other scores is established, and we claim no curriculum value beyond the loss. We could not run additional seeds within the rebuttal period.

W5 / Q5: Quantization. We thank the reviewer for pointing out the relevance of directly comparing sample-level sharpness to quantization damage. Model-level accuracy under quantization does not validate a sample-level difficulty score, and quantization errors and filter-normalized directions differ in direction and distribution. As in our response to Nfv5, we removed the not well-founded "convex-to-chaotic" interpretation and now claim only that the difficulty orderings and the model's accuracy break down at a similar relative weight perturbation; the sample-level comparison is listed as a limitation.

W6 / Q6: Figure 20. Thank you for spotting this. Fig. 20 shows a preliminary CIFAR-10 experiment with a CNN-10 trained with Adam under the same pacing, not the CIFAR-100/WideResNet-28-4 runs of Table 1, which its caption wrongly suggested. CIFAR-10 accuracies are naturally higher (84.684.6-88.0%88.0\%). We corrected the caption and restricted the anti-curriculum statement in Sec. 5.3, which is based on this figure, to the CIFAR-10 setting.

Reviewer_Nfv5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates sample-level sharpness as a difficulty scoring function for curriculum learning. The authors introduce two scoring methods based on random filter-normalized perturbations and gradient-guided perturbations. Experiments on CIFAR-10, CIFAR-100, and MVT examine scoring stability, correlations with existing difficulty metrics, curriculum learning performance, and sensitivity to weight quantization.

Critical strengths
  1. The paper extends loss landscape analysis to sample-level difficulty estimation, providing a potentially useful perspective on curriculum learning.

  2. The authors investigate scoring stability across perturbation directions, model initializations, and perturbation radii, offering useful insights into the behavior of sharpness-based difficulty metrics.

Critical weaknesses
  1. Limited empirical effectiveness. The proposed method underperforms standard training and most existing curriculum scoring functions.

  2. Insufficient justification of the scoring mechanism. The proposed scores may capture gradient sensitivity rather than curvature alone, since individual sample losses are generally not stationary at the converged model parameters. Comparisons with simpler gradient-norm scores are missing, leaving the distinct contribution of sharpness unclear.

Questions for authors
  1. How does the proposed method compare with a simple per-sample gradient-norm baseline? Can the authors demonstrate that the benefits arise from loss landscape curvature rather than first-order gradient information?

  2. What evidence supports interpreting the change in difficulty rankings across perturbation radii as a transition from a locally convex region to a chaotic region, rather than simply a change in sample sensitivity to larger perturbations?

  3. The authors hypothesize that noisy or difficult samples exhibit greater sharpness. Is there a theoretical proof supporting this relationship?

1 comment

We thank the reviewer for taking the time and for the detailed comments on our manuscript, which helped us strengthen our contribution.

W1: Limited empirical effectiveness. We agree that the Sharpness Score curricula outperform neither standard training nor most established scoring functions, and the revised paper states this explicitly. This is in line with prior work showing that, in benchmark settings like ours, most curricula do not outperform standard training (Wu et al., 2021; Rampp et al., 2024); the gap of ≥2.5\geq2.5pp to standard training holds for every scoring function we test. We nevertheless see value beyond raw performance, as studying difficulty scores helps to understand properties of the data and their interplay with the model. Here, (i) the variance across random directions separates from the variance across models; (ii) a single gradient direction recovers the 50-direction ordering (ρ=0.625\rho=0.625) better than independently trained models agree with each other (ρ=0.550\rho=0.550); (iii) sharpness ranks examples much like converged loss (ρ=0.73\rho=0.73), but only moderately like established scores and hardly like human difficulty; (iv) difficulty orderings change regime at about the relative weight perturbation at which quantization destroys accuracy.

W2 / Q1: Curvature vs. gradient norm. We agree that per-sample losses are not stationary at θ⋆\theta^\star, so ge=∇Le(θ⋆)≠0g_e=\nabla\mathcal{L}_e(\theta^\star)\neq0, and the revised Sec. 3 now makes explicit what each score measures. Expanding the direction-averaged profile of the random-direction score gives α ge⊤dˉ+α22D∑jdj⊤Hedj\alpha\,g_e^\top\bar d+\frac{\alpha^2}{2D}\sum_j d_j^\top H_e d_j: the first-order term has zero mean and shrinks as 1/D1/\sqrt{D}, while the second is a Hutchinson-type estimate of a filter-normalized Hessian trace. With D=50D=50 directions, this score is thus an average-case curvature measure plus a small first-order residual. The gradient-guided score, in contrast, is first-order dominated (each step adds ≈Δ∥ge∥\approx\Delta\|g_e\|). In our single-seed runs, the first-order score gives the weaker curriculum (77.72%77.72\% vs. 78.40%78.40\%). During exploration, we also observed very small losses and correspondingly small gradients for a large part of the samples (App. Fig. 16). We did not run a per-sample gradient-norm baseline and cannot add one within the rebuttal period; we list it as a limitation. Since the logit gradient of the cross-entropy is pe−yep_e-y_e, we would expect such an ordering to be closely related to the loss ordering, which is included as the CELoss baseline (78.36%78.36\%).

Q2: Convex vs. chaotic regions. We agree that ranking changes alone do not establish this transition, and the description "chaotic" was not well founded; we replaced it. The more direct evidence we have is how the score scales with the radius (App. Figs. 8-9): on average 155155 at k=0.25k=0.25, 3.3×1063.3\times10^6 at k=1k=1 and 4.8×10134.8\times10^{13} at k=2k=2. A quadratic model predicts a factor of 1616 from k=0.25k=0.25 to 11; we observe ≈2×104\approx2\times10^4. The second-order description thus breaks down in the same interval in which the orderings decorrelate, so we now speak of a near-quadratic and a strongly non-linear regime. This is consistent with the smooth region around the minimum that Li et al. (2018b) visualize, but, as the reviewer suggests, it can equally be read as a change in how samples respond to larger perturbations, and we no longer claim more.

Q3: Theoretical support. We cannot provide a proof that noisy or difficult samples exhibit greater sharpness. The hypothesis is motivated by intuition about smooth loss landscapes and the often-proposed link between flat minima and generalization, which we now state explicitly in the introduction. The revised Sec. 3 adds a partial argument: for cross-entropy, He=Je⊤(diag(pe)−pepe⊤)Je+∑c(pe,c−ye,c)∇θ2ze,cH_e=J_e^\top(\mathrm{diag}(p_e)-p_ep_e^\top)J_e+\sum_c(p_{e,c}-y_{e,c})\nabla^2_\theta z_{e,c}, and both terms vanish as the prediction approaches the one-hot label. Confidently fit examples therefore have nearly flat neighborhoods, consistent with the strong rank correlation between sharpness and converged loss (ρ=0.73\rho=0.73). This links sharpness to how well an example is fit, but it does not prove that noisy samples are sharper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.