acceptodds
Withdrawn from ICLR 2027

Structured Sample-Wise Gradients in Neural Network Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Stochastic gradient descent (SGD) aggregates gradients from individual training examples, yet the organization of these sample-wise optimization directions remains poorly understood. We analyze per-example gradients using pairwise cosine similarity and clustering throughout training. We find that gradient clusters are internally coherent, with highly aligned within-cluster gradients, while gradients from different clusters exhibit much lower similarity. This structure is reproducible across independently trained models and is stronger in deeper layers. Gradient clusters are strongly aligned with class labels early in training, but this correspondence weakens over time. Besides, lower within-cluster similarity is associated with better generalization performance, and pretrained models begin downstream training with lower within-cluster similarity than randomly initialized models. Our results reveal a structured sample-wise organization underlying SGD updates and connect its evolution to class structure, pretraining, and generalization.

Primary area:
optimization
Keywords:
optimization, gradient

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_oterOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper looks into layer-wise per-sample gradients during image classification. It finds that the cosine alignment between these per-sample gradients cluster around class labels, with intra-clusters variance correlating with generalization performance.

Critical strengths
  • The key insight, that gradient directions initially cluster within class labels while becoming more dispersed during training in accordance with better generalization is interesting.
  • This is also the case for the finding that alignment between clusters is not relevant for generalization. It is perhaps even possible to say that between-label clusters should not be aligned.
Critical weaknesses
  • Per-sample gradient analysis itself is not "recent" (line 034) but has been analyzed in fields such as differential privacy for quite some time [1, 2].
  • There is also a range of relevant related work missing, such as [3, 4, 5, 6].
  • It is unclear what the compute overhead of the method is and if it would scale to larger more modern encoders and architectures. This would be particularly interesting as the motivation for some choices, such as using PCA and layer-wise per-sample gradients, is not fully clear and not properly ablated.
  • It seems like PCA will only have a moderate performance gain while introducing unnecessary abstraction and potential noise.
  • On the other hand, the layer-wise approach is a main differentiator to [3, 4, 5] and has only been done (in some sense) on the last layer in [6]. Focusing on the differences and insights here compared to full model or classifier head per-sample gradients would strengthen the work.
  • Generally, connecting the paper's insights to approaches outside of image classification would be desirable to better fit into the current state of research.
  1. Martín Abadi et al. "Deep Learning with Differential Privacy". ACM SIGSAC 2016.
  2. Hanshen Xiao et al. "Geometry of Sensitivity: Twice Sampling and Hybrid Clipping in Differential Privacy with Optimal Gaussian Noise and Application to Deep Learning". ACM CCS 2023.
  3. Stanislav Fort et al. "Stiffness: A New Perspective on Generalization in Neural Networks". 2020. arXiv: 1901.09491
  4. Karthik A. Sankararaman et al. “The impact of neural network overparameterization on gradient confusion and stochastic gradient descent”. ICML 2020.
  5. Jinlong Liu et al. “Understanding Why Neural Networks Generalize Well Through GSNR of Parameters”. ICLR 2020.
  6. Florian A. Hölzl et al. "Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Tasks". NeurIPS 2025.
Questions for authors
  • PCA on normalized gradients is used for efficiency (line 158). Is the PCA computed at each time step? What is the cost of this PCA?
  • Does Figure 5d seems kind of odd. Is it potentially desired to not have clusters aligned, i.e. similar to what we aim for in contrastive learning?
Reviewer_uM9qOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies individual gradient samples in a stochastic gradient descent algorithm which shows that early in the training, the gradients form clusters aligned with the sample class labels, and as the training progresses, this within-cluster alignment decreases. The authors claim that as the within-cluster alignment decreases, the generalization performance of the model improves. This is the underlying reason the pre-trained models experience a better generalization performance compared to randomly initialized models.

Critical strengths
  1. Overall the paper is well written and has a nice logical flow which makes it easy to understand. This includes a clear presentation of the data, making it easy to identify the main findings simply by reviewing the figures.
  2. The topic is interesting and to the best of my knowledge, the observation that the gradients exhibit a clustering structure is novel.
Critical weaknesses
  1. Some of the claims in the paper are under-specified. For example, there is a claim that “lower within-cluster gradient similarity is consistently associated with higher final test accuracy” . Let’s take this claim to the extreme and consider we have only one or two samples per class. In that case the gradient similarity within-clusters is going to be very low, however, since the number of training samples was also extremely low, we cannot have a high final test accuracy. Such claims need to be at least re-written so that the conditions are clear, otherwise it is misleading.

  2. The paper should elevate its findings by introducing an underlying reasoning for why the phenomenon of lower within-cluster similarity emerges as we proceed with training. Is that because as we progress in training, other factors or dimensions, besides the sample labels/classes become part of the learning process? If so, what are those dimensions? Is that a specific input feature? What happens if we re-cluster based on similarity later in the training? What is the common feature those examples have in common later on that causes their gradient to be in the same direction? These are important questions that need to be addressed in such a study, otherwise it appears that the study is just introducing a phenomenon without doing a proper deep dive on it.

  3. Although it is interesting to observe pre-trained models have lower within-cluster gradient similarity and that’s why they are experiencing better generalization performance, I believe that this claim is still too broad given the limited set of experiments this paper offers. How sure are we about the claim that “stronger generalization is associated with lower within-cluster gradient similarity”? What are the edge cases? What are the conditions? What happens in learning tasks besides classification? What are the clusters in that case?

  4. Overall it is not clear how generalizable the observations in the paper are. For example, even though results of Figure 2A are from various initialization techniques, they are all for the same underlying dataset. How much the results of the paper are then dataset agnostic? What are the characteristics the dataset should have in order to have such observations? Going through the appendix clarifies that the results were done only on CIFAR-10 which makes the claims very limited.

Questions for authors
  1. How is the main intuition coming from this paper’s observations any different from the intuition we gained from prior studies that show gradient diversity is related to generalization? I understand that there are some nuance differences between the details, but the high level intuitions gained from the studies remain very similar which makes the impact of this paper insignificant.

  2. How is the observation in this paper related to the early stopping or generalization metrics that have been previously proposed (examples: [1][2][3][4][5][6])? If we take a mini-batch size of 1 then this paper’s claims may be simply redundant?

[1] K. A. Sankararaman et al., 2019. The impact of neural network overparameterization on gradient confusion and stochastic gradient descent. [2] D. Yin, et al., 2017. Gradient diversity: a key ingredient for scalable distributed learning. [3] J. Fu, et al., 2020. Rethinking generalization of neural models: A named entity recognition case study. [4] H. Mehta, et al., 2020 Extreme memorization via scale of initialization. [5] M. Forouzesh and P. Thiran, 2021, Disparity Between Batches as a Signal for Early Stopping. [6] S. Fort, et al., 2019. Stiffness: A new perspective on generalization in neural networks.

Reviewer_ZSiWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how gradients from individual training examples are organized during neural network training. The authors calculate per-example gradients, compare their directions using cosine similarity, and cluster examples with similar gradients. They report that the clusters are coherent and reproducible across training runs, initially align with class labels, and change as training continues. They also find that lower similarity within clusters is associated with higher test accuracy, and that pretrained models show lower within-cluster similarity before fine-tuning.

The authors test these findings across layers and training stages, compare independent runs, and examine different PCA dimensions and cluster counts. They also compare pretrained and randomly initialized models. The results suggest that per-example gradient structure changes during learning and may be related to generalization. However, the accuracy relationship is correlational, and the paper’s descriptions of datasets and model architectures are inconsistent, making some experiments difficult to assess or reproduce.

Critical strengths

The paper studies training at the level of individual examples. This view shows how examples may support or conflict with one another during learning. It could help researchers better understand how neural networks learn.

Critical weaknesses
  • The paper does not show how its findings could be used in practice. It describes patterns in gradients but does not show how this knowledge could help improve model training, choose better data, or make models perform better. This makes the practical value of the work unclear.

  • The evidence is limited. The reported link between gradient patterns and test accuracy is based on a small number of models and does not show that these patterns cause better performance. Some experiment descriptions are also inconsistent, which makes the results harder to assess.

Questions for authors

I have none questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.