acceptodds
Withdrawn from ICLR 2027

Generalization Bounds of Stochastic Gradient Descent in Homogeneous Neural Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Algorithmic stability is among the most potent techniques in generalization analysis. However, its derivation usually requires a stepsize under non-convex training regimes, where denotes iterations. This rigid decay of the stepsize potentially impedes optimization and may not align with practical scenarios. In this paper, we derive the generalization bounds under the homogeneous neural network regimes, proving that this regime enables slower stepsize decay of order under mild assumptions. We further extend the theoretical results from several aspects, e.g., non-Lipschitz regimes. This finding is broadly applicable, as homogeneous neural networks encompass fully-connected and convolutional neural networks with ReLU and LeakyReLU activations.

Primary area:
learning theory
Keywords:
algorithmic stability, generalization analysis, homogeneity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WE7tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides the generalization bounds for the homogeneous neural network regimes and proves that this regime enables a slower step-size decay under mild assumptions. This paper also extends the theoretical results in several aspects, such as other loss functions, the specific case of three-homogeneous networks, and non-Lipschitz regimes.

Critical strengths

This paper establishes generalization bounds for HH-homogeneous neural networks via algorithmic stability, and then it specializes these results to the case of three-homogeneous neural networks. Generalization bounds without the Lipschitz assumption are also derived based on the concept of conditional on-average stability.

Critical weaknesses
  1. The experiments evaluate SGD with various step sizes. For validation of theoretical results, experiments are validated only for the three-layer ReLU network. The experiments may not be sufficient to support the claimed broad applicability of the theoretical findings. It is also unclear how those assumptions can be verified.

  2. It remains unclear how the current theoretical analysis could be extended to more general settings or how some assumptions could be relaxed (see Questions).

Questions for authors
  1. Can the theoretical results of this paper be extended to practical scenarios beyond homogeneous neural networks? For example, transformer or large language models?

  2. Can this paper be extended to more general multi-class classification tasks?

  3. Experimental validation on more practical networks is needed, also for those models mentioned in the paper, including VGG-19?

  4. Could the authors explain why Assumption 1 (lower bounded loss) is mild?

Reviewer_NfBWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents generalization bounds for homogeneous neural networks. Using a fairly popular approach of algorithmic stability they derive the generalization bounds under non-convex setting which allows for a slower decay of step size Ω(1/t)\Omega(1/\sqrt{t}). They don't use any additional assumptions like PL conditions or over-parameterized setting and only requires assumption which are fairly standard and used by other previous works. Authors derive the results for H-homogeneous neural networks and show that H<=3 allows for a slower and more practical decay of learning rate.

Critical strengths

The paper delivers what is claims. The paper shows a slower decay of step size Ω(1/t)\Omega(1/\sqrt{t}) for homogeneous neural networks. They utilize the homogeneous property and the fact that normalized weights do not change the classification boundary to give bounds which allow for a slower decay of step size.

Critical weaknesses

The most critical weakness of this paper is its limited applicability.

  1. The scope of the paper is very limited; it addresses the importance of slower decay of learning rate and presents a generalization bound which for homogeneous neural networks which allows for the same. However, the current literature has now shifted away from solely deriving these data independent bounds, as they are very loose in estimating the generalization gap, see Fig 2 of [1].
    • Motivation of this comes from the paper [2] where authors show that these data independent bounds are also valid for training a neural network on randomly assigned labels, where generalization is not possible.
    • Without integrating these data dependent quantities, the contribution of the paper seems very limited as the bounds are mostly unusable in real world.

[1] H. Pandey, A. Bagchi, S. J. Bedathur, and A. Bhattacharya, "Data-Dependent Generalization Bounds for Neural Networks with ReLU," Transactions on Machine Learning Research, 2024.

[2] Zhang, Chiyuan, et al. "Understanding deep learning requires rethinking generalization." arXiv preprint arXiv:1611.03530 (2016).

Questions for authors

Essential

  1. There is insufficient empirical evidences to validate the assumptions on the bound. The experiments only demonstrate that using a slower decay for step size is useful in practice. However, I am interested in empirical experiments that specifically validates the assumptions and insights present in this paper.
    • For example, does the norm of weights actually decay as the training progresses?
    • What happens when training on randomly assigned labels, as in [2]? Does the same behavior still hold?
    • Such experiments will connect the theoretical assumptions and insights to what actually happens when a neural network is trained.

Minor

  • Theorem 5 assumes minimum loss to be strictly positive. However, the architecture of NN does not have any additional assumption which allows this. E.g. setting all last layer weights to zero can allow for zero loss. Requesting authors to highlight this in the draft.
  • Missing essential citation [1]. This work also shows generalization bounds for neural networks and enables a slower decay of learning rate.

[1] H. Pandey, A. Bagchi, S. J. Bedathur, and A. Bhattacharya, "Data-Dependent Generalization Bounds for Neural Networks with ReLU," Transactions on Machine Learning Research, 2024.

[2] Zhang, Chiyuan, et al. "Understanding deep learning requires rethinking generalization." arXiv preprint arXiv:1611.03530 (2016).

Reviewer_UXhPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies generalization of SGD in homogeneous neural networks using normalized parameters. It connects weight-norm dynamics to the effective stepsize on normalized weights and develops conditional stability bounds, including an extension based on conditional on-average stability.

The paper also proposes stepsize schedules intended to support both generalization and optimization. Theorem 8 and Example 1 are used to argue that these objectives can be compatible. My main concerns concern the applicability of the positive-risk premise and whether the optimization example satisfies the theorem’s assumptions.

Critical strengths
  1. The effective-stepsize perspective is informative.

Lemma 3 makes explicit how homogeneity relates the learning rate in the original parameterization to the effective update on normalized parameters. This provides a useful way to analyze how weight-norm dynamics interact with stability.

  1. The paper develops more than one stability route and makes the schedule construction explicit.

The conditional on-average formulation in Definition 9 and Lemma 10 provides an additional analytical route beyond the earlier stability argument. Appendix B.3.1 also explicitly describes the norm-dependent stepsize construction, which helps readers understand the scope of the resulting guarantees.

Critical weaknesses
  1. The main network/loss application does not satisfy its positive-risk premise as stated.

Section 2 requires [ \sigma^2=\min_{|w|=1}L(w)>0. ] However, Proposition 2 allows the final unnormalized layer to be zero while the remaining parameters have unit norm. The loss in Eq. (2), (\max{-Y\Phi(w;X),0}), is then identically zero.

For example, the permitted three-layer linear model [ \Phi(a,b,c;x)=abcx ] at ((a,b,c)=(1/\sqrt{2},1/\sqrt{2},0)) has unit parameter norm and zero loss for every sample, regardless of label noise. Thus, label noise does not ensure the required positive minimum for the stated model class.

This directly affects the intended application because the bounds use constants involving (1/\sigma^2). The authors need to specify an appropriate model, loss, or domain modification. If the domain is restricted, the analysis should also establish that the SGD trajectory remains within it (Section 2; Proposition 2; Assumption 1; Theorem 5).

  1. The claimed compatibility between optimization and generalization is not established by Example 1.

Example 1 gives (H=3), (\bar\sigma^2=2^{3/2}), and (\mu\approx19.7). Even taking the most favorable strong-growth constant (B=1), Theorem 8 requires [ \mu>4B^2H^2\bar\sigma^2(1+\alpha)>101.8 ] for the intended (\alpha>0). The example therefore fails the theorem’s numerical condition.

There is also a conflict with the asymptotic interpretation. Condition 4 implies [ \mathbb{E}\hat L(v_t)-\hat L(v^)\ge \frac{\hat L(v^)}{\alpha}. ] In Example 1, (\hat L(v^*)=1), so a fixed positive (\alpha) imposes a nonzero excess-loss floor. This does not support the claimed asymptotic approach to the optimum. A corrected example satisfying all assumptions is needed (pp. 7–8, Theorem 8 and Example 1).

Questions for authors
  1. How is (\sigma^2>0) ensured despite the zero-output construction? If the parameter domain is restricted, what guarantees that SGD stays within it?

  2. Can the authors provide an example satisfying all conditions of Theorem 8 and supporting the claimed optimization/generalization compatibility? Please specify the admissible range of (\alpha).

  3. In Theorem 11, does “global (\beta)-smoothness” refer to the original loss on all of parameter space or to a smooth extension near the sphere? A nonzero homogeneous loss of degree greater than two cannot have a globally Lipschitz gradient on all of (\mathbb{R}^d). The intended domain and the self-bounding step in the proof should be clarified.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.