acceptodds
Withdrawn from ICLR 2027

Generalization Bounds for Parameterized Group Equivariant Convolutional Neural Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Equivariant convolutional neural networks have shown strong empirical performance by incorporating symmetry into their architectures, yet their generalization behavior remains theoretically underexplored, particularly for parameterized group equivariant CNNs, where convolutional filters are represented as linear combinations of basis functions. In this paper, we derive explicit generalization error bounds that characterize the effects of network depth, channel width, kernel size, and group size. Our bounds further capture the effects of filter parameterization, including the number of basis functions, coefficient constraints, and convolution-operator spectral norms, providing quantitative insights into their contributions to model complexity and generalization error. In particular, our analysis shows that different basis parameterizations can lead to different generalization guarantees, even for the same network architecture, and that reducing the effective basis dimension can improve the resulting generalization bound. These results provide a theoretical basis for understanding how filter parameterization affects the generalization of parameterized equivariant CNNs.

Primary area:
learning theory
Keywords:
parametrized group equivariant CNNs, generalization bound, basis functions

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_G2L6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents generalization bounds for parameterized group equivariant convolutional neural networks (PG-CNNs). They show both theoretically and empirically that increasing the number of basis functions or channel numbers worsens generalization. Additionally, they show that the generalization error of PG-CNNs matches the theory they provided. Also, they show that fourier basis perform better as in line with their predictions. Overall, they provide complete theoretical justification for the bounds on these networks and show that they are reflected in experiment.

Critical strengths

The biggest strength this paper has is that the empirical results fit quite well to the theoretical findings. This gives high confidence in anyone who reads these papers that the theoretical findings are proper and useful in the design of future convolutional networks. I can take the experimental and theoretical findings of this paper to further approach future networks with a better design to lower generalization error.

Critical weaknesses

The only critical error I see is that the experimental section is somewhat limited in experimenting with larger ∣G∣|G|. I would be interested in seeing an experiment which shows how large ∣G∣|G| can become before its performance no longer sees an improvement. I feel like this is a missing experiment and would be an interesting result to see tied to their theoretical results.

Questions for authors
  1. In section 5.2, table 5, you only experiment with ∣G∣|G| sizes of 2, 4, and 8. Have you tried going much further with the size of ∣G∣|G| to see how that effects generalization error, or does SVD show that the larger sizes really have too many redundancies and have no benefit?
Reviewer_H1SzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The submitted paper presents generalization bounds for parameterized families of equivariant neural networks. They generalized previour work on generalization bounds for equivariant neural networks by addressing the possibility of different parameterization of convolutional filters. They derive these bounds by upper bounds on the logarithm of covering numbers of the family of equivariant networks with bounded parameters. Subsequently, they derive precise results for models more used in practice. Then, they present and empirical evaluation of their theory.

Critical strengths
  • Soundness: The theory and notation in the paper are consistent and sound. In the extent possible for a conference review, I checked the proofs and I was not able to find any error.
  • Potential impact: Addressing the problem of generalization for equivariant neural networks is a relevant problem in the literature, although, as outlined below, the methodology employed may present some issues.
Critical weaknesses
  1. Complexity Measures as generalization bounds: uniform complexity measures such as ones derived by covering numbers and metric entropy are known to often produce vacuous generalization bounds for over-parameterized models as equivariant neural networks often are [1, 2]. The theorical derivations presented by the authors are valuable results in itself but their limitations and scope of application should be more clearly delineated in the paper.
  2. Unclear Empyrical Validation: the empyrical results shown in Section 5 are not clear. In particular, can the authors provide more information on the plot in Figure 1, which is the exactly the quantity plotted? The majority of the intervals presented in table 3 and 4 overlap. Can the authors provide additional information to understand if the comparisons are significative in some sense? Furthermore, the numerical results do not agree with common understanding from [1, 2], can the authors provide more information on the models, e.g., whether they were trained in the interpolation regime?
  3. Reproducibility: the issues highlighted in the previous point are exacerbated by the absense of supplementary material, therefore the code for reproducing experiments is unavailable.
  4. Clarity: in section 3, it would be beneficial to better introduce the notation employed in the paper. Which are the spaces on which ρin\rho_{in} and ρout\rho_{out} act. Is GG a finite group or does the theory hold for a general topological group? Clearly stating this at the beginning of the paper would help the reader. In particular, I would connect this notation with the examples presented in Section 3.3

The score is mainly influenced by the point 1, 2 and 3 outlined above. However, I am open to discussion and to reassess my score.

Not affecting the proposed score:

  • I would streamline the explanation of both the contributions at page 2, removing unnecessary and not yet introduced mathematical notation at L79-81 (Who are ClC_l and NlN_l?). Similarly for L90-91.
  • Possible typo at L152: In Paper Cohen [...].

References:

[1] Zhang et al., Understanding Deep Learning Requires Rethinking Generalization, ICLR 2017.
[2] Belkin et al., Reconciling Modern Machine-Learning Practice and the Classical Bias–Variance Trade-off, PNAS 2019.

Questions for authors

See "Critical Weaknesses" section.

Reviewer_chdAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the generalization error of group equivariant CNNs. The paper claims to provide bounds that includes the effects of network depth, channel width, kernel size, and group size and many more parameters.

Although there are previous works on generalization error analysis of equivariant networks, this work focuses on the impact of different basis parameterizations, deriving generalization bounds for parameterized group equivariant convolutional neural networks (PG-CNNs). The obtained generalization bound is instantiated for circular harmonic and Fourier bases.

The main definition is given in Section 3, with PG-CNN particularly in Section 3.2. The main results are given in Section 4. The technical results are organized as follows:

  • Lemma 4.1 gives the covering number of the space of single layer equivariant networks.
  • Lemma 4.2 extends it to multi-layer networks.
  • Theorem 4.3 provides the generalization error using the previous covering number.
  • Theorem 4.4 provides similar bounds for the case channel width and number of kernel follow size of CC and C/∣G∣C/|G|.

The proof follows the standard machinery of Rademacher complexity: bound the covering number for the underlying hypothesis space, and then use Dudley's inequality to bound the RC and consequently the generalization error. Among existing methods to bound the covering number, the paper uses the perturbation analysis and volumetric type bounds.

Experimental results are given in Section 5 with image classification and super-resolution as two use cases. The paper has ablations on the number of channels and basis functions and parametrization space.

Critical strengths

The analysis in terms of parametrization space is interesting, and covers an angle that is useful for practitioners.

Critical weaknesses

The theoretical derivations of the paper follows the standard Rademacher complexity path, and therefore, it does not solve a particularly new or challenging problem, just application of standard tool to the current problem.

There are some open questions about the experiment section (see below). The paper does not situate its result within other similar works to see the improvements and failures. It also does not discuss limitations of generalization error bounds, which have been highlighted in the past work.

In general, the focus on parametrization space is intriguing and has potential to be expanded by addressing the points below.

  • There are other works that considered generalization bounds for equivariant networks. These works consider general equivariant networks where the effect of group size, channel widths and other network parameters appear as well. The paper should create a table of such results with the corresponding generalization bound from these papers and compare the key terms. For instance, see the following
    • Anna Sepliarskaia, Sophie Langer, Johannes Schmidt-Hieber, On the VC dimension of deep group convolutional neural networks, NeurIPS 2025 (where two other relevant papers are mentioned as well: "VC dimensions of group convolutional neural networks", and "On the Sample Complexity of One Hidden Layer Networks with Equivariance, Locality and Weight Sharing")
  • Following up on the previous comment, it is not clear how the generalization error bound of the paper compares to other bounds under similar assumptions (for instance standard parametrization that can be mapped to equivariant CNNs parametrized using irreps). Is the GE derived in this paper tighter or looser?
  • In general, Rademacher complexity analysis using covering number bounds and Dudley can introduce dependence on various dimensions that are proof artifacts and can be removed. Direct Rademacher analysis as in NeurIPS 2024, The Sample Complexity of One-Hidden-Layer Neural Networks, can get dimension independent bounds (with Forbenius norm, although for CNNs the authors have also dimension dependence with spectral norm). There is a way enhancing dimension dependence, for instance see Golowich et al paper at COLT 2018: “Size-Independent Sample Complexity of Neural Networks”. These works and limitations should be discussed both theoretically and experimentally.
  • Another issue is that the norm based generalization bounds are known to be vacuous for deep networks, See Fantastic Generalization Measures and Where to Find Them, ICLR 2020, and "Uniform convergence may be unable to explain generalization in deep learning", NeurIPS 2010. Therefore, it is reasonable to doubt if the current generalization analysis approach brings new insights.
  • A related objection can be that the smaller generalization error bound for certain parametrization does not by itself establish that the basis parameterization better predicts the generalization of networks trained with SGD, because it is just an upper bound. To establish this, the authors need to run much larger scale set of experiments.
  • Another interesting aspect of the work is that the theory predicts that larger C and N gives higher generalization error. However, in deep learning, for complex data, overparametrized regime might give lower generalization error. There are many theoretical and experimental works in the literature showing this.
  • In Figure 1, why does GE goes to 1? The generalization error should get smaller as the number of epochs decrease.
  • Another concern in Figure 1 is that theoretical bound is not vacuous and already between 0 and 1. This is surprising and in contrast with the bound in Table 1, 2 and other tables, where there are large generalization errors. Can the authors clarify further?
  • Table 4 and 5 are reporting PSNR, SSIM and ΔL1\Delta L_1 , but the caption and text claims this is the generalization error. The GE is the difference of test and training loss, the one defined in Rademacher setup, which by the way needs to be Lipschitz.
Questions for authors
  • For the same input signal (assuming norm bounded input in the ambient space), the bounds need to be normalized with different parametrization. In other words, the same input signal creates coefficients of different magnitude for different parametrization. Is this accounted for in the paper?
  • Can the authors show correlation plots to show the GE bound is correlated with empirical GE for various choices of parameters? This would at least verify empirically the correlation between bounds.
  • The hope of generalization error field is to find metrics and then improve generalization by regularization (as indicated in the paper: "reducing DlD_l through basis normalization and regularizing the norms of the basis weight vectors". Can authors provide empirical evidence that this holds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.