acceptodds
Withdrawn from ICLR 2027

Data-Dependent Complexity Bounds for Saturating Networks via the Pushforward Density

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We derive a new Rademacher complexity bound for neural networks with saturating activations that requires neither input truncation nor activation surrogates. Our key insight is that the data distribution regularizes the inverse Jacobian’s boundary singularity: under mild tail conditions, the pushforward density remains bounded despite divergent inverse derivatives. We characterize this boundedness via asymptotic analysis and show that depth increases the density supremum only polynomially without worsening the singularity, extending to high-dimensional and dimension-reducing maps. Experiments confirm that our density-based measure correlates more closely with generalization gaps and yields strictly tighter bounds than prior methods.

Primary area:
learning theory
Keywords:
Rademacher complexity, generalization bounds, pushforward density, saturating activations, data-dependent bounds

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Xa3YOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies generalization bounds for neural networks with saturating activations such as tanh. The authors use the pushforward density to avoid the infinite bounds caused by inverse Jacobians in prior Koopman-based approaches. They derive data-dependent Rademacher complexity bounds and support the analysis with both theoretical and experimental results.

Critical strengths

None

Critical weaknesses

My main concern is the paper’s limited scope and significance. It mainly addresses a technical limitation of an existing Koopman-based framework, but does not really show what new insight into neural networks or generalization follows from this. The central idea is essentially a standard change-of-variables argument, and the more involved technical results do not add much conceptual understanding beyond the specific problem considered. I also have doubts about the theoretical correctness.

Questions for authors
  1. Could the authors clarify the significance of studying this question?

  2. The proof of Theorem 4.2 seems to have gaps in both the uniform parameter Lipschitz argument and the use of Dudley’s entropy integral. Could you clarify these issues?

  3. Specifically, in lines 1125–1132, the uniform parameter Lipschitz bound relies on bounded layer inputs. However, the first-layer input can be unbounded (e.g., Gaussian). How does the claimed bound hold in this setting?

  4. In lines 1139–1155, how is the application of McDiamid's inequality and Dudley's entropy integral justified? The increments Zθ−Zθ′Z_\theta-Z_{\theta'} are not generally centered, and the pointwise upper bound does not imply the claimed bounded-difference estimate.

Reviewer_NN96OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new Rademacher complexity bound for networks with saturating activations. The framework does not require input truncation or surrogate activations. The main contibutions are theoretical. They complexity derivation is clean and the theory is well motivated. I did not have enough time, or the full expertise to check all the proofs in depth as the paper use and combined several theoretical concepts. If the proofs are correct, I think the paper could be valuable to the community. The theory seems interesting and potentially valuable, but, as in many theory-heavy papers, the empirical validation is relatively weak and does not fully validate all the theoretical claims.

Critical strengths

The paper is built around a very nice theoretical idea: replace a crude worst-case Jacobian bound by an analysis of the actual pushed-forward data density. They analyse it under different tail assumptions, including Gaussian mixtures. This is an interesting theoretical contribution that helps avoid overly restrictive assumptions. They also studies how the density bound evolves with network depth, showing polynomial growth in some specific settings.

Critical weaknesses
  • The authors state that the bound depends only on the largest component scale, whereas theorem 3.3 depends component-wise on mu_j, sigma_j and pi_j.
  • The proof of the main result (thm 4.2) may have some gaps, the uniform parameter-Lipschitz bound also requires bounded inputs? The argument in step 2 does not justify why Zθ−Zθ′Z_\theta-Z_{\theta'} has centered subgaussian increments.
  • The main limitation is not a disconnect between the experiments and the theory but the fact that the empirical evidence is limited.
  • The bound is polynomial in depth KK, but the polynomial degree is also proportional to the dimension d. This may still lead to very large bounds in high dimensions. The bound is still loose as the the push forward density is replaced by the worse case.
  • Figure 2: Are these actual risk bounds, unnormalized complexity terms or log-transformed quantities? I have more concerns about the correlation results statistically. Figure 2 shows the Spearman rank correlation coefficient but these results are not reliable enough given the small number of points. So, the correlation might be high just because we do not have enough data points.
  • The empirical scope is narrow because the experiments seem restricted to tanh MLPs on MNIST. I am wondering if the results persist for other types of functions and architecture. Or variations (for example label noise) which may strongly modify the representation distribution?
  • The experimental protocol is sometimes not very rigourous. For example the authors vary network width while keeping the input distribution and image resolution fixed. This may change the data representation and mixt the relationship between width, density complexity and generalization.
  • Some subsections are referred to as sections, which makes the cross-referencing confusing. In the text around lines 226–227, Proposition 4.1 is referred to as Theorem 4.1.
Questions for authors
  • see above
Reviewer_7LACOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper derives data-dependent complexity bounds for networks with saturating activations by analyzing their pushforward densities. It characterizes when input-tail decay controls inverse-Jacobian singularities, studies depth dependence in specific architectures, and evaluates density-based measures against empirical generalization gaps.

Critical strengths
  1. The idea of analyzing the input density jointly with the inverse Jacobian is clear and well motivated. It provides a natural way to address a limitation of existing Koopman-based bounds.
  2. The paper obtains concrete tail conditions and depth-scaling results for specific network families, and extends the density formulation to dimension-reducing encoders.
Critical weaknesses
  1. There is a gap between the theoretical MM and its empirical estimate. The theory requires a density bound uniform over both outputs and network parameters, whereas experiments fit Gaussian/GMM densities to individual trained networks and maximize over finite samples. No estimation guarantee establishes these quantities as valid upper bounds.
  2. The scope of the depth-scaling claims is limited. Polynomial growth is established for special compositions and product networks, while general learned weights introduce additional dependencies. The full-parameter bound also contains ρL−1\rho^{L-1}, which can grow exponentially with depth.
Questions for authors
  1. Can the authors quantify the density-estimation error and explain how the empirical estimate relates to the supremum over the entire parameter class?
  2. Under what conditions do the polynomial depth-scaling results extend to networks with learned weights? How should these claims be interpreted alongside the ρL−1\rho^{L-1} term?
  3. Do the experimental networks satisfy the assumptions needed for finite uniform MM? Can the authors demonstrate non-vacuous generalization bounds, beyond correlations with observed gaps?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.