acceptodds
Under review as a conference paper at ICLR 2027

Fisher Geometry of Activations for Deep Representation Learning: Diagnosis and Mechanism

Abstract

Residual connections and normalization are central to deep-network training, yet their effects on layer-wise activation spectra remain incompletely characterized. The primary contribution of this paper is a staged diagnosis of those spectra and a local account of the measured patterns. We measure effective rank, participation ratio, and condition number of activation covariance, together with the Fisher information of a local Gaussian activation model. Under that Gaussian model, Fisher information for the mean is the precision, and layer linear Fisher information (LFI) is the squared length of a mean shift in this metric. LFI is separate from parameter Fisher information. Covariance eigenvalues are reciprocals of Fisher eigenvalues. On the tested MLP, CNN, and attention encoders, plain stacks often concentrate covariance energy, residual variants often have broader spectra, and per-channel standardization changes diversity and conditioning. These are coordinate-dependent measurements: normalization rescales those coordinates, so broader covariance is not invariant evidence of preserved information. Locally, a residual identity path changes gain propagation, but the leading Jacobian singular value does not determine effective rank. FisherNorm learns a per-channel standardization exponent from the task loss. Under the shared placement protocol, it is comparable to the strongest baselines.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
activation Fisher geometry, representation learning, normalization, residual connections, spectral geometry, FisherNorm

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9qUEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how residual connections and normalization affect activation covariance spectra and linear Fisher information in MLP, CNN, and attention encoders. It combines empirical comparisons with a local Jacobian interpretation and introduces FisherNorm, which learns a per-channel standardization exponent. The results show broader spectra in many residual variants and competitive task performance for FisherNorm.

Critical strengths
  • The staged comparisons and matched-normalizer residual analyses provide concrete evidence of how architectural choices affect the measured activation spectra.

  • The study distinguishes spectral breadth from information preservation and reports cases where broader spectra do not improve task performance.

Critical weaknesses
  • Under the stated joint propagation model, an invertible Jacobian leaves LFI unchanged when covariance is nonsingular and no eigenvalues are truncated, because sensitivity and covariance transform together. The analysis does not establish which departures from these conditions drive the observed LFI changes, weakening the Fisher-specific explanation.

  • This paper correctly states that the largest Jacobian singular value only checks contraction or expansion, but the reported probe does not establish that the tested blocks lie in the near-identity regime or explain their covariance spectra. Thus, the paired comparisons support the broadening itself, while the proposed mechanism remains unverified.

  • The aggregate residual analysis covers MLPs, while the CNN and attention evidence mainly consists of selected configurations. These results do not establish how consistently the claimed spectral patterns hold across architectures and tasks within the 60-scenario suite.

  • Its average classification accuracy is less than 0.1 percentage points above BatchNorm and PowerNorm, and its paired outcomes are mixed. Some scenarios improve, but the results establish competitiveness more clearly than a consistent practical advantage from the added flexibility.

Questions for authors

See weakness

Reviewer_F1KBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies residual connections’ and normalization’s effects on layer-wise activation spectra across MLP, CNN, and attention encoders. The paper uses a first-order Jacobian analysis to interpret their observations while noting that broader spectra do not imply preserved information or better task performance. The authors finally also introduces FisherNorm, which learns a per-channel exponent controlling normalization strength.

Critical strengths

The experiments presented covers a good breadth, spanning several different encoder families, numerous datasets, and both classification and regression. The proposed FisherNorm is simple and cheap to run, adding d (the number of channels or features) parameters per layer only and O(d) operations relative to BN.

Critical weaknesses

The writing can be significantly improved: abstract is somewhat hard to understand (e.g. the name FisherNorm came out of nowhere); many places contain repeated defensive statements which confuses the actual purpose (Section 2.1 and Section 4.5, to name a few); Line 106 contains unresolved ??'s; Table 1, 2, 3 duplicates Table A.7, A.8, A.6/9, respectively. It is also not clear at first read exactly what the Fisher framework contributes and how it motivates FisherNorm.

Questions for authors

What does the Fisher formulation show, beyond standard covariance spectrum arguments, and how does that motivate/justify FisherNorm’s design?

Reviewer_jDJJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how residual connections and normalization affect layer-wise activation geometry in trained MLP, CNN, and attention encoders. It measures covariance-spectrum statistics and directional linear Fisher information (LFI), interpreted through a local Gaussian mean model. Across 13 datasets and 60 configurations, the authors report spectral concentration in many plain networks and changes in spectral diversity and conditioning after adding normalization or residual connections. A local Jacobian analysis provides an interpretation of these observations. As a secondary contribution, FisherNorm learns a per-channel standardization exponent using the task loss. Under a shared four-placement comparison protocol, its aggregate validation performance is close to BatchNorm and PowerNorm. The paper explicitly distinguishes covariance diversity from information preservation and predictive performance.

Critical strengths
  1. The empirical comparison has useful breadth and includes controlled residual comparisons. The study covers three encoder families, 13 datasets, and 60 configurations, with three seeds for nearly all configurations. Comparisons such as BN Pre versus Res-BN Pre hold the normalization family and placement fixed, making the reported architectural differences more interpretable.

  2. The paper carefully separates spectral diversity, directional sensitivity, and task performance. It explicitly acknowledges coordinate dependence and reports cases where broader covariance spectra accompany lower accuracy, such as the MLP/CIFAR-10 comparison in Section 3.3. This qualification makes the empirical findings more useful than an unqualified claim that higher effective rank implies better representations.

Critical weaknesses
  1. The additional explanatory contribution of the Fisher framework is not sufficiently established. Rank collapse and the effects of normalization and skip connections have substantial precedents, including Daneshmand et al. (2020), Dong et al. (2021), and Lubana et al. (2021). Measuring trained nonlinear networks across architectures is a useful extension, but the paper does not clearly identify a new prediction or explanation obtained from its Fisher-specific analysis. The Jacobian norm measurements cannot explain the principal effective-rank observations, as the paper itself acknowledges. Moreover, under the common linear approximation in Eq. (3), nonsingular covariance and an invertible Jacobian imply that LFI is unchanged when both sensitivity and covariance are propagated consistently. Section 4.5 therefore needs to distinguish effects of nonlinear transformations, separately trained models, and numerical truncation from effects of coordinate rescaling alone. My concern is the limited explanatory advance, rather than the absence of a universal theorem.

  2. The experimental evidence does not yet establish the practical value of learning the standardization exponent. Rules A and B select placements using validation scores and report those same scores; independent test performance is not reported. Equal placement budgets are helpful, but do not remove selection effects. The aggregate classification differences are only 0.08 percentage points over BatchNorm and 0.03 over PowerNorm, without uncertainty estimates for the paired differences or an ablation against a fixed partial-standardization exponent. In addition, the description of PowerNorm in Sections 1.1 and 3.1 does not adequately match Shen et al. (2020), whose method uses running quadratic-mean statistics and approximate backpropagation. Since PowerNorm is a leading comparator, its implementation needs clarification. These limitations leave the secondary method contribution too weak to compensate for the limited explanatory advance.

References cited in the review:

  • Daneshmand et al. (2020): “Batch normalization provably avoids ranks collapse for randomly initialised deep networks.” NeurIPS 2020. Official proceedings

  • Dong et al. (2021): “Attention is not all you need: pure attention loses rank doubly exponentially with depth.” ICML 2021, PMLR 139:2793–2803. Official proceedings

  • Lubana et al. (2021): “Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning.” NeurIPS 2021. Official proceedings

  • Shen et al. (2020): “PowerNorm: Rethinking Batch Normalization in Transformers.” ICML 2020, PMLR 119:8741–8751. Official proceedings

Questions for authors
  1. Clarification of the primary contribution: What specific observation or prediction requires the Fisher/LFI analysis rather than covariance-spectrum analysis alone? In Section 4.5, how do you reconcile the proposed LFI interpretation with the identity
(JΔμ)⊤(JΣJ⊤)−1(JΔμ)=Δμ⊤Σ−1Δμ(J\Delta\mu)^\top(J\Sigma J^\top)^{-1}(J\Delta\mu) =\Delta\mu^\top\Sigma^{-1}\Delta\mu

for invertible JJ and positive-definite Σ\Sigma? Which departures from this setting account for the measured changes?

  1. Baseline clarification: Does the PowerNorm implementation include the running quadratic-mean statistics and approximate backward pass of Shen et al. (2020)? Please specify the implementation and normalization axes, and reconcile them with the description of a fixed tensor-level power.

  2. Evidence for the learned exponent: Are independent test results available after selecting checkpoints and placements on validation data, with paired uncertainty estimates? How does learned FisherNorm compare with a fixed exponent at its initialization, λ=sigmoid⁡(3)\lambda=\operatorname{sigmoid}(3), under identical placements? Reporting the learned exponent distributions would also help determine whether exponent learning materially changes the normalization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.