- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how residual connections and normalization affect activation covariance spectra and linear Fisher information in MLP, CNN, and attention encoders. It combines empirical comparisons with a local Jacobian interpretation and introduces FisherNorm, which learns a per-channel standardization exponent. The results show broader spectra in many residual variants and competitive task performance for FisherNorm.
Critical strengths
-
The staged comparisons and matched-normalizer residual analyses provide concrete evidence of how architectural choices affect the measured activation spectra.
-
The study distinguishes spectral breadth from information preservation and reports cases where broader spectra do not improve task performance.
Critical weaknesses
-
Under the stated joint propagation model, an invertible Jacobian leaves LFI unchanged when covariance is nonsingular and no eigenvalues are truncated, because sensitivity and covariance transform together. The analysis does not establish which departures from these conditions drive the observed LFI changes, weakening the Fisher-specific explanation.
-
This paper correctly states that the largest Jacobian singular value only checks contraction or expansion, but the reported probe does not establish that the tested blocks lie in the near-identity regime or explain their covariance spectra. Thus, the paired comparisons support the broadening itself, while the proposed mechanism remains unverified.
-
The aggregate residual analysis covers MLPs, while the CNN and attention evidence mainly consists of selected configurations. These results do not establish how consistently the claimed spectral patterns hold across architectures and tasks within the 60-scenario suite.
-
Its average classification accuracy is less than 0.1 percentage points above BatchNorm and PowerNorm, and its paired outcomes are mixed. Some scenarios improve, but the results establish competitiveness more clearly than a consistent practical advantage from the added flexibility.
Questions for authors
See weakness