acceptodds
Under review as a conference paper at ICLR 2027

The Geometry of Sparsity: Fisher Information Across Neural Network Pruning Methods

Abstract

Modern neural networks contain substantial parameter redundancy, yet the geometric consequences of removing parameters remain poorly understood. We study pruning through the lens of Fisher geometry and investigate how restricting a network to a sparse parameter subspace affects statistical dependencies among its remaining parameters. We introduce a per-dimension log-determinant ratio that quantifies deviation of the Fisher Information Matrix (FIM) from diagonality, providing a measure of weight-space coupling. We show theoretically that, under a fixed-Jacobian damped Gram-matrix surrogate, the pruning operation monotonically reduces this coupling. Empirically, across architectures, datasets, and pruning methods, we observe a consistent reduction in Fisher coupling over broad sparsity ranges, beyond what can be explained by parameter count alone. As the FIM becomes increasingly diagonal, updates produced by diagonal adaptive optimizers become more closely aligned with natural-gradient updates, whereas the same behavior is not observed for SGD. Together, these results suggest that pruning does more than reduce model size: it systematically simplifies the local geometry of the surviving parameter space, producing representations that are more coordinate-separable and better approximated by diagonal second-order structure.

Primary area:
learning theory
Keywords:
Pruning, Deep Learning, Fisher Information, Hessian

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QLSnOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how pruning affect the geometry of the Fisher Information Matrix (FIM). The authors introduce a log-determinant based measure to characterize the parameter coupling and prove that removing any parameter monotonically decreases this metric under a fixed-Jacobian assumption. Experiments further show that different pruning methods generally reduce Fisher coupling and that Adam updates become increasingly aligned with natural-gradient updates as sparsity increases.

Critical strengths

Pros

The paper addresses an interesting question about how pruning changes the local geometry of neural networks. The theoretical analysis is clearly presented, and the experiments cover multiple architectures, datasets, and pruning methods. The additional analyses of damping sensitivity and parameter-matched compact dense models also strengthen the empirical findings.

Critical weaknesses

Cons

  1. Theorem 3.2 essentially follows from the Schur complement properties of positive-definite matrices. The result holds for general positive-definite matrices and does not rely on any specific property of the Fisher matrix or neural networks. Moreover, the theorem guarantees monotonicity only for the unnormalized LogDet, whereas the experiments primarily use LogDet/dim. It is therefore unclear how much the theorem explains the observed empirical behavior.

  2. Even when the Fisher matrix is diagonal, Adam generally uses a different preconditioning rule from natural gradient. Theorem 3.4 additionally assumes that the diagonal optimizer's preconditioner approximately matches the diagonal of the Fisher matrix, but the paper does not establish whether this assumption holds for Adam in practice. Thus, Fisher diagonalization alone does not necessarily imply alignment between Adam and natural-gradient updates. The experiments show that reduced LogDet does not necessarily correspond to higher accuracy. Although the increasing alignment between Adam and NGD is interesting, it remains unclear whether this alignment leads to faster convergence, better optimization, or other practical benefits.

Questions for authors

Q:

  1. Beyond the mathematical effect of reducing matrix dimensionality, what Fisher specific insight does Theorem 3.2 provide?

  2. To what extent does Adam's actual diagonal preconditioner satisfy the assumptions of Theorem 3.4?

  3. Can reduced Fisher coupling predict or improve practical optimization behavior?

2 comments

Q1, W1 — Fisher-specific interpretation of Theorem 3.2. We agree that the underlying matrix inequality follows from the general Schur complement identity. Its significance in our setting lies in its interpretation through parameter sensitivities. Specifically, the decrease from removing parameter ii can be written as f(H)−f(H−i)=−log⁡(1−ρi2)f(H)-f(H_{-i})=-\log(1-\rho_i^2), where ρi2\rho_i^2 measures how well the corresponding Jacobian column can be reconstructed from the surviving columns through ridge regression. Thus, removing a parameter with redundant output sensitivities (large ρi2\rho_i^2) produces a larger reduction in coupling, whereas removing one with a distinctive effect on the outputs (ρi2≈0\rho_i^2 \approx 0) produces little or no reduction. Importantly, the reduction depends on the redundancy of the removed parameter's sensitivities, not merely on the number of parameters eliminated. We have incorporated this interpretation into Theorem 3.2. The result provides a fixed-Jacobian structural baseline, while our experiments investigate diagonalization after pruning and retraining. The reviewer is also right that Theorem 3.2 applies to the unnormalized LogDet, not to LogDet/dim; we now make this distinction explicit in the paragraph following Theorem 3.2. Motivated by this comment, Appendix E also compares normalized and unnormalized LogDet along IMP: under fixed-Jacobian restriction, the unnormalized value decreased in all 33 paired comparisons, as the theorem predicts, and the normalized value in 31 of 33.

Q2, W2 — Adam and natural-gradient alignment. We agree that Fisher diagonalization alone does not guarantee Adam–NGD alignment. As noted in the original Adam paper (Kingma & Ba, 2015), Adam's second-moment estimator can approximate the diagonal of the Fisher matrix, which motivated our investigation. However, Adam uses inverse-square-root scaling rather than inverse-Fisher scaling, so it does not necessarily satisfy the preconditioner-tracking assumption of Theorem 3.4. Our experiments independently show increasing directional alignment between Adam and SINGD updates as sparsity increases, without establishing preconditioner equivalence. We have clarified this distinction and the conditional nature of the theorem in the revised manuscript.

Q3, W2 — Practical optimization behavior. Reduced Fisher coupling suggests that diagonal curvature approximations may become more accurate, potentially reducing the importance of off-diagonal preconditioning. Our experiments provide evidence consistent with this interpretation. Adam becomes increasingly aligned with SINGD under pruning, whereas SGD does not exhibit the same consistent trend. However, this directional alignment does not establish faster convergence or better optimization performance. Indeed, Figure 1 shows that lower Fisher coupling does not necessarily correspond to higher accuracy. We therefore interpret reduced coupling as a geometric property that may influence optimization dynamics, rather than as a guaranteed predictor of optimization quality. Establishing whether it can be exploited to improve optimization remains an open question.

I thank the author for their detailed response and the modification of the paper, I think those modification have solve my concern.

First, Jacobian sensitivity redundancy and the experiments make Theorem 3.2 more clear, and those modifications give a greater support to the paper.

Second, I agree that the author clarified the connection between adam and natural gradient, and the revised statement is more accurate.

Though the author can not provide the results about the practical optimization behavior, I think this is an open question rather than a significant shortcoming, so I'm willing to raise my rating despite this problem remains.

Reviewer_9HkpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how pruning changes Fisher coupling and whether this brings diagonal adaptive optimizers closer to natural-gradient updates. The geometric perspective is interesting, but I am not convinced that the theoretical analysis and empirical observations support the breadth of the conclusions.

Critical strengths
  • The paper examines pruning from a geometric perspective rather than proposing another parameter-importance score.
  • Comparing IMP, random pruning, SNIP, GraSP, and RigL helps establish that the observed behavior is not specific to one pruning procedure
  • The compact-dense controls and damping sensitivity experiments are useful. The authors also acknowledge that lower coupling does not necessarily imply better predictive performance
Critical weaknesses
  • Architecture coverage is thin: only ResNet18/DenseNet121 beyond ConvSmall; generalization unclear; larger-net curvature uses subsampled blocks, missing whole-network coupling.
  • Theorem ≠ full pruning: it covers coordinate restriction only. Separate fixed-Jacobian restriction, immediate post-pruning curvature, and post-retraining curvature.
  • Theory–experiment mismatch: monotonicity is proved for unnormalized LOGDET, but experiments report per-dimension LOGDET; normalized case is unsupported.
  • Fig. 1 is descriptive only: coupling/accuracy trajectories differ but are unexplained; coupling can fall while accuracy worsens, so it doesn’t show diagonalization helps.
  • Adam–natural-gradient link is weak: theorem assumes preconditioner tracks diagonal curvature; SINGD alignment doesn’t isolate this; need exact small-model comparison.
  • Add a limitations section: fixed Jacobian, normalized metric, curvature subsampling, damping, and conditional optimizer result define the conclusion boundaries. This is my main reason for not being able to give a positive evaluation. Please adjust during rebuttal.
Questions for authors
  1. What supports extending the conclusions beyond the convolutional architecture families evaluated here?
  2. How do the results differ between fixed-Jacobian coordinate restriction, immediate post-pruning curvature recomputation, and post-retraining curvature?
  3. What explanatory conclusion should the reader draw from Figure 1 beyond the observation that different pruning methods produce different coupling trajectories?
  4. Does pruning improve the agreement between Adam’s actual preconditioner and diagonal Fisher curvature?
  5. Which curvature estimator is used in each experiment, and how stable are the findings across curvature samples and parameter subsamples?
4 comments

W6, W2, W3 — Scope and limitations. We agree that the relationship between our theoretical guarantees, empirical measurements, and conclusions needs to be more clearly articulated. Our theoretical result establishes a baseline for coordinate restriction under a fixed Jacobian, while our experiments characterize the behavior of trained sparse networks beyond this idealized setting. We have clarified this distinction throughout the revision:

  • Fixed Jacobian: Theorem 3.2 covers coordinate restriction under a fixed Jacobian only; pruning and retraining change the Jacobian (see Q2).
  • Normalized metric: Monotonicity is proved for LogDet, not LogDet/dim, which we use only as an empirical diagnostic to compare matrices of different sizes.
  • Curvature subsampling: ResNet18/DenseNet121 use sampled parameter blocks, which may miss cross-layer interactions. On ConvSmall, layer-wise and full Fisher show similar trends (Appendix B). On a ResNet18/CIFAR10 tensor (Appendix G), our 9-of-90 selection matches uniform draws and tracks the all-weights trend; across three 8,000-example samples, density ordering holds, with deviations ≤ 3.7%.
  • Damping sensitivity: Damping affects the metric's magnitude, but the trends and method differences are stable across the damping values in Appendix D.
  • Conditional optimizer result: Theorem 3.4 gives sufficient conditions (vanishing off-diagonal curvature and accurate diagonal tracking) that Adam need not satisfy; our experiments show increasing directional alignment with SINGD, not preconditioner equivalence.

Overall, pruning frequently reduces Fisher coupling in the evaluated settings, but this is not guaranteed after retraining and does not imply better accuracy or optimization (see Q2, W2).

Q1, W1 — Other architectures. We extended the experiments to a Transformer trained on AG News text classification. The new results confirm decreasing Fisher coupling beyond convolutional architectures. We have included this additional evidence (Appendix F) while acknowledging that generalization to arbitrary architectures remains open. In addition, Theorem 3.2 is architecture-independent: it only uses the damped Gram form of the curvature. We focused on CNNs because of the cost of Fisher estimation.

Q2, W2 — Distinguishing pruning stages. We now distinguish three regimes: (i) coordinate restriction of the original Jacobian, (ii) curvature recomputed immediately after applying the pruning mask, and (iii) curvature after retraining. Theorem 3.2 applies exactly to (i), but not necessarily to (ii) or (iii), since the Jacobian may change. New experiments explicitly compare these regimes and confirm that reduced coupling persists beyond fixed-Jacobian restriction, including after retraining. We have added these results (Appendix E), together with a discussion of the three regimes based on the linearization literature, and clarified the distinction in the revised manuscript.

Q3, W4 — Interpreting Figure 1. The main conclusion is that parameter count alone does not determine Fisher geometry. At matched densities, methods yield substantially different coupling, so the pruning procedure and subsequent optimization shape the geometry. IMP and random pruning reduce coupling most, while RigL can retain higher accuracy at extreme sparsity with greater coupling. Thus, LogDet/dim characterizes sparse-network geometry rather than pruning quality. This shows that coupling and accuracy capture different properties.

Q4, W5 — Adam's preconditioner. We agree that Fisher diagonalization alone does not guarantee Adam–NGD alignment. As noted in the original Adam paper (Kingma & Ba, 2015), Adam's second-moment estimator can approximate the diagonal of the Fisher matrix, which motivated our investigation. However, Adam uses inverse-square-root scaling rather than inverse-Fisher scaling, so it does not necessarily satisfy the preconditioner-tracking assumption of Theorem 3.4. Our experiments independently show increasing directional alignment between Adam and SINGD updates as sparsity increases, without establishing preconditioner equivalence. We have revised the discussion surrounding Theorem 3.4 and the interpretation of the experiments in Section 4.2 to explicitly distinguish the theorem's sufficient conditions from the observed Adam–NGD alignment.

Q5, W1 — Curvature estimation and stability. ConvSmall uses the full-network Fisher; ResNet18/DenseNet121 use sampled within-layer parameter subsets (curvature matrices up to 8,000 × 8,000). We report mean ± standard deviation over ten trials, and Appendix D shows that the main trends persist across four damping values. Appendix G shows robustness to parameter and example subsampling (see W6). We have clarified the estimation and sampling procedures in the revision. The NGD reference is SINGD with dense Kronecker factors, which captures within-layer off-diagonal curvature.

Thank you for the rebuttal and the added Appendices E–G. While the new edits improve transparency and are appreciated, they confirm that the theoretical guarantees do not support the empirical metrics or optimization claims. I am keeping my score of 2 (Weak Reject), as an increase in score would require a major reframing of the paper's core claims including the Abstract and Introduction, which still overstate the link between the fixed-Jacobian theory, post-retraining curvature, and Adam–NGD alignment:

  1. Theory–Experiment Gap (W2, W3, Q2): Theorem 3.2 only holds for unnormalized f(H)f(H) under a fixed Jacobian, whereas the paper relies on normalized f(H)/dimf(H)/\text{dim} after retraining. Appendix E (Fig. 7) even shows f(H)/dimf(H)/\text{dim} rebounding at high sparsity across all three regimes, confirming the theorem does not govern the reported metric.
  2. Broken Link to Adam (W5, Q4): You concede Adam's inverse-square-root scaling violates Theorem 3.4's preconditioner-tracking assumption, and Q4 (testing if Adam's preconditioner actually tracks the Fisher diagonal) remains unanswered. Theorem 3.4 does not explain the SINGD cosine-distance plots.
  3. Toy Scale, Subsampling Bias & Unjustified Bottleneck (W1, Q1, Q5): The new Transformer test (Appendix F) uses a tiny 2-block model (d=32d=32, 7,168 weights/block) on IMP only. Worse, Appendix G (Fig. 9) shows the 9-of-90 subsampling rule underestimates single-tensor coupling by over 50% at 60% density (≈0.95\approx 0.95 vs. ≈2.05\approx 2.05). While I recognize the O(P2)O(P^2) memory cost of explicit dense FIMs, computing log⁡det⁡(C⊤C+λI)\log\det(C^\top C + \lambda I) via the sample-space Gram Weinstein–Aronszajn identity (M×MM \times M with M=8,000M=8,000), K-FAC blocks, or stochastic Lanczos quadrature is completely manageable on modern VRAM. If a specific layer or VRAM bottleneck prevents this, the exact hardware constraints and per-layer parameter counts should be disclosed rather than relying on biased coordinate subsampling.
  4. Purely Descriptive (W4, Q3): Since f(H)/dimf(H)/\text{dim} neither predicts accuracy nor guarantees better optimization, the contribution remains a descriptive observation that falls short of its mathematical framing.

Please correct me if I am wrong in any of these aspects. I am aware of the computational cost for the 3rd point, which looks manageable to my knowledge. Disclosing the exact Fisher bottleneck (e.g., VRAM limits and per-layer parameter support) would help identify the issue better, or sharing the code for the experiments would assist in evaluating the implementation. I will also discuss these points with the other reviewers who have higher confidence in the evaluation. I remain available for fast insight in case any of the points can be addressed.

We sincerely thank the reviewer for pressing us to examine more carefully the relationship between our theoretical results and the empirical metric. This prompted us to revisit the analysis and led to a stronger theoretical result. We believe this substantially improves the mathematical understanding of the phenomenon we investigate. For ease of review, we have highlighted the new and revised text in blue throughout the updated manuscript.

  1. Theory–Experiment Gap. The new Corollary 3.3 establishes a necessary and sufficient condition for decreasing dimension-normalized LogDet under fixed-Jacobian coordinate removal. We further extend this result to the simultaneous removal of multiple coordinates (Appendix A.3): normalized LogDet decreases precisely when the average reduction per removed coordinate exceeds the initial LogDet per dimension. These results provide a mathematical connection between coordinate restriction and the metric used in our experiments. In particular, they establish why normalized LogDet may increase even under fixed-Jacobian restriction, without contradicting Theorem 3.2. We have revised the Abstract, Introduction, experimental interpretation, and Conclusion accordingly. Whether decreasing normalized coupling persists after retraining remains an empirical question, not a theoretical guarantee.

  2. Adam–NGD Alignment. We acknowledge that Theorem 3.5 does not, by itself, explain the observed Adam–SINGD alignment. Although Adam's scaling does not necessarily satisfy the theorem's preconditioner-tracking assumption, this does not preclude empirical directional alignment. We have clarified that the theorem provides sufficient conditions, whereas the cosine-distance results are independent empirical observations. We are examining whether Adam's diagonal preconditioner tracks the inverse Fisher diagonal up to a scalar factor along IMP trajectories, as required by the theorem.

  3. Curvature Estimation and Subsampling. We acknowledge the substantial difference in absolute normalized LogDet between the full-tensor and 9-of-90 estimates in Figure 9, particularly at 60% density. Our robustness results support the qualitative decreasing trends, but do not establish that coordinate subsampling preserves the magnitude of full-tensor coupling. We are investigating the feasibility of sample-space Gram computations and collecting the relevant hardware, memory, and parameter-support details. We will report additional findings if they can be verified within the discussion period.

  4. Scientific Contribution. We agree that reduced Fisher coupling does not necessarily predict accuracy or guarantee improved optimization. However, understanding how pruning transforms neural-network geometry is a scientifically meaningful objective in its own right. A structural characterization need not directly yield better predictive performance or faster optimization to advance our understanding of the underlying phenomenon. Our revised theoretical analysis goes beyond describing empirical trends. It identifies how redundancy among parameter sensitivities determines coupling reduction and establishes exact conditions for decreasing normalized LogDet under fixed-Jacobian coordinate restriction. Empirically, we show that networks with comparable parameter counts can exhibit different Fisher geometries depending on how sparsity is imposed. Together, these results provide new insights into the geometric consequences of neural-network pruning, independently of immediate performance gains.

Reviewer_9HkpOpenReview ↗

Closing Discussion Note: Scope Updates vs. Unverified Implementation Details

Thank you for the revisions, for deriving Corollary 3.3 to formally characterize dimension-normalized LogDet\text{LogDet} under fixed-Jacobian coordinate removal, and for tightening the scope boundaries in the Abstract, Introduction, and Section 4.2. These updates put the paper in much better condition in terms of theoretical clarity.

I waited until the close of the discussion window to post this final follow-up in case the additional empirical checks and implementation details you mentioned could be verified in time.

specifically regarding sample-space Gram feasibility, hardware/VRAM or backend bottlenecks (e.g., operator support limits), Adam's preconditioner tracking, or code disclosure

Because those findings have not landed, I am maintaining my rating of 2 (Weak Reject) (with Confidence: 3, to ensure my evaluation remains fair and balanced rather than over-weighted, I on purpose avoided putting it to 4 evem though im very condifident with my new assessment and have hypothesis on scale limits that I wanted to ask before and it is an answer that I was supposed as a Reader to find it in the paper). While the scope is now much more accurately framed, a score increase requires empirical verification rather than tighter claims alone, particularly to resolve the >50% coordinate-subsampling underestimation (Fig. 9) and test whether Adam's diagonal preconditioner actually tracks the inverse Fisher diagonal along pruning trajectories.

I will share both the strengths of your updated scope and theoretical framing as well as these remaining open questions with the Area Chair and the other reviewers, including those with higher confidence in this subfield, to ensure a fair final assessment. Thank you again for your constructive engagement throughout the rebuttal.

Reviewer_k5fiOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The work analyzes how pruning weights within a neural network alters its geometry, specifically from the perspective of the Fisher Information Matrix. The authors mathematically demonstrate that pruning weights monotonically reduces the unnormalized coupling measure across all pruning algorithms, under a fixed Jacobian assumption. Furthermore, the authors study, in CNN models, how various pruning algorithms impact both the geometry via this coupling and the performance of the pruned network, and how these phenomena are related to natural-gradient updates.

Critical strengths

The paper is written in a manner that is easy to follow and tackles an important problem in terms of understanding how sparsity impacts model characteristics.

The empirical testing of multiple pruning methods across various CNN architectures is appreciated, and it is interesting that the decreasing Fisher coupling still generally holds even without the fixed Jacobian assumption, with retraining, and following normalization.

I also found the relation of gradient descent interesting, and how with increasing sparsity, ADAM became closer to NGD, especially later in training, while SGD did not exhibit the same consistent alignment with NGD.

Critical weaknesses

While useful, the mathematical demonstration that removing parameters decreases coupling seems relatively straightforward, since it follows directly from the Schur complement formula for any positive definite matrix (as noted in A.3 as well). Thus, perhaps it could be simplified more in the main text, with additional empirical results in the appendix moved up.

Questions for authors

Is there any further explanation for why smaller dense models seem to have comparable or higher Fisher coupling than larger sparse models of the same total parameter count? How does performance compare between the smaller dense and larger sparse models?

2 comments

W1 — Simplification of the proof and empirical results. We agree that the proof follows directly from the Schur complement identity. We have simplified its presentation and added a geometric interpretation: the LogDet decrease from removing a parameter is determined by how well its Jacobian column can be reconstructed from those of the surviving parameters. The theorem establishes a fixed-Jacobian baseline; our experiments investigate whether diagonalization persists after pruning and retraining. We will give greater prominence to these empirical findings.

Q1 — Coupling in smaller dense versus larger sparse models. Fisher coupling is determined not only by parameter count, but also by the geometry of parameter sensitivities. One possible explanation is that smaller dense networks must represent the task using fewer intermediate features, potentially increasing dependencies among their parameters. In contrast, larger sparse networks preserve the original width while reducing connectivity, potentially retaining more diverse representations and less correlated parameter sensitivities. This hypothesis is consistent with Section 4.3, which shows that networks with comparable parameter counts can exhibit different Fisher coupling. However, we acknowledge that the proposed mechanism remains to be directly verified. We have included this hypothesis, along with its limitations, in the revised Section 4.3.

Q2 — Accuracy at matched parameter budgets. At matched parameter budgets, the compact dense models remain competitive but are consistently less accurate than the corresponding IMP subnetworks. On MNIST, the compact models achieve 98.36%, 98.02%, and 96.73% test accuracy at 60%, 40%, and 20% of the original parameter count, respectively, compared with 98.66%, 98.58%, and 98.05% for IMP. The differences are larger on KMNIST, where the compact models achieve 86.73%, 83.64%, and 77.25%, compared with 89.37%, 88.75%, and 81.97% for IMP. The comparison across all pruning methods is less uniform: compact models match or outperform some Random, SNIP, and GraSP configurations. We have added these matched-budget accuracy comparisons to Section 4.3 of the revised manuscript.

I would like to thank the reviewers for their reply. The updated presentation of the proof, the hypothesis and limitations, and the discussion of accuracy in large sparse vs. small dense models is appreciated. I thus retain my score and positive assessment of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.