- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how pruning affect the geometry of the Fisher Information Matrix (FIM). The authors introduce a log-determinant based measure to characterize the parameter coupling and prove that removing any parameter monotonically decreases this metric under a fixed-Jacobian assumption. Experiments further show that different pruning methods generally reduce Fisher coupling and that Adam updates become increasingly aligned with natural-gradient updates as sparsity increases.
Critical strengths
Pros
The paper addresses an interesting question about how pruning changes the local geometry of neural networks. The theoretical analysis is clearly presented, and the experiments cover multiple architectures, datasets, and pruning methods. The additional analyses of damping sensitivity and parameter-matched compact dense models also strengthen the empirical findings.
Critical weaknesses
Cons
-
Theorem 3.2 essentially follows from the Schur complement properties of positive-definite matrices. The result holds for general positive-definite matrices and does not rely on any specific property of the Fisher matrix or neural networks. Moreover, the theorem guarantees monotonicity only for the unnormalized LogDet, whereas the experiments primarily use LogDet/dim. It is therefore unclear how much the theorem explains the observed empirical behavior.
-
Even when the Fisher matrix is diagonal, Adam generally uses a different preconditioning rule from natural gradient. Theorem 3.4 additionally assumes that the diagonal optimizer's preconditioner approximately matches the diagonal of the Fisher matrix, but the paper does not establish whether this assumption holds for Adam in practice. Thus, Fisher diagonalization alone does not necessarily imply alignment between Adam and natural-gradient updates. The experiments show that reduced LogDet does not necessarily correspond to higher accuracy. Although the increasing alignment between Adam and NGD is interesting, it remains unclear whether this alignment leads to faster convergence, better optimization, or other practical benefits.
Questions for authors
Q:
-
Beyond the mathematical effect of reducing matrix dimensionality, what Fisher specific insight does Theorem 3.2 provide?
-
To what extent does Adam's actual diagonal preconditioner satisfy the assumptions of Theorem 3.4?
-
Can reduced Fisher coupling predict or improve practical optimization behavior?
2 comments
I thank the author for their detailed response and the modification of the paper, I think those modification have solve my concern.
First, Jacobian sensitivity redundancy and the experiments make Theorem 3.2 more clear, and those modifications give a greater support to the paper.
Second, I agree that the author clarified the connection between adam and natural gradient, and the revised statement is more accurate.
Though the author can not provide the results about the practical optimization behavior, I think this is an open question rather than a significant shortcoming, so I'm willing to raise my rating despite this problem remains.
Q1, W1 — Fisher-specific interpretation of Theorem 3.2. We agree that the underlying matrix inequality follows from the general Schur complement identity. Its significance in our setting lies in its interpretation through parameter sensitivities. Specifically, the decrease from removing parameter i can be written as f(H)−f(H−i)=−log(1−ρi2), where ρi2 measures how well the corresponding Jacobian column can be reconstructed from the surviving columns through ridge regression. Thus, removing a parameter with redundant output sensitivities (large ρi2) produces a larger reduction in coupling, whereas removing one with a distinctive effect on the outputs (ρi2≈0) produces little or no reduction. Importantly, the reduction depends on the redundancy of the removed parameter's sensitivities, not merely on the number of parameters eliminated. We have incorporated this interpretation into Theorem 3.2. The result provides a fixed-Jacobian structural baseline, while our experiments investigate diagonalization after pruning and retraining. The reviewer is also right that Theorem 3.2 applies to the unnormalized LogDet, not to LogDet/dim; we now make this distinction explicit in the paragraph following Theorem 3.2. Motivated by this comment, Appendix E also compares normalized and unnormalized LogDet along IMP: under fixed-Jacobian restriction, the unnormalized value decreased in all 33 paired comparisons, as the theorem predicts, and the normalized value in 31 of 33.
Q2, W2 — Adam and natural-gradient alignment. We agree that Fisher diagonalization alone does not guarantee Adam–NGD alignment. As noted in the original Adam paper (Kingma & Ba, 2015), Adam's second-moment estimator can approximate the diagonal of the Fisher matrix, which motivated our investigation. However, Adam uses inverse-square-root scaling rather than inverse-Fisher scaling, so it does not necessarily satisfy the preconditioner-tracking assumption of Theorem 3.4. Our experiments independently show increasing directional alignment between Adam and SINGD updates as sparsity increases, without establishing preconditioner equivalence. We have clarified this distinction and the conditional nature of the theorem in the revised manuscript.
Q3, W2 — Practical optimization behavior. Reduced Fisher coupling suggests that diagonal curvature approximations may become more accurate, potentially reducing the importance of off-diagonal preconditioning. Our experiments provide evidence consistent with this interpretation. Adam becomes increasingly aligned with SINGD under pruning, whereas SGD does not exhibit the same consistent trend. However, this directional alignment does not establish faster convergence or better optimization performance. Indeed, Figure 1 shows that lower Fisher coupling does not necessarily correspond to higher accuracy. We therefore interpret reduced coupling as a geometric property that may influence optimization dynamics, rather than as a guaranteed predictor of optimization quality. Establishing whether it can be exploited to improve optimization remains an open question.