acceptodds
Under review as a conference paper at ICLR 2027

How Much of the Natural Gradient's Value Do Cheap Approximations Retain?

Abstract

Cheap approximations of natural gradient descent (NGD), such as K-FAC, are justified by the argument that a direction close to the natural gradient inherits its benefits. We test this argument on small dense and convolutional networks whose exact Fisher is computable and strongly ill-conditioned. Every update is normalised to unit Fisher norm, damping and horizon are shared, and each method tunes its own step size. We score each method by its retention: the fraction of exact NGD's improvement over gradient descent that it achieves. Under this protocol, on one-hidden-layer networks with at most parameters, K-FAC, EK-FAC and the exact diagonal retain – of the training-loss improvement, and Kronecker approximations beat diagonal ones in every configuration tested. The angle to exact NGD ranks the approximations, but a GD-to-NGD interpolation at the same angle predicts only a third to a tenth of their retention. At a Euclidean angle, error in the lowest-curvature decile keeps all of NGD's benefit, while error in the highest-curvature decile makes the update far worse than gradient descent; at a matched damped-Fisher angle this contrast largely disappears, so it mainly reflects the Fisher-norm step budget. On dense networks, most retention survives at exact NGD's step size. These results point to relative curvature fidelity as a promising design target. In this regime, the exact layer-wise block retains , so much higher fidelity is achievable, at higher cost.

Primary area:
optimization
Keywords:
natural gradient descent, Fisher information matrix, K-FAC, EK-FAC, second-order optimization, curvature approximation, deep learning optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ftk8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper measures how much of the exact natural gradient descent’s (NGD) improvement over gradient descent is kept by approximations like K-FAC and EK-FAC. Every step is normalised to unit Fisher norm and each method tunes a constant learning rate to control the desired effect. The scoring function is the fraction of the GD-to-NGD gap in training loss that a method closes. The experiments use one-hidden-layer tanh networks with 66–106 parameters on three small regression datasets. Additional runs use a few small CNNs (about 1.5k parameters, 800 examples) and increase the size up to ~6k parameters. The paper also proposes a condition number as a design target for future approximations.

Critical strengths

The question is reasonable, and computing the exact Fisher rather than the empirical Fisher avoids a known pitfall.

Critical weaknesses
  1. The setting is a toy problem and says little about how these methods are used in practice.
  • The headline numbers come from networks with 66 and 106 parameters (a single hidden layer) on regression tasks.
  • The CNN experiments are, quoting, “a case study, not a test of external validity”.
  • Training is full batch, with no momentum and one fixed learning rate. Damping is below what K-FAC uses in practice, and retention drops as damping increases.
  • Every method’s step is normalised with the exact Fisher. No practical approximation has access to that, so the protocol hands each approximation information it would never have.
  • The “value” being measured is training loss only. On held-out data, exact NGD is worse than GD in all nine runs. The method with the best retention has the worst held-out loss, see Table 15.
  • Small scale is not forced by the need for an exact reference. Benzing (2022) computed exact natural gradients for wide ResNets. Abreu et al. (2026) compared full Gauss–Newton against layerwise Gauss–Newton, SOAP and Muon on 45M–150M-parameter language models.
  1. The retention number is not reliable.
  • It moves when the damping changes and by up to a third when learning rate grid is made finer.
  • K-FAC and EK-FAC have nearly identical confidence intervals, and their order flips depending on the settings. Only “Kronecker beats diagonal” is stable, and that was already known.
  • All statistics rest on nine, small runs: three datasets, two of them synthetic, with three seeds each.
  • The reference itself is unstable. Changing the initial weights by a negligible amount moves exact NGD’s direction by more than 70 degrees in two of the nine runs.
  • Exact NGD is not even clearly the right reference. The layer-wise block beats it and the approximations beat it on CNNs on short horizons.
  1. There is no real evidence for the proposed metric as a target.
  • Plain relative Frobenius error does just as well.
  • No approximation is ever built or tuned to minimise it, so there is no evidence that optimising it helps.
  • It is driven by the extreme eigenvalues, while Section 6 argues that the differences between methods lie in the middle of the spectrum.
  • It is the standard preconditioned condition number and has already been used this way in deep learning (e.g., Zhang et al., 2019).
  1. Several claims are overclaimed.
  • The introduction says that without the protocol’s two controls, the correlation between angle and final loss flips. But the flip happens in a different setup. Under the paper’s actual protocol, it is in an interval that includes zero.
  • Finding 3 mostly disappears, and at 60 degrees reverses, when the damped-Fisher angle is matched instead (shown in Appendix L). What is left restates the fact that a Fisher-norm budget penalises high-curvature error.
  • The “3 to 10 times” factor in Finding 2 shrinks to 1 to 2 times when the angles are measured along exact NGD’s trajectory instead.
  • “The gap even widens with scale” is based on one dataset generator and three model sizes.
  1. Prior work is missing and the idea is well-known.
  • Abreu et al. (2026) study essentially the same question at language-model scale and are not cited. There, layerwise Gauss–Newton comes close to the full method but stays below it, unlike the 115% reported here.
  • Bernacchia et al. (2018) already showed that block-diagonal natural gradients can match exact ones in deep linear networks.
  • Morwani et al. (2025) measure how closely K-FAC and Shampoo match the true curvature on ResNets and ViTs.
  • The formal results are from textbooks: Lemma 1 is the Levenberg–Marquardt property, Proposition 1 is Cauchy–Schwarz, and Remark 2 is the Kantorovich inequality. They should not be presented as contributions.
  1. Writing and structure.
  • The paper is hard to read and badly organised. It uses twelve definitions and five “conventions” for simple bookkeeping. The abstract claims percentages that the paper itself says are not stable.

References

Questions for authors
  1. How do the results change with practical settings: larger or adaptive damping, mini-batches, momentum, and each method normalised by its own preconditioner instead of the exact Fisher?
  2. Can you run the protocol at a realistic scale with a classification loss, using your matrix-free solver? How do your results relate to Abreu et al. (2026)?
  3. Is there any experiment where making the proposed metric smaller actually improves an approximation?
  4. Given how sensitive exact NGD is to tiny changes in the initial weights, can you report retention over several perturbed reruns of each run?
1 comment
AuthorsOpenReview ↗

Please let us know if you have any further concers

Dear Reviewer Ftk8,

Thank you again for the thoughtful review. Please let us know if you have any remaining concerns. As we approach the end of the discussion period, we would appreciate the opportunity to provide further clarification if needed.

Reviewer_ou2pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyses various approximation methods for Natural Gradient Descent (NGD) on small MLP/CNN models. With the manageable model size, this paper exactly computes the Fisher matrix, and consequently the exact NGD reference. The paper then focuses on estimating the retention of NGD end-of-training loss reduction w.r.t. Gradient Descent (GD) for different approximation methods. By analysing the error in the Euclidean space across the Fisher spectrum and comparing it against the retention level, the paper has observed that the error at different spectral bands has different effect on retention, along with other findings. Finally, based on these findings, the paper proposes the relative curvature fidelity as a design target for new NGD approximation methods, which matches the retention levels of various approximation methods on setups tested in this paper well.

Critical strengths
  1. The paper compares various relevant NGD approximation methods against exact NGD updates in small but well-controlled ML setups. The ranking of retention and exact behaviours of these methods is an interesting finding.
  2. The paper very carefully analyses the retention levels and relations between these approximation methods in the Euclidean space, and proposes an applicable criterion, the relative curvature fidelity, for future approximation method design. These analyses are well-supported within the setups investigated in this paper.
Critical weaknesses
  1. The setups used in the model are too toy-like. These small-scale setups are well-explored in prior works due to its manageable compute of exact Fisher matrix. However, their relevance to modern DL, where much larger setups are involved, is questionable. Extending and investigating the retention analysis to a more realistic setup, e.g. the PEFT setup in [1], could greatly improve the impact of the paper. Note that the practical applicability of the key contributions of the paper, the “relative curvature fidelity”, is also limited because it requires the exact Fisher matrix / many repeated Fisher matrix vector product, which is impractical in realistic DL setups.
  2. As is discussed by the authors, the paper shows weaker validation loss for NGD than GD, despite better final training loss. In my opinion, this weakens the relevance of the retention measurements overall, because the validation performance is more important than training performance in practical DL.
  3. The design target “relative curvature fidelity” is well justified in the tested setups of the paper. However, it lacks stronger evidence that it can indeed be used to generate/distinguish better NGD approximation methods.

[1] Wu, X., Yu, W., Zhang, C., & Woodland, P. (2024). An Improved Empirical Fisher Approximation for Natural Gradient Descent. Neurips, 37.

Questions for authors
  1. Can the authors show that their analysis can be extended to larger transfomer-based setups (such as the PEFT setup mentioned in Weakness 1), and justify that the finding in this paper remains relevant in these more realistic DL setups? This would greatly affect my view on this paper.
  2. Can the author provide stronger evidence (e.g., by providing a new NGD approximation method using the proposed design target) to demonstrate the practical usefulness of the proposed relative curvature fidelity?
Reviewer_9UvyOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper asks how much of exact natural gradient descent’s (NGD) improvement over gradient descent (GD) is retained by cheap approximations of it: K-FAC, EK-FAC, the exact diagonal and an Adam-like square-root diagonal. It also asks which properties of an approximation predict that retention. Main contributions:

  • Kronecker approximations retain about two-thirds of exact NGD’s benefit and diagonal ones less than half.
  • At t a fixed Euclidean angle, error in high-curvature directions is far more costly than error in low-curvature ones.
  • On dense networks, most retention survives when approximations are run at exact NGD’s step size.

I have used AI to look for a summary of SOTA on second order methods, and to avoid English errors.

Critical strengths
  1. Clean separation of direction from step size. Unit Fisher-norm steps, shared damping and per-method tuned step sizes isolate what the approximate direction contributes. The ablation shows that without both controls the angle–loss correlation has the wrong sign.

  2. Exact, verified reference. The true Fisher and exact Kronecker factors, checked against finite differences, make exact NGD a trustworthy baseline. Structural approximation error is separated from sampling noise.

  3. Interpretable retention metric. Normalising against both GD and exact NGD makes results comparable across datasets and seeds. It is reported only where the GD–NGD gap is large enough to be well posed.

  4. Insights beyond end-to-end benchmarks. Angle ranks the approximations but under-predicts their retention by 3–10×, and the step-size split explains when K-FAC “beats” exact NGD. The layer-wise block result points to the Kronecker factorisation as the main source of loss.

  5. Honest robustness analysis. The authors test sensitivity to damping, grid density, width, scale and band choice, and report that only the ordering generalises. They also show that their own Appendix L largely dissolves the spectral-band headline.

Critical weaknesses
  1. Limited scale and realism. The headline numbers come from networks of about 100 parameters, trained full batch with low damping, and scored only on training loss, where exact NGD is worse than GD on held-out loss. Nothing approaches transformer, LLM or MoE scale, so it is unclear how relevant the results are to modern optimisers.

  2. Learned preconditioners are ignored. FishLeg, APO and Frobenius-fitted Kronecker methods already optimise approximations against a fidelity objective, and FishLeg’s loss is exactly the paper’s damped-Fisher metric. Omitting them weakens the novelty claims and leaves the §8 design target untested.

  3. Retention levels depend on the protocol. Levels shift by up to 22 points with damping and by a third with grid density, adjacent confidence intervals overlap, and the K-FAC/EK-FAC order flips. Only the coarse Kronecker-over-diagonal ordering is robust, and exact NGD is not a ceiling (115% for the layer-wise block).

  4. The spectral headline is mostly a normalisation artefact. The abstract leads with the top-versus-bottom band contrast, but Appendix L shows it largely vanishes, and at 60° reverses, under a matched damped-Fisher angle. It mainly reflects how the Fisher-norm step budget prices error, not a property of the approximations.

  5. The design target is weakly supported. κ_P’s angle bound is nearly vacuous, finer orderings hold in only 7/9 pairs, and simpler metrics rank the methods equally well. No method is built or improved using κ_P, so §8 is a suggestion rather than a supported contribution.

Questions for authors
  1. Architecture. Would the Kronecker-over-diagonal ordering still hold for transformer layers or MoE experts, given that weight sharing already broke K-FAC’s accuracy on your convolutions? Could you test this on a single small transformer block?

  2. Generalisation. Since exact NGD is worse than GD on held-out loss, does the retention ordering in Table 2 still hold if retention is computed on held-out loss instead of training loss?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.