- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper measures how much of the exact natural gradient descent’s (NGD) improvement over gradient descent is kept by approximations like K-FAC and EK-FAC. Every step is normalised to unit Fisher norm and each method tunes a constant learning rate to control the desired effect. The scoring function is the fraction of the GD-to-NGD gap in training loss that a method closes. The experiments use one-hidden-layer tanh networks with 66–106 parameters on three small regression datasets. Additional runs use a few small CNNs (about 1.5k parameters, 800 examples) and increase the size up to ~6k parameters. The paper also proposes a condition number as a design target for future approximations.
Critical strengths
The question is reasonable, and computing the exact Fisher rather than the empirical Fisher avoids a known pitfall.
Critical weaknesses
- The setting is a toy problem and says little about how these methods are used in practice.
- The headline numbers come from networks with 66 and 106 parameters (a single hidden layer) on regression tasks.
- The CNN experiments are, quoting, “a case study, not a test of external validity”.
- Training is full batch, with no momentum and one fixed learning rate. Damping is below what K-FAC uses in practice, and retention drops as damping increases.
- Every method’s step is normalised with the exact Fisher. No practical approximation has access to that, so the protocol hands each approximation information it would never have.
- The “value” being measured is training loss only. On held-out data, exact NGD is worse than GD in all nine runs. The method with the best retention has the worst held-out loss, see Table 15.
- Small scale is not forced by the need for an exact reference. Benzing (2022) computed exact natural gradients for wide ResNets. Abreu et al. (2026) compared full Gauss–Newton against layerwise Gauss–Newton, SOAP and Muon on 45M–150M-parameter language models.
- The retention number is not reliable.
- It moves when the damping changes and by up to a third when learning rate grid is made finer.
- K-FAC and EK-FAC have nearly identical confidence intervals, and their order flips depending on the settings. Only “Kronecker beats diagonal” is stable, and that was already known.
- All statistics rest on nine, small runs: three datasets, two of them synthetic, with three seeds each.
- The reference itself is unstable. Changing the initial weights by a negligible amount moves exact NGD’s direction by more than 70 degrees in two of the nine runs.
- Exact NGD is not even clearly the right reference. The layer-wise block beats it and the approximations beat it on CNNs on short horizons.
- There is no real evidence for the proposed metric as a target.
- Plain relative Frobenius error does just as well.
- No approximation is ever built or tuned to minimise it, so there is no evidence that optimising it helps.
- It is driven by the extreme eigenvalues, while Section 6 argues that the differences between methods lie in the middle of the spectrum.
- It is the standard preconditioned condition number and has already been used this way in deep learning (e.g., Zhang et al., 2019).
- Several claims are overclaimed.
- The introduction says that without the protocol’s two controls, the correlation between angle and final loss flips. But the flip happens in a different setup. Under the paper’s actual protocol, it is in an interval that includes zero.
- Finding 3 mostly disappears, and at 60 degrees reverses, when the damped-Fisher angle is matched instead (shown in Appendix L). What is left restates the fact that a Fisher-norm budget penalises high-curvature error.
- The “3 to 10 times” factor in Finding 2 shrinks to 1 to 2 times when the angles are measured along exact NGD’s trajectory instead.
- “The gap even widens with scale” is based on one dataset generator and three model sizes.
- Prior work is missing and the idea is well-known.
- Abreu et al. (2026) study essentially the same question at language-model scale and are not cited. There, layerwise Gauss–Newton comes close to the full method but stays below it, unlike the 115% reported here.
- Bernacchia et al. (2018) already showed that block-diagonal natural gradients can match exact ones in deep linear networks.
- Morwani et al. (2025) measure how closely K-FAC and Shampoo match the true curvature on ResNets and ViTs.
- The formal results are from textbooks: Lemma 1 is the Levenberg–Marquardt property, Proposition 1 is Cauchy–Schwarz, and Remark 2 is the Kantorovich inequality. They should not be presented as contributions.
- Writing and structure.
- The paper is hard to read and badly organised. It uses twelve definitions and five “conventions” for simple bookkeeping. The abstract claims percentages that the paper itself says are not stable.
References
- Abreu, Vyas, Kakade, Morwani. The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton. ICLR 2026. https://arxiv.org/abs/2510.09378
- Benzing. Gradient Descent on Neurons and its Link to Approximate Second-order Optimization. ICML 2022. https://proceedings.mlr.press/v162/benzing22a.html
- Bernacchia, Lengyel, Hennequin. Exact natural gradient in deep linear networks and its application to the nonlinear case. NeurIPS 2018. https://papers.nips.cc/paper/2018/hash/7f018eb7b301a66658931cb8a93fd6e8-Abstract.html
- Morwani et al. A New Perspective on Shampoo’s Preconditioner. ICLR 2025. https://arxiv.org/abs/2406.17748
- Zhang et al. Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model. NeurIPS 2019. https://arxiv.org/abs/1907.04164
Questions for authors
- How do the results change with practical settings: larger or adaptive damping, mini-batches, momentum, and each method normalised by its own preconditioner instead of the exact Fisher?
- Can you run the protocol at a realistic scale with a classification loss, using your matrix-free solver? How do your results relate to Abreu et al. (2026)?
- Is there any experiment where making the proposed metric smaller actually improves an approximation?
- Given how sensitive exact NGD is to tiny changes in the initial weights, can you report retention over several perturbed reruns of each run?
Please let us know if you have any further concers
Dear Reviewer Ftk8,
Thank you again for the thoughtful review. Please let us know if you have any remaining concerns. As we approach the end of the discussion period, we would appreciate the opportunity to provide further clarification if needed.