- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes that the similarity in forward outputs of models doesn't mean they will have similar backward gradients or parameter updates. The authors learned numerical behaviors in SKA and episodic ridge classifiers via comparing the difference in fp32/64 precision across gram construction, cholesky factorization and solution-side solves. To validate that, they built experiments over 18 matched SKA comparisons and found that the mixed precision q110 yields a small median logit discrepancy compared with the FP64 q111, however, it incurs a large discrepancy in gradients and optimization. Overall, I think the experiments designed in this paper are quite responsible. The authors validate their claims via cross-platform runs, custom VJP backward re-computations and so on. However, the problem discussed here may not be that novel to me, and I will argue that this is more like an overcomplication of a common engineering problem.
Critical strengths
-
The issue discussed is important, especially for the realistic training of LLMs. The authors experimentally proved that evaluating forward convergence is not sufficient. Furthermore, they also showed that gradients and VJPs are very sensitive to precision settings. I think the paper can provide a warning to those who works in linear attention or implicit AI.
-
The experimental control and attribution of issue is strong and solid in this paper. Specifically, isolating gradient shifts by fixing optimizer states and inputs, proving backward discrepancy originates from the solver submap via VJPs, showing that algebraic dual reformulations attenuate sensitivity creates a tight. These steps convince a logical loop.
Critical weaknesses
-
The models/benchmarks use are too small to support their claims. The authors did a lot of experiments, however, they are just restricted to small models and datasets. It is hard to say whether the finding in this paper is meaningful for modern LLMs. To my best knowledge, LLMs rarely run explicit cholesky gram solves in attention blocks due to the bottleneck of GPU memory.
-
The authors over-engineer NLA basics in this paper. The "low-precision Cholesky autodiff is numerically shaky" is textbook knowledge. The paper used pages and a cluttered notation scheme to prove that, which may distract readers and cover the real contributions.
Questions for authors
-
Does the proposed precision sensitivity happens similarly in current scale-up architectures, such as Mamba and RWKV? They replace explicit cholesky solves with chunkwise parallel scans or stateful recurrent solves.
-
Have you tried that in a very long training (I will say over 50k steps), does a <10% AdamW update discrepancy will really degrade the final performance? I guess maybe the momentum buffers can smooth out such noise?
Response to Reviewer F2Te
Thank you for your careful assessment of the experimental controls and for raising the question about AdamW’s momentum buffers.
We agree that the numerical sensitivity of Cholesky and its derivatives is well known. The paper spent too much space and notation on that point. The finding that changed our own precision choice was what happened when we transferred the ridge schedule. On WikiText, the setting with FP64 only for Gram construction (q100) finished all five runs when it learned its own ridge, with task loss similar to the FP64 reference. When q100 replayed the same-seed q111 ridge schedule from the start of training, four of five runs failed. Moving Cholesky to FP64 allowed all five to finish. We will give that result more prominence in the next version and use descriptive stage names in the main text.
Does momentum smooth out the discrepancy? In the ridge classifier, yes, in most of the matched comparisons we measured. With the saved AdamW state restored, the update discrepancy was smaller than the gradient discrepancy in 355 of 360 comparisons. For the same gradients, a first step from zero moments gave a larger discrepancy in all 360. Your suggestion about optimizer history fits this model well.
In SKA, the Table 3 updates already include the restored AdamW state. The median update discrepancy exceeded the median gradient discrepancy at both checkpoints: 6.14% versus 5.95% at step 100, and 14.44% versus 12.12% at step 1,000. All nine early update discrepancies were below all nine later values. The pooled median of 9.88% obscures that separation. Over the same interval, the median logit discrepancy rose from 0.83% to 1.37%.
Our longest runs ended at step 5,000. By then q110 and q111 had similar WikiText task cross-entropies under the same evaluation solver (2.38697 ± 0.00135 and 2.38716 ± 0.00098, mean ± sample SD over five seeds, Table 6). We did not train to 50,000 steps, so whether the update differences eventually affect loss remains open.
Scale and architecture. We have not tested Mamba or RWKV, and neither uses the regularized Cholesky solve studied here in the same way. A closer example is MesaNet, which uses conjugate gradients to solve a regularized least-squares problem at each token and was trained at scales up to 1B parameters. Tumma et al. also report instability in their own MesaNet training runs. This gives a concrete larger model in which to test update agreement, though its solver would need its own precision measurements. MesaNet, ICLR 2026 · Preconditioned DeltaNet, Appendix E.5