- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Sven constructs an optimizer from the pseudoinverse of the Jacobian of individual sample residuals. It truncates singular directions to control the update and defaults to using each sample's loss as its effective residual. The paper connects this construction to natural gradients in a least-squares setting and develops a batch Gram implementation to reduce the memory cost. Experiments on regression, MNIST and a small transformer show competitive results, with additional studies of singular spectra, numerical stability and backend costs. Larger-model pilots expose important limitations.
Critical strengths
-
The batch Gram formulation is a useful implementation contribution. Section 2.4 and Appendix E obtain the update through a small eigendecomposition and a vector-Jacobian product, avoiding the full sample-by-parameter Jacobian in supported architectures. Table 14 and Appendix R directly profile the available backends. This makes the global pseudoinverse construction considerably more practical than its straightforward implementation.
-
The experimental analysis is unusually informative for an optimizer study. Figures 1 and 2 compare against a broad baseline set using held-out loss and actual training time. Figure 3 and Appendix G examine which singular directions and residual components survive truncation. The paper also reports unfavorable CIFAR-10 and GPT-2 results. These analyses help explain both the method's behavior and the limits of the positive results.
Critical weaknesses
-
I am not clear about the natural-gradient interpretation of the default algorithm. Section 2.3 correctly derives the residual least-squares case, and Appendix C explains the construction. Sven uses by default, which makes the Jacobian of the sample losses. My understanding is that the stated objective has gradient , while Sven uses . Section 2.2 acknowledges the different norms. I would like the authors to connect this default to the broad equivalence claimed in the abstract, Section 2.4 and Section 3 by specifying the exact metric, objective and scope. I see this as a fixable theoretical interpretation issue.
-
I find the practical advantage limited by the combination of implementation and task settings. Table 14 shows substantial cost increases when the efficient capture is unavailable, and the favorable main results use small models. Appendix N explains that sequence output heads can also force expensive sample-gradient blocks, beyond the headline BatchNorm and tied-weight restrictions. Sven takes about 2.5 seconds per step in the GPT-2 pilot, compared with about 0.8 for AdamW. I appreciate that the authors describe the accuracy result as preliminary, but the measured cost still limits the broader efficiency claim. I would like the supported regime to be stated more precisely.
Questions for authors
-
I would like the authors to identify the exact metric and objective that make a natural gradient for the default and a generic loss such as cross-entropy. I would also like to know whether the equivalence statements can be restricted to the cases actually derived.
-
I would like the authors to identify, using the existing results, the model and loss setting that shows a useful quality-versus-time tradeoff against the strongest baseline. I would also like to know the conditions under which sequence-layer Gram capture avoids explicit sample-gradient blocks.