acceptodds
Under review as a conference paper at ICLR 2027

Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method

Abstract

We introduce Sven (Singular Value dEsceNt), a new optimization algorithm for neural networks that exploits the natural decomposition of loss functions into a sum over individual data points, rather than reducing the full loss to a single scalar before computing a parameter update. Sven treats each data point's residual as a separate condition to be satisfied simultaneously, using the Moore-Penrose pseudoinverse of the loss Jacobian to find the minimum-norm parameter update that best satisfies all conditions at once. In practice, this pseudoinverse is approximated via a truncated singular value decomposition, retaining only the most significant directions. We show that Sven can be understood as a natural gradient method generalized to the overparametrized regime, recovering natural gradient descent in the underparametrized limit. We test Sven on a variety of regression and classification tasks, including small-scale language modeling with transformers, and find that it is competitive with leading baselines such as Adam, Muon, and K-FAC. We also discuss Sven's memory overhead, which presents a barrier to scaling under a naive implementation, and introduce an optimized implementation that keeps memory usage on par with standard baselines under mild restrictions on model architecture. Beyond standard machine learning benchmarks, we anticipate that Sven will find natural application in scientific computing settings where custom loss functions decompose into several conditions.

Primary area:
optimization
Keywords:
optimization, natural gradients

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_9VgiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Sven constructs an optimizer from the pseudoinverse of the Jacobian of individual sample residuals. It truncates singular directions to control the update and defaults to using each sample's loss as its effective residual. The paper connects this construction to natural gradients in a least-squares setting and develops a batch Gram implementation to reduce the memory cost. Experiments on regression, MNIST and a small transformer show competitive results, with additional studies of singular spectra, numerical stability and backend costs. Larger-model pilots expose important limitations.

Critical strengths
  1. The batch Gram formulation is a useful implementation contribution. Section 2.4 and Appendix E obtain the update through a small eigendecomposition and a vector-Jacobian product, avoiding the full sample-by-parameter Jacobian in supported architectures. Table 14 and Appendix R directly profile the available backends. This makes the global pseudoinverse construction considerably more practical than its straightforward implementation.

  2. The experimental analysis is unusually informative for an optimizer study. Figures 1 and 2 compare against a broad baseline set using held-out loss and actual training time. Figure 3 and Appendix G examine which singular directions and residual components survive truncation. The paper also reports unfavorable CIFAR-10 and GPT-2 results. These analyses help explain both the method's behavior and the limits of the positive results.

Critical weaknesses
  1. I am not clear about the natural-gradient interpretation of the default algorithm. Section 2.3 correctly derives the residual least-squares case, and Appendix C explains the κ=1\kappa=1 construction. Sven uses κ=2\kappa=2 by default, which makes MM the Jacobian of the sample losses. My understanding is that the stated objective has gradient M⊤1M^\top\mathbf{1}, while Sven uses −M†ℓ-M^\dagger\ell. Section 2.2 acknowledges the different norms. I would like the authors to connect this default to the broad equivalence claimed in the abstract, Section 2.4 and Section 3 by specifying the exact metric, objective and scope. I see this as a fixable theoretical interpretation issue.

  2. I find the practical advantage limited by the combination of implementation and task settings. Table 14 shows substantial cost increases when the efficient capture is unavailable, and the favorable main results use small models. Appendix N explains that sequence output heads can also force expensive sample-gradient blocks, beyond the headline BatchNorm and tied-weight restrictions. Sven takes about 2.5 seconds per step in the GPT-2 pilot, compared with about 0.8 for AdamW. I appreciate that the authors describe the accuracy result as preliminary, but the measured cost still limits the broader efficiency claim. I would like the supported regime to be stated more precisely.

Questions for authors
  1. I would like the authors to identify the exact metric and objective that make −M†ℓ-M^\dagger\ell a natural gradient for the default κ=2\kappa=2 and a generic loss such as cross-entropy. I would also like to know whether the equivalence statements can be restricted to the cases actually derived.

  2. I would like the authors to identify, using the existing results, the model and loss setting that shows a useful quality-versus-time tradeoff against the strongest baseline. I would also like to know the conditions under which sequence-layer Gram capture avoids explicit sample-gradient blocks.

Reviewer_XbEPOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a Sven optimizer for training neural networks that considers each example’s error separately. It estimates how small changes to the network’s weights would affect these errors, then chooses a change that tries to reduce them together. It also filters out directions that could cause unstable updates. An efficient implementation reduces memory costs for supported networks. Sven is competitive on several small tasks, but the larger experiments show weaker performance and higher computational costs.

Critical strengths
  • An interesting motivation for a new optimizer: given the individual residuals of every data point in the batch, what single parameter update would bring each of them closest to zero simultaneously
Critical weaknesses
  • Sven underperforms/almost competitive with baselines on small tasks. Results on CIFAR-10 and the preliminary GPT-2 experiment are substantially weaker. I don't find any usefulness of this optimizer from the provided empirical study.

  • While the authors stated clearly that this work is a proof of concept, I think it is not ready to be published on top tire conference. The methodological novelty is not supported empirically, while theoretical analysis is also weak.

  • I found a similar method, improved empirical Fisher (iEF) [1]. It already studies per-example loss Jacobians and it substantially narrows the novelty claim.

[1] Wu et al., An Improved Empirical Fisher Approximation for Natural Gradient Descent , NEURIPS 2024

Questions for authors
  • I think that “We focus our proof-of-concept studies” should be stated earlier, in the title / abstract. Do you agree?
Reviewer_LAXnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Sven, an optimizer that uses a truncated pseudoinverse of the per-sample residual Jacobian. It constructs a joint parameter update through a local least-squares formulation over the data-dependent residuals in a batch. The authors connect this update to natural gradients. They develop a Gram-based implementation that avoids storing the full Jacobian on supported architectures. Experiments on regression, classification, and language modeling evaluate its performance, spectral behavior, and computational cost.

Critical strengths
  1. The idea of constructing a joint parameter update from individual sample losses is interesting and has a clear local least-squares interpretation.

  2. The authors develop a Gram-based implementation that reduces the cost of the direct Jacobian approach while retaining the update across model parameters.

  3. The paper provides detailed spectral analysis, sensitivity studies, and runtime measurements, and reports both positive and negative results.

Critical weaknesses
  1. The natural-gradient interpretation needs clearer justification. For cross-entropy with the default setting, the method uses observed-label gradient outer products and a gradient weighted by sample losses. This generally differs from natural gradient descent on the original loss using the true Fisher. The spectral analysis does not resolve this gap.

  2. Scalability remains unclear. Each step constructs and decomposes the full batch-sized Gram matrix, and reducing the truncation rank does not remove this cost. The language-model experiments use small batches, leaving the algorithm's scalability to larger batches training setting unclear.

  3. The practical benefit is not convincing. The main experiments do not show a clear performance–cost advantage over the strongest baseline for each task. Sven matches AdamW on nanoGPT but requires more time and memory, and performs worse in the larger GPT-2 experiment. The weak Muon and SOAP results in Figure 1 also raise concerns about the baseline settings. In Table 15, Muon's validation gap disappears on the test set.

  4. The contribution beyond closely related methods needs clarification. iEF [1] already uses a per-sample loss Jacobian and a similar Gram-based update, with different sample weights and damping. The paper does not discuss this connection or directly compare hard truncation with damping.

[1] Wu et al. An Improved Empirical Fisher Approximation for Natural Gradient Descent. 2024.

Questions for authors
  1. For cross-entropy, which metric and objective define the claimed natural gradient? Under what conditions does the practical truncated update decrease the original loss? Please also discuss the relation to the Fisher estimation used in Sophia-G [1].

  2. Could the authors test larger batch size and report training quality, wall-clock time, and peak memory? Please separate the costs of Gram construction, eigendecomposition, and the final backward pass.

  3. Why do Muon and SOAP perform worse than AdamW on the nanoGPT validation set? Could the authors verify the baseline settings for Muon and SOAP?

  4. Could the authors clarify the relationship to iEF [2] and provide controlled comparisons that separate the effects of sample weighting and hard truncation, including a damped variant?

[1] Liu et al. Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training. 2023.

[2] Wu et al. An Improved Empirical Fisher Approximation for Natural Gradient Descent. 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.