acceptodds
Withdrawn from ICLR 2027

When Spectral Accuracy Controls Low-Rank Updates

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Equivalent factor coordinates of a low-rank product lead to different Euclidean factor-gradient updates, and a spectral curvature objective can be precisely optimized over such coordinates while the resulting product update remains undetermined. We characterize when the spectral objective's accuracy controls the product response. A latent coverage matrix is formed by a spectral optimality certificate for convex monotone spectral criteria on a bounded factor-coordinate budget. Compared with the current gradient, this matrix provides an explicit upper bound on the response error that allows for repeated eigenvalues, optima on the budget boundary, and rank-deficient factors. Full-support criteria identify the response under gradient compatibility. For the largest eigenvalue, at interior optimal points with a simple top eigenvalue, the criterion is exact: the response is fixed if and only if the gradient has no leakage into the free subspace. When identification fails, we provide the extremal first-order response pairing for a certified spectral block in closed form. On reduced-rank digit classifiers, coordinates whose relative spectral excess is certified to be less than still change the product response by as much as 4.9% under squared loss. Under softmax cross-entropy, a measured relative objective deviation of at most from a solved reference allows changes of up to 54%. Each evaluated certificate contains the observed change; under squared loss, projecting the gradient onto the predicted zero-leakage subspace eliminates the effect to numerical precision, and the block extremum improves the response pairing over both uniform endpoints exactly when predicted.

Primary area:
optimization
Keywords:
low-rank adaptation, LoRA, gauge freedom, Hessian, spectral criteria, coordinate selection, matrix factorization, optimization geometry

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KdzoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper asks whether equivalent factorizations that achieve nearly optimal values of a spectral criterion also produce similar matrix updates. The analysis focuses on the non-uniqueness of internal coordinates in matrix factorization. Specifically, for M=UVM=UV, any invertible matrix GG gives an equivalent factorization UG,G−1VUG,G^{-1}V. These factorizations represent the same matrix, but gradient descent on the factors can produce different matrix updates. Therefore, choosing factor coordinates by optimizing spectral criteria does not necessarily lead to similar updates. Building on this, the authors develop a theoretical framework that connects the accuracy of spectral optimization to differences in the resulting matrix updates. The key condition is that the internal directions constrained by the spectral criterion must cover those used by the gradient update. Using a coverage matrix and coefficient, they derive an upper bound on the update error in terms of the spectral objective gap. For the largest-eigenvalue objective, they further characterize when different optimal coordinates lead to the same update. The paper also provides a formula for choosing coordinates that maximize the first-order loss decrease within a fixed region. Toy digit classification experiments support the theoretical predictions.

Overall, I think the paper presents an interesting theoretical idea, but the motivation and practical implications are not sufficiently clear to me. The presentation could also be improved.

Critical strengths
  1. The idea of relating spectral accuracy to the consistency of low-rank updates is original.

  2. The paper provides explicit conditions and error bounds, and the toy experiments support the predictions of the theory.

Critical weaknesses
  1. The motivation is not clear. The authors study guarantees of update consistency without showing what training problems arise when such guarantees are absent. In Abstract and Introduction, the authors unnecessarily emphasizes specific numerical results without a clear explanation of why the underlying question matters for training.

  2. The paper is hard to follow. Some figures are difficult to read because the text overlaps with the plots. The writing is sometimes defensive, and the text repeats many unnecessary details. These issues make it hard to focus on the main ideas.

Questions for authors
  1. Why are the data reduced to 12 dimensions using PCA? Is there a theoretical reason for this choice? Do the conclusions still hold with the original features or a different number of PCA components?

  2. How can the theory guide low-rank optimization in practice? For example, in real low-rank problems (low-rank matrix recovery, LoRA fine-tuning, etc.), what would we do differently based on the theory, and how would that improve optimization?

Reviewer_yVS4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when the accuracy of a spectral objective can reliably reflect the behavior of a low-rank product. The authors establish several theoretical results, including guarantees for convex spectral objectives, an exact condition for the largest eigenvalue, and an explicit characterization of identification failure. Numerical experiments are conducted on reduced-rank digit classification tasks.

Critical strengths
  1. The research question is interesting and worth exploring.
  2. The paper follows a logical structure and provides extensive theoretical analysis.
Critical weaknesses

The experiments are limited to relatively simple classification tasks. This narrow evaluation raises concerns about the practical relevance and generalizability of the theoretical results.

Questions for authors
  1. What is the computational complexity of the proposed methods? Can they scale to more complex and practical problems?
  2. Could the authors provide more intuitive explanations of the main theorems to help readers better understand their meaning and implications?
1 comment

Thank you for the response. I will keep the current rating.

Reviewer_TV2yOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

A low-rank matrix M=UV admits equivalent factorizations M=(UG)(G−1V),G∈GL(k).M=(UG)(G^{-1}V), \qquad G\in \mathrm{GL}(k).

Ordinary gradient descent on the factors depends on the choice of GG. Writing S=GG⊤S=GG^\top, the induced product update is

=-\eta\underbrace{\left(USU^\top D+DV^\top S^{-1}V\right)}_{\mathcal P_S(D)} +\eta^2DM^\top D,$$ where $D=\nabla\ell(M)$. Suppose we select $S$ to minimize a objective $f(S)$, where $f$ belongs to the class of Loewner-monotone spectral functions. The paper asks: If $S$ is nearly optimal for $f$, must its induced gradient update $\mathcal P_S(D)$ be close to the update at an exact optimizer? It is established that the answer is no in general, and yes under additional conditions connecting the curvature and the gradient.
Critical strengths
  1. I checked the core proof, and the paper looks mathematically correct.
  2. The paper provides a precise characterization of the relationship between exact spectral optimality and the induced gradient update. In particular, Proposition 5 establishes a necessary and sufficient condition for update identification under suitable assumptions, offering a clear explanation of when different curvature-optimal factorizations produce different updates.
Critical weaknesses
  1. The motivation is unclear. I don't see a particular reason why we should be interested in choosing the gauge by minimizing a Loewner-monotone spectral function. The function that matters in optimization is condition number, not such monotone spectral function (for example, spectral norm, which is the most extensively studied objective in this paper).
  2. The presentation is hard to understand. The writing often involves non-standard terminologies used before being introduced, and unnecessary abstractions and jargons, though the paper does have reasonable technical content.
  3. From the experiments, the quantitative certificate established in the paper is often too conservative to be useful. In all 84 full-support comparisons in Table 1, the upper bound on the response difference exceeds the norm of the reference response. In the certified stopping experiment, a 1% guaranteed tolerance corresponds to observed errors far below 1%, with a reported median conservatism factor of about 500.
Questions for authors

What is the motivation for selecting factor coordinates by minimizing the largest eigenvalue, trace, or Schatten norm of the pullback curvature, rather than a condition-number criterion or an objective directly related to optimization progress? In particular, can the authors demonstrate that minimizing any of these spectral objectives improves convergence (when the learning rate is allowed to be tuned)?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response for ALL

We uploaded a revised PDF on 13 Nov. Since we had not listed the changes, here they are (numbering as in the revised PDF):

  1. Figure 1 is new and shows the geometry behind coverage and leakage. The introduction now begins with the one-step loss example and the decisions our results are meant to support, and the abstract no longer quotes numbers.
  2. Terms are defined when they first appear, and Table 3 lists the notation. (TV2y)
  3. Figures 2 and 3 were redrawn: larger labels, no text on top of the data. (Kdzo)
  4. Appendix A.3 has the sharper bound that treats the two factor-gradient blocks jointly; Section 6.1 reports what it changes. (TV2y)
  5. Appendix A.6 contains the example with two tuned learning rates. (TV2y, Kdzo)
  6. Section 6.6 and Appendix E.12 add PCA24, PCA48 and raw features; Appendix E.11 reports the cost of certified stopping. (Kdzo, yVS4)
  7. What the paper does not show is collected in Section 8.

The main text is now 10 pages.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.