acceptodds
Under review as a conference paper at ICLR 2027

From Spectral Measurement to Feedback Collapse: The Limits of Adaptive Orthogonal Optimization

Abstract

Low-rank orthogonal optimizers reduce computation and communication but require a rank hyperparameter that is costly to tune. We ask whether this rank can be inferred directly from the momentum spectrum. We develop a spectral diagnostic based on random matrix theory, using corrected residual-energy and tail-quantile estimators to identify structured directions above the estimated noise edge. Experiments on GPT-style models from 60M to 350M parameters show that momentum spectra are gapless and approximately power-law, while sampling noise accounts for under of the estimated noise floor. The diagnostic remains stable across scales, selecting a rank fraction of – as model size grows sixfold, but validation loss continues to improve beyond the estimated rank, indicating that it provides a lower bound on useful rank rather than an optimization optimum. When used online with error feedback, the criterion induces a positive feedback loop that drives – of weight matrices toward rank one. A pool-wide subtraction modification eliminates this collapse across all tested scales. These results distinguish spectral rank measurement from adaptive rank control in low-rank orthogonal optimization.

Primary area:
optimization
Keywords:
Low-rank optimization, Orthogonal optimizers, Random matrix theory, Error feedback, Spectral diagnostics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1Um9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper explores whether the momentum spectrum may be used to determine good rank parameters for low rank optimizers. The paper introduces a noise threshold diagnostic procedure to estimate the rank for which higher rank singular vectors in the momentum spectra consist of noise. The paper empirically finds that momentum spectra are approximately gapless and that model performance reliably improves with higher rank optimization. It also identifies a "feedback collapse" phenomenon that may occur with online used of the noise threshold diagnostic procedure and introduces a method to correct for it. Experiments are conducted on langauage models ranging from 60M to 350M parameters. The paper claims that its method provides a " lower bound on useful rank", but this claim requires further justification. The fact that validation loss improves beyond the selected rank is not evidence that the procedure finds a lower bound on "useful rank".

Critical strengths

The paper identifies the failure mechanism of feedback collapse and successfully introduces a method to correct for it, empirically preventing feedback collapse across the experimental model scales.

Critical weaknesses

The rank selection procedure is not practically justified by the paper's findings. The paper does not demonstrate that the rank selection method it introduces gives a procedure for finding favorable points on a cost/quality curve. The paper shows that its diagnostic is stable across model scales, but the case is not made that the paper's rank selection method is one that should actually be used. In particular the paper does not show why its rank selection method would be favorable to a simple fixed rank choice. The procedure should be evaluated against such alternatives under a specified objective, for example loss achieved for a fixed computational budget. In its current form, the paper does not make a clear connection between a stable spectral measurement and practically useful optimizer rank.

Questions for authors

Are there any ways in which the rank hyperparameter found by your procedure could be shown to select ranks that are favorable in some sense to a simple baseline?

Reviewer_R4XSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the potential and limitations of adaptive rank selection based on the noise edge of the momentum spectrum. To this end, the authors propose a spectral diagnostic that combines corrected residual energy with a tail-quantile estimator. However, this paper shows that in the context of LLM pre-training, the spectrum of momentum lacks a clear spectral gap; furthermore, validation loss continues to improve even when the rank increases beyond the estimated rank. When combined with selective error feedback, the paper demonstrates that residual accumulation raises the noise threshold, which can incur rank collapse. The authors mitigate this issue via "pool-wide subtraction".

Critical strengths
  1. The paper highlights that stable spectral measurement does not necessarily guarantee stable adaptive rank control, providing a useful perspective on low-rank orthogonalization.

  2. Pool-wide subtraction eliminates rank collapse while keeping the rank selection criterion and parameter update unchanged, which provides empirical evidence for the failure cases.

  3. The authors demonstrate rank collapse across various LLM scales while identifying settings in which it does not happen. This suggests that the instability depends on the interaction between spectral thresholding and selective error feedback.

Critical weaknesses

I have several concerns about the paper.

  1. In my opinion, there is not enough justification for applying the C+4SC+4S assumption to the actual spectral estimator, which requires more explanation in order to strenghten the paper's main claim.

  2. Singular value thresholding does not evaluate the optimization value of individual directions, and the meaning of structured rank remains unclear.

  3. Since computational costs are not compared at the same optimization quality, it is difficult to clearly evaluate the practical benefits of the proposed diagnostic.

  4. Pool-wide subtraction eliminates rank collapse but does not fully recover fixed-rank performance, and the main experiments are limited to relatively small models.

Questions for authors

Please refer to the weaknesses.

Reviewer_TRLyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates rank selection in low-rank orthogonal optimization. The authors propose a method to analyze the impact of discarded components and identify and address feedback collapse. Experiments are conducted on language models with 60M, 124M, and 350M parameters using the FineWeb dataset.

Critical strengths
  1. The research question is interesting and very worth exploring.
  2. The proposed methods ppear novel and interesting.
Critical weaknesses

The experiments are limited to the FineWeb dataset. This narrow evaluation raises concerns about the generalizability of the proposed methods to other datasets and tasks.

Questions for authors
  1. Can the proposed methods generalize to other datasets and tasks beyond those evaluated in the paper?
  2. Do the proposed methods rely on any specific assumptions about the model architecture or training process?
  3. Could the authors provide more intuitive explanations of how the proposed methods are derived and why they work?
1 comment

Thank you for the response, and I will keep the current rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: (1/2)

We thank the reviewers again for their detailed feedback. All five experiments announced in our first response are complete. We report every result, including those that contradict claims in the submission.

The main change concerns what the noise-edge criterion measures. We built a split-batch probe that measures the sampling noise of the momentum directly, at fixed weights (E2). This noise turns out to be strongly anisotropic, so the Marchenko–Pastur edge cannot separate signal from noise. We therefore withdraw the claims that relied on a small noise share. This finding does not affect Feedback Collapse or the pool-wide fix, and both replicate on a second corpus (E4).

Main results

  • Split-batch noise probe (E2; Section 4.2, Fig. R1). At fixed weights, sampling noise accounts for 78–96% of the estimated noise floor, compared with under 11% in the submission. The criterion selects the same rank on pure noise as on the momentum (34–39% of min⁡(m,n)\min(m,n)).
  • Two-run decomposition by shape (E1). At 350M, the two-run estimate is 0.011 for rectangular matrices and 0.146 for square ones. E2 shows that the two-run method itself is biased, so we no longer rely on it.
  • Per-layer rank profile (E3). Compared with uniform rank at the same budget, allocating rank by the profile changes the loss by −0.0012 (60M) and +0.0026 (124M). Neither difference is significant over three seeds.
  • Second corpus (E4; SlimPajama-6B, 124M). With selected-EF, 85% of matrices are flagged as collapsing (FineWeb: 88%). Pool-wide subtraction removes the collapse (0%).
  • Sparse rank updates (E5). Recomputing the rank only every 10 or 100 steps leaves the loss and the subspace turnover unchanged, and cuts step time by 35%.

Revisions to our claims

We withdraw:

  1. The two-run noise decomposition (Section 4.2, App. D) and the claims based on it. These are the claims that sampling noise accounts for under 11% of the estimated noise floor and that over 89% of the sub-threshold component is not sampling noise, together with the fit of ff against the momentum coefficient.

  2. The lower bound on useful rank, including the narrower version in our first response.

  3. The 2.2× cost claim, already dropped in our first response, and the per-layer allocation suggestion (App. A.13). What still holds:

  4. The noise edge gives no natural rank cutoff, but for a different reason than we stated. The sampling noise itself does not follow the Marchenko–Pastur law, so the edge falls inside the noise spectrum.

  5. For LowRankMuon, loss improves monotonically from rank 1/16 to 1/2 at all three scales, although most of the energy in the added directions is noise. If low-rank orthogonal updates worked as spectral denoising, adding these directions should hurt. It helps instead, and we leave the mechanism open.

  6. Feedback Collapse and its resolution, now shown on two corpora. The loop works through the noise-floor estimate: unselected directions accumulate in the buffer and raise the threshold. It does not depend on whether the selected directions carry signal or noise, so E2 leaves it intact. Part 2 describes the probe in detail and lists the planned revisions.

AuthorsOpenReview ↗

General Response: (2/2)

How the noise probe works

At each step, we split the micro-batches into a first half and a second half. Half the difference of their mean gradients has the same covariance as the sampling noise of the full-batch gradient, and it contains no signal. We accumulate this difference with the optimizer's own momentum rule. The result, DD, has the distribution of the sampling noise in the momentum MM. The probe only reads gradients and uses its own random generator, so it does not change the training trajectory.

The probe needs a momentum that is linear in the gradients, so we ran it on LowRankMuon; the error feedback in Dion is not linear. We probed the 60M run every 50 steps up to step 2400 and the 350M run every 100 steps up to step 800. Both runs follow the learning-rate schedule of the 5000-step runs.

60M, steps 100–80060M, steps 2000–2400350M, steps 100–800
Noise share of the noise floor, ff (sketch / full spectrum)0.80 / 0.780.96 / 0.940.92 / 0.90
Noise share of momentum energy, ∥D∥F2/∥M∥F2\Vert D\Vert_F^2 / \Vert M\Vert_F^20.420.630.43
Rank kept on MM / on DD, as a fraction of min⁡(m,n)\min(m,n)0.391 / 0.3930.336 / 0.3430.377 / 0.373
Singular values of MM above s1(D)s_1(D), per matrix1.61.21.3
s1(D)s_1(D) relative to the isotropic Marchenko–Pastur edge6.98.310.8

The last row compares s1(D)s_1(D) with the edge of an isotropic matrix that has the same Frobenius norm (median over matrices). The spectra in Fig. R1 show three more things:

  • Beyond the first few directions, the momentum spectrum coincides with the noise spectrum. Above index 0.03min⁡(m,n)0.03\min(m,n), their squared singular values differ by at most 13%.
  • Nearly all of the momentum's energy in excess of the noise lies in its top five singular values. The leading direction of the noise alone carries 37–40% of its energy.
  • At initialization, the noise carries only 2–3% of the momentum's energy, and the leading direction of the momentum holds 98% of it. So when the momentum contains strong structure, the probe detects it. The gapless, power-law spectrum we reported therefore mostly reflects the sampling noise, and the selected rank reflects the shape of that noise spectrum.

Revision plan

We will make the following changes in the revised PDF and highlight them:

  • Abstract, introduction, contributions and Section 6: we will rewrite them around the corrected findings and remove the withdrawn claims.
  • Section 4.2: we will replace the two-run decomposition with the split-batch probe and add the spectra (Fig. R1). In App. D, we will give the probe protocol and keep the two-run results as a superseded method.
  • Section 5.2: we will add a schematic of the feedback loop.
  • Appendix A: we will add E3, E4 and E5, revise the hypothesis in A.9 in light of E5, and rewrite A.15 to report the anisotropic noise as a measured property.
  • Terminology: we will rename the selected rank the "MP-edge count", define each term where it first appears, and correct the label in Probe 2. We are happy to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.