acceptodds
Under review as a conference paper at ICLR 2027

RETHINKING SUBSPACE SIMILARITY IN LOW-RANK ADAPTATION: WHAT GOVERNS LOCAL UPDATES?

Abstract

Low-rank adaptation is usually understood as optimization under a low-rank constraint, but its update map is not a projection: it weights a gradient by the spectrum the factors were built from. We reframe it as optimization under a spectrum-weighted map, and identify spectral allocation—how the spectrum is distributed across the two factors—as the variable governing early training, not the rank or the selected subspace. We test this reframing with paired upstream updates, shared-gradient probes, and natural-magnitude component interventions. A modified initialization can preserve its selected subspaces almost exactly while changing the weighted update map appreciably—a divergence we call the subspace–update gap. The distinction reaches training: two factorizations of one weight with identical initial effective weight, identical selected subspaces, and a shared learning rate produce ten-step updates lying 77.5°–88.0° apart, so spectral weighting changes the direction of adaptation, not only its speed. The gap is not an artifact of one spectral choice: cross-subspace coupling reproduces the response under principal-component selection but not under minor-component selection, where changes within the selected block dominate. The criterion is measurable with the same interventions and applies to any modification of the reference weights, including quantization and pruning.

Primary area:
optimization
Keywords:
parameter-efficient fine-tuning, low-rank adaptation, spectral initialization, spectral allocation, spectrum-weighted updates, subspace-update gap, optimization geometry

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_yGtaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how perturbations to a reference weight change the gradient-to-update map of spectrally initialized low-rank adapters. It decomposes perturbations into selected, complementary, and cross-subspace components, and argues that preserving selected subspaces need not preserve local update behavior.

Critical strengths

The paper raises a potentially useful distinction between preserving a selected subspace and preserving the resulting update behavior.

Critical weaknesses

The paper is hard to follow, and this is a substantive obstacle to evaluating its contribution.

  1. The abstract introduces “spectrum-weighted map,” “spectral allocation,” “subspace–update gap,” “paired upstream updates,” and “natural-magnitude component interventions” in rapid succession, without first giving a concrete example of the question being tested. Section 3 then adds “SplitProbe” and several response measures. These labels often replace a plain explanation of what changes, what is held fixed, and what each experiment is intended to show. I had to reconstruct that logic from the equations and appendix.

  2. The abstract blurs two distinct results. It highlights the 77.5∘77.5^\circ–88.0∘88.0^\circ update angles from the factor-redistribution experiment (Section 4.6). The naturally delivered perturbations in Section 4.5 produce much smaller ten-step separations: 0.0076%–0.0609% of the reference update for DeBERTa and 0.0054%–0.251% for RoBERTa. The paper should distinguish these settings and their implications more clearly.

  3. Numerical precision is inconsistent and excessive. Section 4.4 reports errors as 0.00014%–0.220%, while Appendix Table 11 includes values such as 0.0001378% and 0.220200%. Such precision for means over ten seed aggregates, without uncertainty in the same table, is hard to justify. Consistent rounding would make the results easier to interpret.

Overall, the paper may contain a good mathematical observation, but its terminology, organization, and numerical presentation make it unnecessarily hard to tell which conclusions are proved, which are observed under controlled probes, and which remain proposed applications.

Questions for authors

I encourage the authors to substantially improve the academic writing, especially by defining terms only when needed, explaining each experiment’s purpose before presenting its metrics, and clearly separating results from different experimental settings.

Reviewer_62whOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines how weight perturbations affect spectrally initialized LoRA. It argues that subspace similarity alone cannot characterize local updates because factor spectra also reweight gradients. Its SplitProbe framework decomposes perturbations and measures their contributions under shared gradients. Experiments show that cross-subspace components largely explain responses for PiSSA, whereas changes within selected subspaces contribute substantially for MiLoRA.

Critical strengths
  • This paper clearly distinguishes identical representations from identical learning behavior. The update map K(G)K(G) makes this distinction explicit and helps prevent from interpreting subspace similarity as evidence of equivalent training behavior.
  • The paper builds solid evidence with analysis. The mathematical analysis shows that the weighted update map can change without subspace displacement.
Critical weaknesses
  • The selected-only intervention clearly demonstrates that updates can change while the subspaces remain fixed. However, the cross intervention may alter both the subspaces and the spectral weighting structure. Therefore, showing that the cross components reproduce the full response establishes their importance, but does not directly demonstrate that subspace rotation is the primary cause. The paper identifies which perturbation blocks matter, but does not fully disentangle the mechanisms through which they affect the response. This limits its ability to explain the dominant mechanism.
  • The methodology and Figure 1 are difficult to follow, making it hard to see how the conclusions relate to one another. This limits the reader’s ability to assess the contributions and supporting evidence. The narrative and main figures should be reorganized to clarify the approach and the connections among the findings.
Questions for authors
  • Does the attribution difference between PiSSA and MiLoRA persist under actual task gradients? The cross-construction comparison in Section 4.3 primarily uses isotropic gradients, whereas task gradients may exhibit specific alignments with different spectral directions.
  • For initialization differences arising from actual perturbations, can the response changes measured by SplitProbe predict subsequent changes in cumulative updates or task loss? If this has not been established, should Section 4.6 be framed as an existence demonstration that parameterization can affect training direction, rather than evidence that actual perturbations generally have substantial training consequences?
Reviewer_q3foOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how weight perturbations affect subsequent low-rank fine-tuning. The authors decompose the perturbations into different components and use controlled experiments to examine their effects on the direction and magnitude of updates. The topic is interesting, but the presentation is difficult to follow. The novelty, practical implications, and relevance of the fixed-gradient analysis to actual training need clarification.

Critical strengths

The topic is interesting and relevant. Low-rank fine-tuning is an important research area, and understanding how weight perturbations affect gradient-based optimization is a meaningful question.

Critical weaknesses
  1. The writing and organization do not meet the standard expected for an ICLR paper. The presentation is difficult to follow, and the connections between the problem formulation, analysis, and experiments need to be clearer.

  2. The practical implications are unclear. Although the paper analyzes how different perturbation components affect updates, it does not explain how these findings can guide training or how undesirable effects should be addressed.

  3. The contribution needs to be stated more clearly. In particular, the paper should distinguish conclusions that follow directly from the factor-SGD update formula from new insights established by the proposed analysis and experiments.

  4. In Eq. (2), G=∂L/∂WeffG=\partial L/\partial W_{\mathrm{eff}} would generally change when perturbations change WeffW_{\mathrm{eff}}. Since GG can also change, is comparing YFY_F with a fixed GG enough to understand what happens during training?

Questions for authors

see the weakenesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.