acceptodds
Under review as a conference paper at ICLR 2027

Beyond Adapter Rank: The Shared-Subspace Complexity of LoRA Training

Abstract

Low-rank adaptation constrains each weight update, but its training checkpoints need not share a small input subspace. This distinction matters whenever one fixed representation must describe a training history or preserve its predictions. We study the minimum shared input dimension of a LoRA path and how the weighting of checkpoints affects its approximation. In aligned linear regression, simultaneous gradient descent produces rank-one checkpoints whose shared hidden input space has dimension , even though the initialization space already contains a solution. We also derive the exact minimax error for shared linear measurements when the observer does not know the probe orientation. Spectral approximation, finite-horizon bounds, and a conditional network perturbation bound connect these results to measurable interventions. Across 24 language-model fine-tuning runs, paths move substantially outside the initial input space, yet a few shared directions retain 99% of their energy. Joint projections across adapted matrices approximately preserve aggregate prediction quality in language modeling and arithmetic reasoning, and outperform random directions at the same dimension. Similar aggregate accuracy nevertheless coexists with changes in individual decisions. These results distinguish instantaneous rank, shared path dimension, and predictive fidelity as separate properties of low-rank training.

Primary area:
optimization
Keywords:
Low-rank adaptation, Training dynamics, Optimization geometry

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wPMZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper distinguishes the rank of individual LoRA updates from the dimension of a shared input subspace covering their training trajectory, and from the preservation of predictions after projection. In a linear-regression construction, gradient descent produces rank-one checkpoints whose shared hidden input space has dimension d−1, even though the initialization space already contains a solution. The paper also analyzes shared measurements and spectral approximation. Experiments across 24 language-model fine-tuning runs examine trajectory energy and the effects of shared-subspace projections on aggregate performance and individual predictions.

Critical strengths
  1. The paper clearly separates instantaneous rank, shared trajectory dimension, and predictive fidelity. The gradient-descent construction is a nontrivial formal demonstration of their distinction, rather than an example assembled from unrelated checkpoints.

  2. The empirical analysis connects the geometric quantities to actual model predictions through projection interventions and controls. The paper also acknowledges the limits of its conclusions, including the difference between preserving aggregate performance and preserving individual decisions.

Critical weaknesses
  1. Limited significance of the theoretical and empirical insights. The distinction between checkpoint rank and trajectory-wide subspace dimension is useful, and I acknowledge the technical value of establishing it under gradient descent. Nevertheless, the underlying possibility—that low-rank updates can change direction during training—is intuitive. I am not convinced that the formal result yields sufficiently deep or consequential new understanding beyond establishing this possibility. Similarly, the findings that trajectory energy concentrates in a few principal directions and that preserving most matrix energy need not preserve predictions are informative, but do not substantially change my understanding of LoRA.

  2. Unclear practical implications. The paper does not sufficiently explain when these analyses would change a concrete decision about LoRA training or deployment. It does not demonstrate improvements in training efficiency, parameter compression, or another practical application, and the value of analyzing the full trajectory rather than the final checkpoint remains unclear. An analysis paper need not introduce a new algorithm, but its conceptual or practical consequences should be sufficiently compelling. Despite the formal and empirical value of the work, I do not find that level of significance established here, which motivates my clear-rejection recommendation.

Questions for authors
  1. Which conclusion from the gradient-descent analysis provides the most consequential insight beyond the intuitive distinction between instantaneous rank and trajectory-wide subspace dimension?

  2. What evidence shows that analyzing the training trajectory, rather than the final checkpoint, provides information that changes a useful decision about LoRA training, compression, or preservation of model behavior?

These are clarification questions about the significance of the existing results; I am not requiring the paper to introduce a new training algorithm.

Reviewer_79zhOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This work examines how LoRA explores the input data space over the course of training by considering linear regression, where the weight matrix is replaced by BA. It observes that at any one time step, the rank of A restricts the input dimension. However, over the course of training, the iterates may explore a larger input space of dimension d−1d-1, as long as specific conditions hold on the initialization and covariance. The authors also prove some guarantees about whether restricting the input space affects the model error. Experiments are performed with AdamW and large language models, demonstrating that the iterates explore beyond the initial input space, but mostly remain in a few shared directions.

Critical strengths

The observation is interesting and somewhat novel.

Critical weaknesses

I found the writing dense and confusing to read. In the current state, I do not believe this manuscript is publishable. For example, the following paragraph is impenetrable:

"For absolutely continuous paths this measure is invariant to admissible order-preserving changes of traversal speed. Repeated traversals contribute separately to this measure. A constant path has no normalized arc measure; we use its single-state spectrum by convention. From the saved factors, we integrate the piecewise-linear matrix path exactly. This interpolated path approximates the underlying training trajectory. The arc measure is defined separately for the total operator, hidden operator, and each token-response path."

Some additional weaknesses are as follows.

  1. The linear regression setting is highly idealized and does not really inform the general behavior of LoRA. The conclusions for this setting rely on the fact that A is directly applied to the input data. For a neural network, this would only be true for the input layer weights, and says nothing about LoRA reparameterization of other weights. This makes me wonder if the conclusions are even related to the LoRA structure, or if it is just an observation of what happens when the weights project the input data down into a smaller space. For example, suppose we want to minimize ∥Ax−y∥2\lVert A x - y \rVert^2 or ∥f(B)Ax−y∥2\lVert f(B)A x - y \rVert^2 where AA is 1×d1 \times d, and ff is some arbitrary function. Then can one show that with gradient descent, AA could also traverse through d−1d-1 dimensions? My understanding is that this would produce the same result.

  2. It's unclear whether the observations of this paper are genuinely useful. While the statements are true, it seems to be a fairly obvious fact that as the weights change, their exploration of the input space will also change beyond the initial subspace. What implications does this work have for practical implementations of LoRA?

  3. My understanding is that section 4 demonstrates that restricting the subspace increases the model error. However, why would we want to restrict the subspace of the trajectory? This result does not seem to contribute any new insights.

Questions for authors

See questions above.

Reviewer_1Ag8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the input subspace actually traversed by LoRA adapters during training, rather than the rank of a single checkpoint. The authors define endpoint rank and pathwise complexity, and prove that ordinary gradient descent can produce a training path whose individual checkpoints are all rank one yet whose shared hidden input space requires d−1d-1 dimensions (Theorem 1). They further connect the path spectrum to prediction loss: they provide recovery lower bounds, conditions for spectral concentration, and sufficient conditions linking matrix residuals to teacher-forced NLL. Experiments on Qwen2.5-1.5B and Pythia-1.4B measure 24 LoRA runs, reporting that the hidden-to-total path norm reaches 81.5%−86.4%81.5\%-86.4\%, while only 1.67−31.67-3 shared directions retain 99%99\% of hidden path energy. Joint projection interventions compare full-path, endpoint, and random subspaces on language modeling and GSM8K.

Critical strengths
  • The rank-one trajectory construction in Theorem 1. It proves that even when the initial input space already contains a solution and every checkpoint is rank one, gradient descent can make the shared hidden path reach d−1d-1 dimensions. This directly supports the central claim that adapter rank cannot describe the training history, rather than relying on arbitrarily chosen rotating matrices.
  • The full chain from path spectrum to prediction interventions. From the spectral characterization in Proposition 1, to the recovery frontier in Theorem 3, to the path-weighted NLL bound in Corollary 3, the paper does not stop at matrix approximation but attempts to connect geometric quantities to model behavior. The joint intervention experiments (full-path vs endpoint vs random) also provide functional evidence aligned with the theory.
Critical weaknesses
  • The empirical advantage of path directions over endpoint directions is very small, and the comparison budget may be too tight. In Table 3, the NLL reduction of full-path over endpoint is only 0.000045−0.0012400.000045-0.001240 nats/token. The paper itself notes that endpoint directions can reconstruct the final update exactly at budget k=rk=r, whereas full-path directions are not exact at the final checkpoint. Since the experiments use k=rk=r, while path k99k_{99} is 2−32-3 in several settings and k99.9k_{99.9} can reach 9−109-10, this budget may disadvantage both fits. It is difficult for readers to judge whether the small full-path advantage reflects genuine path geometry or is an artifact of budget choice.
  • The random-subspace control confounds “direction selection” with “retained energy.” The paper claims the random control tests whether dominant directions matter “at the same per-module dimension,” but Table 4 shows that random subspaces retain only 1.31%−3.26%1.31\%-3.26\% of pooled path energy, whereas principal directions retain 99.35%−99.66%99.35\%-99.66\%. The worse NLL of random subspaces therefore cannot be attributed solely to direction selection; it may simply reflect retaining far less path energy. A random or alternative control that retains the same energy would be cleaner.
  • The generation experiments show substantial answer flips, which is in tension with the phrasing “retaining average prediction quality.” Tables 2 and 6 show that after 99%99\%-energy projection, 40%−44%40\%-44\% of complete generated sequences change, and in one rank-four run 53 correct answers become wrong while 53 wrong answers become correct, leaving accuracy nearly unchanged. The paper correctly notes that aggregate accuracy and answer agreement answer different questions, but this also means that the claim that dominant directions “retain predictive behavior” needs to be stated more cautiously: stable average loss and average accuracy do not equal predictive fidelity.
  • The bridge from theory to prediction loss relies on unestimated sensitivity constants. The bound in Corollary 3 contains downstream Lipschitz constants LjL_j and activation second moments βj\beta_j, and the paper explicitly states that the experiments do not estimate these constants. The bound therefore provides conditions for loss control rather than a numerical certificate. The main text should more clearly distinguish “geometric upper bounds” from “empirically observed loss changes.”
Questions for authors
  1. For the full-path vs endpoint comparison, can the authors add results at equal retained energy rather than equal k=rk=r? Under the current k=rk=r budget, endpoint directions can reconstruct the final update exactly, while path k99k_{99} is often larger than rr. Does this systematically affect the comparison?
  2. Can the random-subspace control include a random or alternative subspace that retains the same path energy as the principal directions? The current random control retains only 1.31%−3.26%1.31\%-3.26\% energy, making it difficult to separate direction selection from energy retention.
  3. Do the 40%−44%40\%-44\% answer flips in the generation experiments imply that dominant path directions mainly preserve average loss rather than individual predictions? How do the authors define the scope of “prediction fidelity” under this result?
  4. Can the constants LjL_j and βj\beta_j in Corollary 3 be estimated on small models or fixed batches, so that the geometric bound can be converted into a testable numerical prediction?
  5. Theorem 1 concerns fixed-feature regression with ordinary gradient descent. Do the authors expect this phenomenon to appear as “high-rank transients” in practical AdamW + language-model training? What is the relationship between the transient hidden response in Figure 3 and the theorem’s construction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.