acceptodds
Under review as a conference paper at ICLR 2027

Rank Changes What Overlap Means: Dimension-Matched Geometry of LoRA Adapters

Abstract

Subspace overlap is routinely read as evidence that two LoRA adapters share a task-relevant direction. Rank moves the reference point of that measurement: independent q-dimensional subspaces in a d-dimensional module have expected normalized overlap q/d. We call the resulting rank-induced occupancy component "phantom overlap" and measure it with a module-local Haar residual, followed by a marginal-preserving null for pair structure. In a controlled Llama-3.1-8B library, raw overlap rises 186% from ranks 24 to 512 while the Haar residual falls 40%; at three measured ranks, pair-interaction dispersion remains 12.2–13.1× its null expectation. Positive residuals recur across language and vision libraries, model scales, and training seeds, while the rank trajectory remains library-specific. Controlled two-task compositions show source acquisition and positive joint utility at moderate source strength, with target retention and coefficient sensitivity measured separately. A preregistered study over all 28 task pairs separates measurement from selection: dimension matching corrects cross-rank interpretation, whereas composition utility depends on the evaluated operating condition.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LoRA adapters · Model merging · Subspace geometry · Principal angles · Measurement calibration · Rank analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JCqNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how LoRA rank affects the interpretation of subspace overlap between independently trained task adapters. The authors introduce phantom overlap, showing that raw overlap naturally increases with rank even for independent random subspaces, and propose a dimension-matched Haar residual to correct this bias. Experiments across multiple model families, scales, and modalities demonstrate that meaningful pairwise geometric structure remains after this correction. The paper further evaluates additive LoRA composition under different ranks and source strengths, showing that moderate source strength can yield positive joint utility. However, the corrected geometric metric does not reliably predict the optimal rank for adapter merging, highlighting the limitations of subspace overlap as a standalone indicator of merging performance.

Critical strengths
  1. The paper shows that raw overlap naturally increases with LoRA rank, potentially leading to misleading interpretations of adapter similarity. The dimension-matched Haar reference provides a simple and theoretically grounded correction.
  2. The paper evaluates multiple model families, scales, ranks, and modalities. The marginal-preserving null analysis further demonstrates that consistent task-pair-specific geometric structure remains after accounting for dimensional occupancy and individual task effects.
Critical weaknesses
  1. The central theoretical result, that the expected overlap between independent random subspaces is q/dq/d, is a well-established property of random subspace geometry. The proposed Haar residual is a relatively straightforward correction, making the technical novelty beyond existing geometric analysis somewhat limited.

  2. Although the related work discusses numerous established model merging approaches, including TIES-Merging, AdaMerging, and various LoRA-specific methods, none of these alternative methods are included as experimental baselines. Instead, the composition experiments are limited to simple additive LoRA merging, making it difficult to assess the practical significance of the reported results relative to existing methods.

  3. Although the Haar residual corrects rank-induced overlap, maximizing it performs worse than a fixed-rank baseline in the rank-selection experiments. The paper therefore does not clearly demonstrate how the proposed geometric measurement can improve practical adapter merging decisions.

  4. Positive Haar residuals and consistent pairwise interactions do not necessarily indicate meaningful task similarity or functional compatibility. Since adapters share the same pretrained model and similar training configurations, further validation is needed to establish whether the observed geometric structure reflects task-specific relationships rather than shared optimization biases.

Questions for authors

The authors are encouraged to address the concerns raised in the Weaknesses section, particularly regarding technical novelty, comparisons with existing merging methods, practical utility, and the interpretation of task-specific geometric structure.

Reviewer_SzcAOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

I did not understand the paper well enough to produce a summary.

Critical strengths

Although I did not understand the paper, I appreciate the use of statistical significance tests, confidence intervals, and I found the figures visually pleasing.

Critical weaknesses

Despite persistent efforts, I found this paper extremely difficult to understand. I am not sure if this is because not at all familiar with the research area and/or the paper's fault. It sounds 100% AI generated from beginning to finish. The abstract and introduction are very unclear and do not provide an explanation of why this problem is important and what implications their work could have.

Questions for authors

/

Reviewer_a3ZyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper shows that LoRA subspace overlap can increase mechanically with rank. It introduces a module-local Haar correction based on effective rank and output dimension, followed by a marginal-preserving null test of task-pair structure across modules. Experiments across language and vision libraries find recurring positive residuals, while composition studies show that corrected geometry does not automatically provide an effective rank-selection rule.

Critical strengths
  1. The measurement problem is clear and convincing. Section 4 derives the q/d baseline for equal ranks and extends it to unequal effective ranks.
  2. The analysis goes beyond subtracting a random baseline. Section 5.2 removes additive task marginals before permuting task labels within modules. The interaction dispersion in Table 2 and consistency across module splits provide additional evidence that task-pair structure recurs across modules.
Critical weaknesses
  1. The correction removes dimensional occupancy but not systematic orientation preferences induced by a shared base, initialization, or optimization. Although the paper acknowledges these factors, the current results do not quantify how much of the residual represents task-related structure. A positive residual therefore cannot directly be interpreted as reusable capability.
  2. Independent relabeling within each module destroys task-pair patterns shared across layers, so the 12–13-fold separation primarily establishes cross-module recurrence. It does not independently establish functional compatibility, and the effect of shared training mechanisms across layers on the null distribution's scale is not fully addressed. A sensitivity analysis using permutations that preserve correlations within layer groups would help.
  3. The coefficient comparison mainly uses four task pairs with 128 examples per side and a single exact additive operator. The reported exact sign-test p-value of 0.375 means that the advantage of moderate coefficients remains primarily descriptive. The 28-pair selection study also does not extend to other model families or composition operators.
Questions for authors

see in the weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.