- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how LoRA rank affects the interpretation of subspace overlap between independently trained task adapters. The authors introduce phantom overlap, showing that raw overlap naturally increases with rank even for independent random subspaces, and propose a dimension-matched Haar residual to correct this bias. Experiments across multiple model families, scales, and modalities demonstrate that meaningful pairwise geometric structure remains after this correction. The paper further evaluates additive LoRA composition under different ranks and source strengths, showing that moderate source strength can yield positive joint utility. However, the corrected geometric metric does not reliably predict the optimal rank for adapter merging, highlighting the limitations of subspace overlap as a standalone indicator of merging performance.
Critical strengths
- The paper shows that raw overlap naturally increases with LoRA rank, potentially leading to misleading interpretations of adapter similarity. The dimension-matched Haar reference provides a simple and theoretically grounded correction.
- The paper evaluates multiple model families, scales, ranks, and modalities. The marginal-preserving null analysis further demonstrates that consistent task-pair-specific geometric structure remains after accounting for dimensional occupancy and individual task effects.
Critical weaknesses
-
The central theoretical result, that the expected overlap between independent random subspaces is , is a well-established property of random subspace geometry. The proposed Haar residual is a relatively straightforward correction, making the technical novelty beyond existing geometric analysis somewhat limited.
-
Although the related work discusses numerous established model merging approaches, including TIES-Merging, AdaMerging, and various LoRA-specific methods, none of these alternative methods are included as experimental baselines. Instead, the composition experiments are limited to simple additive LoRA merging, making it difficult to assess the practical significance of the reported results relative to existing methods.
-
Although the Haar residual corrects rank-induced overlap, maximizing it performs worse than a fixed-rank baseline in the rank-selection experiments. The paper therefore does not clearly demonstrate how the proposed geometric measurement can improve practical adapter merging decisions.
-
Positive Haar residuals and consistent pairwise interactions do not necessarily indicate meaningful task similarity or functional compatibility. Since adapters share the same pretrained model and similar training configurations, further validation is needed to establish whether the observed geometric structure reflects task-specific relationships rather than shared optimization biases.
Questions for authors
The authors are encouraged to address the concerns raised in the Weaknesses section, particularly regarding technical novelty, comparisons with existing merging methods, practical utility, and the interpretation of task-specific geometric structure.