Rank Changes What Overlap Means: Dimension-Matched Geometry of LoRA Adapters
Abstract
Subspace overlap is routinely read as evidence that two LoRA adapters share a task-relevant direction. Rank moves the reference point of that measurement: independent q-dimensional subspaces in a d-dimensional module have expected normalized overlap q/d. We call the resulting rank-induced occupancy component "phantom overlap" and measure it with a module-local Haar residual, followed by a marginal-preserving null for pair structure. In a controlled Llama-3.1-8B library, raw overlap rises 186% from ranks 24 to 512 while the Haar residual falls 40%; at three measured ranks, pair-interaction dispersion remains 12.2–13.1× its null expectation. Positive residuals recur across language and vision libraries, model scales, and training seeds, while the rank trajectory remains library-specific. Controlled two-task compositions show source acquisition and positive joint utility at moderate source strength, with target retention and coefficient sensitivity measured separately. A preregistered study over all 28 task pairs separates measurement from selection: dimension matching corrects cross-rank interpretation, whereas composition utility depends on the evaluated operating condition.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.