- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper analyzes LoRA through a "coordinate view," , where shared frozen bases and define the available update directions and only a sparse coordinate matrix is learned and stored per task. Using this framework, the authors study three questions on Qwen2.5-3B and Qwen3.5-2B across six multiple-choice tasks (plus GSM8K, MATH, and MBPP): (i) whether the basis needs model-specific information, finding that normalized random bases perform comparably to pretrained-SVD bases while raw Gaussian bases require learning-rate calibration; (ii) how much coordinate connectivity is needed, finding that wider, sparser supports can match or outperform narrower, denser ones at equal budgets; and (iii) what structure task directions have, showing signed dose responses along trained residuals, useful single-step few-shot gradient directions, and that the top 10% of coordinates retain most of the gain while relocating them removes it. The paper also provides two supporting results characterizing the reachable update space via bipartite matching and interpreting top- masking as best sparse approximation of the projected gradient.
Critical strengths
Separating reconstruction from task learning. The paper shows that the same random basis captures only 0.63% of the LoRA update's Frobenius norm under direct reconstruction, yet reaches 72.74% accuracy when is trained on the task. This contrast clarifies that the success of random bases does not come from approximating a particular LoRA solution, but from finding a different useful update within the reachable space, and it connects naturally to Theorem 1.
Practical implications for parameter budget allocation. The finding that, at the same nominal budget , wider and sparser configurations can outperform narrower and denser ones offers a useful perspective for adapter design. It aligns with Theorem 1's distinction between the dimension of the reachable space () and its maximum attainable rank (the matching number).
Transparent reporting and reproducibility. The authors avoid presenting small gaps as significance claims, explicitly acknowledge that Theorem 2 consists of standard results, and scope their contribution as an analytical perspective rather than a new adapter. Data sizes, learning rates, seeds, and protocol differences are documented in detail in the Appendix, and code listings for the core operations are provided, which supports reproducibility.
Critical weaknesses
Evaluation protocol confounds the interpretation of Part III. Multiple-choice predictions are scored by summed continuation log-probability without length normalization. As a result, base HellaSwag accuracy is 44.10% (Qwen2.5-3B) and 28.63% (Qwen3.5-2B), far below commonly reported values for models of this size.
A single 64-shot gradient step raises ARC-Easy from 71.55% to 91.08% (Table 5), and the responses in Figure 6 look nearly step-like around . This suggests that a substantial part of the identified "task direction" may correspond to a low-dimensional calibration of answer length or format rather than task knowledge. Without re-evaluating with length-normalized accuracy, or analyzing whether the gradient direction mainly shifts length preference, the main claims of Part III cannot be distinguished from this alternative explanation.
Limited novelty. The parameterization is essentially that of LoRA-XS and SVFT, and LoRA-XS already compares random and SVD bases, as the authors acknowledge. Theorems 1 and 2 are elementary linear-algebra results, and the authors themselves describe Theorem 2 as standard.
The findings in Part III also largely confirm known facts: reversing a task vector degrades performance (task arithmetic), following the negative gradient reduces the loss, and magnitude-based top- selection preserves most of an update's effect (TIES, DARE). For a paper whose stated contribution is an analytical perspective, I would expect at least one clear insight that is only visible through this coordinate view, and this is currently missing.
Missing controls in the support analysis. Comparing top- selection against relocation does not separate "which coordinates" from "where they are placed," since relocating values effectively yields a random direction. A random-mask control that keeps a random subset at the original locations (as in DARE, with and without rescaling) is needed.
In addition, the top- directions are evaluated without renormalization, so their norm shrinks, yet accuracy remains nearly unchanged. This may indicate that already lies in a saturated regime. Without top- curves across doses, the meaning of "the top 10% retains approximately 93% of the gain" remains unclear.
Insufficient baselines and single learning-rate comparison. There is no parameter-matched LoRA baseline (e.g., , with roughly 1M parameters), which weakens the claim that fixed-basis methods approach LoRA with about 7% of its parameters. There is also no quantitative comparison with the closest prior methods, such as VeRA, NOLA, LoRA-XS, and SVFT.
Moreover, the main comparison in Table 1 uses a single common learning rate of . Recent work reports that conclusions about LoRA variants can change substantially once learning rates are properly tuned (e.g., arXiv:2602.04998), so a per-method learning-rate sweep would make the comparison considerably more convincing.
Questions for authors
-
It would be helpful to report the results of Table 1 and Part III (Figures 5 and 6, and Table 3) using length-normalized accuracy (acc_norm). In particular, showing how the average length of the selected answers changes as the model moves along the few-shot gradient direction would help distinguish whether the gain reflects task knowledge or a calibration of length bias.
-
In addition to top- selection and relocation, do you have results for a control that randomly keeps the same fraction of coordinates at their original locations (with and without rescaling)? If this control performs similarly to top-, the conclusion that "magnitude matters" would be weakened; if it performs similarly to relocation, the conclusion would be strengthened.
-
It would be helpful to see a comparison with a parameter-matched LoRA (e.g., ), as well as VeRA, LoRA-XS, and SVFT, under the same setting. I am also curious whether the rankings and gaps in Table 1 hold when a learning-rate sweep is applied to each method.
-
What do you consider to be the novelty and main contribution of this paper? Please explicitly explain how it differs from prior work.
2 comments
Response to Reviewer 3
Effect of retained fraction at
| Retained fraction | Top, raw | Random, raw | Relocated Top |
|---|---|---|---|
| 10% | 65.30 | 59.18 ± 0.09 | 53.43 ± 0.09 |
| 20% | 65.98 | 62.80 ± 0.13 | 53.49 ± 0.07 |
| 40% | 66.29 | 65.20 ± 0.08 | 53.41 ± 0.10 |
| 60% | 66.32 | 65.58 ± 0.08 | 53.50 ± 0.34 |
| 80% | 66.35 | 65.79 ± 0.17 | 53.51 ± 0.09 |
Coordinate placement. At 10% support, the same signed coefficients yield 65.30% at their original locations and 53.43% after relocation, close to the 53.57% zero-step baseline. As the retained fraction grows from 10% to 80%, Random improves from 59.18% to 65.79%, while Relocated Top stays near baseline throughout. Thus, random subsets can recover much of the gain when enough original assignments are preserved. Reassigning the selected values removes that gain even at 80% support.
3. How do the fixed-basis variants compare with prior methods?
Table R1 compares our four fixed-basis variants with the base model, rank-16 LoRA, VeRA, NOLA, LoRA-XS, and SVFT. All trained methods use a learning rate of and three random seeds, with no development-set checkpoint selection. Reasoning evaluations use greedy decoding with a 768-token budget.
Table R1. Task-group means (%) under the shared protocol
| Method | Qwen2.5-3B MC | Qwen2.5-3B reasoning | Qwen3.5-2B MC | Qwen3.5-2B reasoning |
|---|---|---|---|---|
| Base (evaluation only) | 66.01 | 47.66 | 53.33 | 52.99 |
| LoRA () | 85.97 | 56.51 | 82.89 | 48.57 |
| VeRA | 74.74 | 48.05 | 70.26 | 49.87 |
| NOLA | 81.79 | 53.91 | 78.88 | 49.61 |
| LoRA-XS | 78.46 | 53.13 | 81.44 | 50.26 |
| SVFT | 81.42 | 54.43 | 78.78 | 50.39 |
| Raw Gaussian | 32.20 | 0.00 | 32.40 | 0.65 |
| Unit Gaussian | 84.55 | 58.33 | 81.71 | 51.17 |
| QR Random | 84.62 | 59.64 | 81.69 | 51.82 |
| SVD() | 84.69 | 57.29 | 81.42 | 51.56 |
Under this protocol, QR Random comes within 1.35 points of rank-16 LoRA on Qwen2.5 MC and 1.20 points on Qwen3.5 MC. Reasoning results depend on the backbone. QR Random reaches 59.64% on Qwen2.5, while all adapted methods score below the base model on Qwen3.5 reasoning.
The approximately 7% parameter figure refers specifically to the rank-16 LoRA comparison. The table establishes performance at the shared learning rate. Rankings under a matched LoRA parameter budget or independently tuned learning rates remain open.
4. What does the coordinate view contribute?
LoRA-XS and SVFT establish fixed-basis parameterizations, and LoRA-XS already compares random and SVD bases. We build on this work by studying how basis choice, direction, and sparsity shape a low-rank update.
Basis choice. An effective basis can be fixed independently of the pretrained weights. Under our shared protocol, QR Random closely matches the weight-derived SVD basis on multiple-choice tasks and scores higher on both reasoning evaluations.
Direction and sparsity. The coordinate view makes a task direction interpretable through direct edits. A positive step along the 64-shot direction improves accuracy, and reversing it lowers accuracy under both scoring rules. Its effect is also concentrated: at , the largest 10% of coordinates preserve 91% of the full accuracy gain. This sparse edit outperforms a norm-matched random 10% subset (65.30% vs. 64.09%). Relocating the same signed values brings accuracy back to baseline (53.43% vs. 53.57%).
These findings suggest a design opportunity for LoRA: choose a basis independently of the pretrained weights, then identify the small subset of coordinates that carries most of a task direction’s effect.
Response to Reviewer 3
Thank you for highlighting two possible explanations for our results: answer-length preferences may contribute to the signed response, and update norm may account for the apparent advantage of selecting the largest coordinates. We address each with a targeted control, then discuss the baseline comparisons and the contribution of the coordinate view.
1. Does the signed response persist under length-normalized scoring?
We rescored the same edited model states using two rules.
Sumadds the log-probabilities of the continuation tokens in each candidate answer.acc_normtakes their mean. Holding the model states fixed lets us test how the choice of scoring rule affects the observed response.64-shot coordinate-gradient curve. We computed the gradient once at initialization using mean-token scores on 64 examples, then varied the signed step size η without further training. Entries are six-task macro-average accuracies (%).
acc_normAt η=+0.5, accuracy improves over the zero-step baseline by 12.87 percentage points under
Sumand 14.98 points underacc_norm. Reversing the step to η=−0.5 lowers accuracy under both rules. The signed response therefore persists when each candidate is scored by its mean token log-probability. The separate question of whether the edit changes the average length of the selected answer remains open.Trained-coordinate oracle curve. We repeated the comparison by scaling the trained displacement as N(α)=αNtrained. Entries are six-task macro-average accuracies (%).
acc_normThe trained direction shows the same sign dependence. At α=1, the two rules give similar accuracies (81.61% and 81.89%) despite their different zero-step baselines (53.57% and 47.93%).
We also tested the trained displacement on ALFWorld with Qwen3.5-4B, where the outcome is episode success. Success was 0.00% at both α=−0.5 and α=0, rose to 29.69% at α=0.5 (23.44% seen, 35.94% unseen), and reached 62.50% at α=1.0 (59.38% seen, 65.63% unseen).
In the revision, we will report the full Table 1 multiple-choice (MC) results under
acc_norm. On Qwen2.5, the Table 1 means are 61.84% for Base, 80.12% for Unit Gaussian, 80.25% for QR Random, and 80.31% for SVD(W0). The Part III curves above use their own zero-step reference.2. How much do magnitude selection and coordinate placement matter?
We added the requested random-mask control, which samples coordinates from an existing update. The random initialization in the original figure serves a different purpose. We compare three edits at the same retained fraction:
Both tables report six-task macro-average
Sumaccuracy (%). For randomized controls, ± denotes one standard deviation across three draws. We match L2 norms in coordinate space.Effect of step size at 10% support
Magnitude selection and norm. At η=0.5, matching norms reduces the gap between Top 10% and Random 10% from 6.12 to 1.21 points. At η=0.125, the norm-matched gap is 4.68 points. The reviewer’s concern about saturation therefore qualifies the original result: the advantage of selecting the largest coefficients depends on step size, and a single gain-retention percentage obscures that dependence.