acceptodds
Under review as a conference paper at ICLR 2027

A COORDINATE VIEW OF LORA: FIXED BASES AND LEARNED COMBINATIONS

Abstract

Storing fewer parameters per task makes it less costly to adapt one model to many tasks. Understanding those parameters could also help us identify and modify the changes that affect task performance. Methods with fixed factors already reduce task-specific parameters, but their performance depends jointly on the basis and its learned coefficients. We separate basis information, scale, and coefficient budgets, then study the effects of changing the coefficients. The coordinate view , with shared bases and task-specific coordinates , provides a common framework for analyzing basis information, connectivity, and task directions. Normalized random bases closely approach pretrained-SVD performance, while learning-rate calibration substantially recovers raw Gaussian performance. At comparable coordinate budgets, wider sparse systems can outperform narrower dense ones, with connectivity effects also depending on update scale. Scaling task residuals forward or backward produces structured changes in accuracy; a single few-shot gradient also identifies beneficial directions from weak starting states. Across six tasks, the largest 10% of a 64-example direction's coordinates retain approximately 93% of its accuracy gain from zero and randomized starts; relocating the same values largely removes the benefit. These findings connect compact task-specific storage with interpretable, direction- and location-dependent changes in task performance.

Primary area:
interpretability and explainable AI
Keywords:
LoRA, Coordinate space, Sparse adaptation, Task directions

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8e91OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes LoRA through a "coordinate view," ΔW=sANB\Delta W = sANB, where shared frozen bases AA and BB define the available update directions and only a sparse coordinate matrix NN is learned and stored per task. Using this framework, the authors study three questions on Qwen2.5-3B and Qwen3.5-2B across six multiple-choice tasks (plus GSM8K, MATH, and MBPP): (i) whether the basis needs model-specific information, finding that normalized random bases perform comparably to pretrained-SVD bases while raw Gaussian bases require learning-rate calibration; (ii) how much coordinate connectivity is needed, finding that wider, sparser supports can match or outperform narrower, denser ones at equal budgets; and (iii) what structure task directions have, showing signed dose responses along trained residuals, useful single-step few-shot gradient directions, and that the top 10% of coordinates retain most of the gain while relocating them removes it. The paper also provides two supporting results characterizing the reachable update space via bipartite matching and interpreting top-kk masking as best sparse approximation of the projected gradient.

Critical strengths

Separating reconstruction from task learning. The paper shows that the same random basis captures only 0.63% of the LoRA update's Frobenius norm under direct reconstruction, yet reaches 72.74% accuracy when NN is trained on the task. This contrast clarifies that the success of random bases does not come from approximating a particular LoRA solution, but from finding a different useful update within the reachable space, and it connects naturally to Theorem 1.

Practical implications for parameter budget allocation. The finding that, at the same nominal budget RdRd, wider and sparser configurations can outperform narrower and denser ones offers a useful perspective for adapter design. It aligns with Theorem 1's distinction between the dimension of the reachable space (∣S∣\left|S\right|) and its maximum attainable rank (the matching number).

Transparent reporting and reproducibility. The authors avoid presenting small gaps as significance claims, explicitly acknowledge that Theorem 2 consists of standard results, and scope their contribution as an analytical perspective rather than a new adapter. Data sizes, learning rates, seeds, and protocol differences are documented in detail in the Appendix, and code listings for the core operations are provided, which supports reproducibility.

Critical weaknesses

Evaluation protocol confounds the interpretation of Part III. Multiple-choice predictions are scored by summed continuation log-probability without length normalization. As a result, base HellaSwag accuracy is 44.10% (Qwen2.5-3B) and 28.63% (Qwen3.5-2B), far below commonly reported values for models of this size.

A single 64-shot gradient step raises ARC-Easy from 71.55% to 91.08% (Table 5), and the responses in Figure 6 look nearly step-like around η≈0\eta\approx 0. This suggests that a substantial part of the identified "task direction" may correspond to a low-dimensional calibration of answer length or format rather than task knowledge. Without re-evaluating with length-normalized accuracy, or analyzing whether the gradient direction mainly shifts length preference, the main claims of Part III cannot be distinguished from this alternative explanation.


Limited novelty. The ANBANB parameterization is essentially that of LoRA-XS and SVFT, and LoRA-XS already compares random and SVD bases, as the authors acknowledge. Theorems 1 and 2 are elementary linear-algebra results, and the authors themselves describe Theorem 2 as standard.

The findings in Part III also largely confirm known facts: reversing a task vector degrades performance (task arithmetic), following the negative gradient reduces the loss, and magnitude-based top-kk selection preserves most of an update's effect (TIES, DARE). For a paper whose stated contribution is an analytical perspective, I would expect at least one clear insight that is only visible through this coordinate view, and this is currently missing.


Missing controls in the support analysis. Comparing top-kk selection against relocation does not separate "which coordinates" from "where they are placed," since relocating values effectively yields a random direction. A random-mask control that keeps a random subset at the original locations (as in DARE, with and without rescaling) is needed.

In addition, the top-kk directions are evaluated without renormalization, so their norm shrinks, yet accuracy remains nearly unchanged. This may indicate that η=0.5\eta=0.5 already lies in a saturated regime. Without top-kk curves across doses, the meaning of "the top 10% retains approximately 93% of the gain" remains unclear.


Insufficient baselines and single learning-rate comparison. There is no parameter-matched LoRA baseline (e.g., r=1r=1, with roughly 1M parameters), which weakens the claim that fixed-basis methods approach LoRA with about 7% of its parameters. There is also no quantitative comparison with the closest prior methods, such as VeRA, NOLA, LoRA-XS, and SVFT.

Moreover, the main comparison in Table 1 uses a single common learning rate of 2×10−42\times 10^{-4}. Recent work reports that conclusions about LoRA variants can change substantially once learning rates are properly tuned (e.g., arXiv:2602.04998), so a per-method learning-rate sweep would make the comparison considerably more convincing.

Questions for authors
  1. It would be helpful to report the results of Table 1 and Part III (Figures 5 and 6, and Table 3) using length-normalized accuracy (acc_norm). In particular, showing how the average length of the selected answers changes as the model moves along the few-shot gradient direction would help distinguish whether the gain reflects task knowledge or a calibration of length bias.

  2. In addition to top-kk selection and relocation, do you have results for a control that randomly keeps the same fraction of coordinates at their original locations (with and without rescaling)? If this control performs similarly to top-kk, the conclusion that "magnitude matters" would be weakened; if it performs similarly to relocation, the conclusion would be strengthened.

  3. It would be helpful to see a comparison with a parameter-matched LoRA (e.g., r=1r=1), as well as VeRA, LoRA-XS, and SVFT, under the same setting. I am also curious whether the rankings and gaps in Table 1 hold when a learning-rate sweep is applied to each method.

  4. What do you consider to be the novelty and main contribution of this paper? Please explicitly explain how it differs from prior work.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 3

Thank you for highlighting two possible explanations for our results: answer-length preferences may contribute to the signed response, and update norm may account for the apparent advantage of selecting the largest coordinates. We address each with a targeted control, then discuss the baseline comparisons and the contribution of the coordinate view.

1. Does the signed response persist under length-normalized scoring?

We rescored the same edited model states using two rules. Sum adds the log-probabilities of the continuation tokens in each candidate answer. acc_norm takes their mean. Holding the model states fixed lets us test how the choice of scoring rule affects the observed response.

64-shot coordinate-gradient curve. We computed the gradient once at initialization using mean-token scores on 64 examples, then varied the signed step size η\eta without further training. Entries are six-task macro-average accuracies (%).

Step size η\eta−1.0−0.75−0.5−0.25−0.1250.0+0.125+0.25+0.5+0.75+1.0
Sum36.8137.4138.6939.5941.5053.5763.2265.4766.4465.2063.20
acc_norm36.5936.6837.2938.2839.5147.9359.4861.8162.9161.6360.32

At η=+0.5\eta=+0.5, accuracy improves over the zero-step baseline by 12.87 percentage points under Sum and 14.98 points under acc_norm. Reversing the step to η=−0.5\eta=-0.5 lowers accuracy under both rules. The signed response therefore persists when each candidate is scored by its mean token log-probability. The separate question of whether the edit changes the average length of the selected answer remains open.

Trained-coordinate oracle curve. We repeated the comparison by scaling the trained displacement as N(α)=αNtrainedN(\alpha)=\alpha N_{\mathrm{trained}}. Entries are six-task macro-average accuracies (%).

Scale α\alpha−1.0−0.75−0.5−0.250.0+0.125+0.25+0.5+0.75+1.0+1.25+1.5+2.0
Sum35.0135.7037.3539.5653.5766.0573.9779.5581.2081.6181.8081.2579.70
acc_norm36.2236.6537.0138.3647.9363.4273.1779.6781.1681.8981.7081.2779.74

The trained direction shows the same sign dependence. At α=1\alpha=1, the two rules give similar accuracies (81.61% and 81.89%) despite their different zero-step baselines (53.57% and 47.93%).

We also tested the trained displacement on ALFWorld with Qwen3.5-4B, where the outcome is episode success. Success was 0.00% at both α=−0.5\alpha=-0.5 and α=0\alpha=0, rose to 29.69% at α=0.5\alpha=0.5 (23.44% seen, 35.94% unseen), and reached 62.50% at α=1.0\alpha=1.0 (59.38% seen, 65.63% unseen).

In the revision, we will report the full Table 1 multiple-choice (MC) results under acc_norm. On Qwen2.5, the Table 1 means are 61.84% for Base, 80.12% for Unit Gaussian, 80.25% for QR Random, and 80.31% for SVD(W0W_0). The Part III curves above use their own zero-step reference.

2. How much do magnitude selection and coordinate placement matter?

We added the requested random-mask control, which samples coordinates from an existing update. The random initialization in the original figure serves a different purpose. We compare three edits at the same retained fraction:

  • Top: Keep the coefficients with the largest absolute values at their original coordinates.
  • Random: Keep a uniformly sampled subset of coefficients at their original coordinates.
  • Relocated Top: Take exactly the same signed coefficients as Top and randomly reassign them to different coordinates, preserving their values and norm.

Both tables report six-task macro-average Sum accuracy (%). For randomized controls, ±\pm denotes one standard deviation across three draws. We match L2 norms in coordinate space.

Effect of step size at 10% support

Editη=0.125\eta=0.125η=0.25\eta=0.25η=0.5\eta=0.5
Full update63.2265.4766.44
Top 10%, raw62.3964.7065.30
Top 10%, norm matched to Full62.7064.8465.41
Random 10%, raw55.14 ± 0.1656.61 ± 0.2159.18 ± 0.09
Random 10%, norm matched to raw Top 10%57.71 ± 0.0961.13 ± 0.1064.09 ± 0.12
Relocated Top 10%53.50 ± 0.0853.37 ± 0.0953.43 ± 0.09

Magnitude selection and norm. At η=0.5\eta=0.5, matching norms reduces the gap between Top 10% and Random 10% from 6.12 to 1.21 points. At η=0.125\eta=0.125, the norm-matched gap is 4.68 points. The reviewer’s concern about saturation therefore qualifies the original result: the advantage of selecting the largest coefficients depends on step size, and a single gain-retention percentage obscures that dependence.

AuthorsOpenReview ↗

Response to Reviewer 3

Effect of retained fraction at η=0.5\eta=0.5

Retained fractionTop, rawRandom, rawRelocated Top
10%65.3059.18 ± 0.0953.43 ± 0.09
20%65.9862.80 ± 0.1353.49 ± 0.07
40%66.2965.20 ± 0.0853.41 ± 0.10
60%66.3265.58 ± 0.0853.50 ± 0.34
80%66.3565.79 ± 0.1753.51 ± 0.09

Coordinate placement. At 10% support, the same signed coefficients yield 65.30% at their original locations and 53.43% after relocation, close to the 53.57% zero-step baseline. As the retained fraction grows from 10% to 80%, Random improves from 59.18% to 65.79%, while Relocated Top stays near baseline throughout. Thus, random subsets can recover much of the gain when enough original assignments are preserved. Reassigning the selected values removes that gain even at 80% support.

3. How do the fixed-basis variants compare with prior methods?

Table R1 compares our four fixed-basis variants with the base model, rank-16 LoRA, VeRA, NOLA, LoRA-XS, and SVFT. All trained methods use a learning rate of 2×10−42\times10^{-4} and three random seeds, with no development-set checkpoint selection. Reasoning evaluations use greedy decoding with a 768-token budget.

Table R1. Task-group means (%) under the shared protocol

MethodQwen2.5-3B MCQwen2.5-3B reasoningQwen3.5-2B MCQwen3.5-2B reasoning
Base (evaluation only)66.0147.6653.3352.99
LoRA (r=16r=16)85.9756.5182.8948.57
VeRA74.7448.0570.2649.87
NOLA81.7953.9178.8849.61
LoRA-XS78.4653.1381.4450.26
SVFT81.4254.4378.7850.39
Raw Gaussian32.200.0032.400.65
Unit Gaussian84.5558.3381.7151.17
QR Random84.6259.6481.6951.82
SVD(W0W_0)84.6957.2981.4251.56

Under this protocol, QR Random comes within 1.35 points of rank-16 LoRA on Qwen2.5 MC and 1.20 points on Qwen3.5 MC. Reasoning results depend on the backbone. QR Random reaches 59.64% on Qwen2.5, while all adapted methods score below the base model on Qwen3.5 reasoning.

The approximately 7% parameter figure refers specifically to the rank-16 LoRA comparison. The table establishes performance at the shared learning rate. Rankings under a matched LoRA parameter budget or independently tuned learning rates remain open.

4. What does the coordinate view contribute?

LoRA-XS and SVFT establish fixed-basis parameterizations, and LoRA-XS already compares random and SVD bases. We build on this work by studying how basis choice, direction, and sparsity shape a low-rank update.

Basis choice. An effective basis can be fixed independently of the pretrained weights. Under our shared protocol, QR Random closely matches the weight-derived SVD basis on multiple-choice tasks and scores higher on both reasoning evaluations.

Direction and sparsity. The coordinate view makes a task direction interpretable through direct edits. A positive step along the 64-shot direction improves accuracy, and reversing it lowers accuracy under both scoring rules. Its effect is also concentrated: at η=0.5\eta=0.5, the largest 10% of coordinates preserve 91% of the full accuracy gain. This sparse edit outperforms a norm-matched random 10% subset (65.30% vs. 64.09%). Relocating the same signed values brings accuracy back to baseline (53.43% vs. 53.57%).

These findings suggest a design opportunity for LoRA: choose a basis independently of the pretrained weights, then identify the small subset of coordinates that carries most of a task direction’s effect.

Reviewer_ZXL1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Fixed-basis adapters freeze the outer LoRA factors and learn only a small coefficient matrix, but it is unclear which design choice matters. The paper writes them all as ΔW = A N B and varies basis source, basis width, and coefficient sparsity one at a time on two Qwen backbones. It reports that normalized random bases match SVD bases and come close to LoRA at 7% of its parameters, that sparse coordinates can beat dense ones at the same budget, and that a single few-shot gradient finds a useful direction whose effect sits in 10% of its coordinates.

Critical strengths
  1. The paper is well written and the notation is clean. Separating basis source, basis width, and support is a useful way to place the fixed-basis adapter literature on one axis.
  2. The evaluation covers two backbones, nine tasks, four basis types, and three kinds of coordinate edits.
Critical weaknesses
  1. Two of the three main insights are not supported by the paper's own controls. The paper claims that basis normalization matters. The evidence for that would be normalized bases beating raw Gaussian when each gets its best learning rate, but Figure 3 shows raw Gaussian reaching 82.42 at a smaller rate, at or above the normalized bases. The paper also claims that sparse coordinates can beat dense ones. The evidence for that would be the dense penalty surviving a matched update scale, but the sqrt(32/d) control gives almost flat scores across degrees. Both insights therefore read as learning rate effects rather than structural ones.

  2. CoSA (arXiv 2602.05148) shows that a normalized random basis with a trainable core matches pretrained-SVD bases and LoRA, while LoRA-One (ICML 2025) shows that a single gradient from a batch of 8 examples already gives a useful direction that beats the pretrained model. Could the authors provide a clear distinction from these works?

  3. Appendix C.4 says the Qwen3.5 Unit, QR, and SVD MATH entries were revised with one epoch, LR 2e-6, and dev-based checkpoint selection, while the original protocol used LR 5e-5 and 4 to 12 epochs, and that is where Unit is named strongest (49.22 against LoRA's 36.72, below the 43.36 base).

Questions for authors
  1. If each basis and each degree is given its own tuned learning rate, or the sqrt(32/d) scale correction is applied to the (512,32) versus (128,128) comparison, do the normalization and sparsity gaps remain?
  2. What do the normalization and few-shot findings show beyond CoSA and LoRA-One?
  3. Was LoRA re-run under the revised MATH protocol, and what are all five methods under one protocol?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 2

Thank you for raising three questions about our analysis: how scale affects the comparison between adapter designs, what our gradient intervention adds to prior work, and whether the reasoning results hold under a common protocol. We address each below.

1. How do normalization, connectivity, and adapter gain affect performance?

Basis normalization changes the learning rate that works well. In Figure 3, the raw Gaussian basis reaches 82.42% on a development subset when trained with a much smaller learning rate. Its poor performance at LR = 2e-4 in Table R1 therefore reflects sensitivity to scale. The held-out test results support a specific design implication: at this common learning rate, a normalized random basis performs comparably to a basis derived from the pretrained weights by SVD. The ranking after tuning each method separately remains open.

The effect of connectivity depends on the adapter gain. We repeated the Figure 4 grid with seed = 43. Here, RR is the basis width, dd is the connectivity degree, and the gain is the multiplier applied to the adapter update. We trained each configuration independently from N=0N=0 for one epoch at LR = 2e-4, keeping the gain fixed at α/R=2\alpha/R=2.

Six-task test means (%) with fixed gain, seed 43.

Basis width RRd=1d=1d=4d=4d=8d=8d=16d=16d=32d=32d=64d=64d=128d=128d=256d=256d=512d=512
128——77.4779.2680.3781.4881.27——
25670.9577.5179.4180.8982.1482.1180.2075.74—
512———82.0382.0780.1877.01—45.61

Dashes indicate configurations outside the main grid. Two additional configurations with the same nominal coordinate budget scored 80.35% at (R,d)=(64,64)(R,d)=(64,64) and 80.50% at (512,8)(512,8).

At R=256R=256, the six-task mean rises from 70.95% at d=1d=1 to 82.14% at d=32d=32, then falls to 75.74% at full connectivity (d=256d=256). To test how much gain contributes to the drop at high connectivity, we compared the fixed gain of 22 with a degree-dependent gain of 232/d2\sqrt{32/d} on three tasks.

Three-task means (%) on ARC-Challenge, HellaSwag, and WinoGrande at R=256R=256.

Degree ddScale factor 32/d\sqrt{32/d}Fixed gainDegree-dependent gainDifference (pp)
321.00077.7878.13+0.35
640.70777.7278.23+0.52
1280.50077.0977.74+0.65
2560.35471.5977.47+5.88

The largest change occurs at full connectivity: lowering the gain raises the three-task mean from 71.59% to 77.47%, close to the 78.13% observed at d=32d=32. At d=32d=32, both formulas give the same gain, so the 0.35-point difference between runs cannot be attributed to rescaling. Additional checks using configuration-specific learning rates or matched update norms also narrow the gap, with a score of 78.42% at d=256d=256. For LoRA design, the practical lesson is to check gain and learning rate before attributing a performance difference to sparsity itself.

2. What does the intervention add beyond CoSA and LoRA-One?

CoSA provides a close precedent for fixed random bases with a trainable core. LoRA-One uses a gradient with respect to the full weight matrix to construct a LoRA initialization. Our contribution is a controlled comparison of what different basis and coordinate choices allow the adapter to learn.

Our intervention asks a specific question: with the basis fixed, how useful is the direction suggested by a single gradient in coordinate space? At initialization, we compute one gradient with respect to the coordinate matrix NN. We then evaluate the same direction at positive and negative scales without further training. The oracle experiment answers a separate question: it uses the displacement from the initial NN to a fully trained NN to measure the effect of moving along a direction already found by training. This distinction makes clear which intervention uses information available at initialization and which requires a trained endpoint.

AuthorsOpenReview ↗

Response to Reviewer 2

3. Do the reasoning results hold under a common protocol?

We inadvertently compared MATH scores from two different training and evaluation setups (49.22% for Unit Gaussian and 36.72% for LoRA). We thank the reviewer for pointing this out. We have corrected the comparison by re-evaluating all methods under the same conditions within each model and task.

All trained methods use the same task-specific GPT training data and learning rate (2e-4), with no dev-set checkpoint selection. For reasoning, we train Qwen2.5 for one epoch and Qwen3.5 for four. We use greedy decoding with a 768-token limit and report means over three seeds. VeRA, NOLA, LoRA-XS, and SVFT are approximately matched to our variants in trainable parameter count. LoRA (r=16r=16) provides a higher-budget reference.

Table R1. Mean test scores (%) under the shared protocol. MC averages six multiple-choice tasks. Reasoning averages GSM8K, MATH, and MBPP.

MethodQwen2.5 MCQwen2.5 ReasoningQwen3.5 MCQwen3.5 Reasoning
Base (evaluation only)66.0147.6653.3352.99
LoRA (r=16r=16, higher budget)85.9756.5182.8948.57
VeRA74.7448.0570.2649.87
NOLA81.7953.9178.8849.61
LoRA-XS78.4653.1381.4450.26
SVFT81.4254.4378.7850.39
Raw Gaussian (ours)32.200.0032.400.65
Unit Gaussian (ours)84.5558.3381.7151.17
QR Random (ours)84.6259.6481.6951.82
SVD(W0W_0) (ours)84.6957.2981.4251.56

The coordinate view identifies a simple design choice: sample a Gaussian basis, normalize it, and train only the coordinates. Normalizing the basis raises the Qwen2.5 reasoning mean from 0.00% to 58.33%, compared with 57.29% for a basis extracted from the pretrained weights by SVD. On Qwen3.5, the corresponding means are 51.17% for Unit Gaussian and 51.56% for SVD. More broadly, Unit Gaussian and QR Random each score above all four budget-matched baselines on all four task-group means. These results show that a simple random basis can be competitive without extracting directions from the pretrained model.

This conclusion concerns the choice of basis. The unadapted Qwen3.5 model still has the highest reasoning mean (52.99%) and scores above every trained method on MATH.

Reviewer_NN7BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper writes a LoRA update as dW = A N B, with shared fixed bases A and B and a small per-task matrix N. It studies how the basis, the sparsity of N, and the direction of N affect performance. Main findings: a normalised random basis nearly matches LoRA; a wider, sparser basis can beat a narrower, dense one at equal budget; and a few-shot gradient gives a useful direction, whose effect is concentrated in a few coordinates.

Critical strengths
  • Clean and controlled setup that separates basis, connectivity and update scale.
  • Some useful observations, e.g. the sensitivity of raw bases to the learning rate and the non-monotonic effect of connectivity.
  • The intervention experiments (signed scaling, relocation of coordinates) are an original way to probe task updates.
Critical weaknesses
  1. Unclear purpose. The paper motivates the setting with cheap per-task storage, but it is purely analytical and shows no concrete benefit (no multi-task serving or merging experiment, no memory measurement, no direct comparison with VeRA, NOLA or LoRA-XS). It is unclear what to do differently after reading it.
  2. Limited rigour. A single learning rate in the main comparison, no standard deviations or seeds, and differences of 1-2 points are interpreted without significance tests. Only two small models and mostly multiple-choice tasks.
  3. Writing. The paper is dense and hard to follow: heavy notation, compressed experimental sections, and tables that mix different settings (e.g. Table 2). The intuition behind the theorems is not explained.
Questions for authors
  1. What is the concrete use case, and can you show a measured benefit?
  2. How does the method compare with VeRA, NOLA and LoRA-XS at matched parameters and tuned learning rates?
  3. Do the results hold across seeds, larger models and other task types?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 1

Thank you for asking us to clarify the motivation and practical value of this work. We have added comparisons with four baselines and extended our evaluation to an interactive task. These experiments help answer a question that matters for LoRA design: when the number of trainable coordinates is limited, how much does the choice of basis matter?

1. Motivation and value for LoRA design

We study LoRA from a coordinate perspective. Fixing the basis lets us vary where the update directions come from, how many are available, how they are connected, and how they are scaled. Our goal is to help researchers decide which of these choices matter when designing LoRA variants.

One practical finding is that a basis derived from the pretrained weights by SVD is not always necessary. Unit Gaussian and SVD give nearly identical multiple-choice scores on Qwen2.5 (84.55% and 84.69%) and complete 97 and 96 of 128 ALFWorld tasks, respectively. The distinction between raw and normalized Gaussian bases is important: on Qwen2.5 reasoning, normalization raises the score from 0.00% to 58.33%. For designers, this makes a normalized Gaussian basis a simple, informative baseline for testing what an SVD-derived basis adds.

2. Comparison with existing methods at comparable trainable budgets

To test whether this finding holds against existing methods, we compared our four basis variants with VeRA, NOLA, LoRA-XS, and SVFT at approximately matched trainable parameter counts. LoRA (r=16r=16) serves as a higher-budget reference.

Unit Gaussian and QR Random each outperform all four budget-matched baselines on all four task-group means in Table R1. For example, on Qwen2.5 reasoning, Unit Gaussian scores 58.33% and QR Random scores 59.64%, compared with 54.43% for the strongest matched baseline, SVFT. The margins are smaller on Qwen3.5, and its unadapted base model retains the highest reasoning mean (52.99%). We therefore report these comparisons as differences in observed means.

Table R1. Task-group means (%) under a common protocol.

MethodQwen2.5 MCQwen2.5 ReasoningQwen3.5 MCQwen3.5 Reasoning
Base (evaluation only)66.0147.6653.3352.99
LoRA (r=16r=16, higher budget)85.9756.5182.8948.57
VeRA74.7448.0570.2649.87
NOLA81.7953.9178.8849.61
LoRA-XS78.4653.1381.4450.26
SVFT81.4254.4378.7850.39
Raw Gaussian (ours)32.200.0032.400.65
Unit Gaussian (ours)84.5558.3381.7151.17
QR Random (ours)84.6259.6481.6951.82
SVD(W0W_0) (ours)84.6957.2981.4251.56

All training runs use a learning rate of 2×10−42\times10^{-4} and no dev-set checkpoint selection. We report means over three seeds. MC averages six multiple-choice tasks using summed scoring. Reasoning averages GSM8K, MATH, and MBPP using greedy decoding with a 768-token limit. Reasoning runs use one epoch for Qwen2.5 and four for Qwen3.5.

3. Parameter counts, uncertainty, and theory

The matched budget refers to trainable parameters. For example, our Qwen3.5 setup uses 1.238M trainable coordinates and 252.512M frozen basis parameters. We will report both counts and remove unmeasured claims about memory and inference speed.

We will distinguish variation across support masks from variation across independent training seeds, and present the original mixed-protocol Table 2 separately from Table R1. We will also simplify the theory around two quantities relevant to design: how many independent update directions a basis and mask allow, and the maximum rank of a single update.

AuthorsOpenReview ↗

Response to Reviewer 1

4. Larger model and interactive evaluation

We tested Qwen3.5-4B on ALFWorld to see whether the basis choice also matters when a model must complete a sequence of actions. We trained on 512 trajectories for one epoch at a learning rate of 2×10−42\times10^{-4}. All methods used the same action-format prompt, with up to 64 tokens per action and 50 steps per task.

Table R2. ALFWorld success rates (%) on 64 Seen and 64 Unseen tasks.

MethodSeenUnseenAverage
Base (evaluation only)0.000.000.00
LoRA (r=16r=16, higher budget)76.5687.5082.03
VeRA1.560.000.78
NOLA68.7578.1373.44
LoRA-XS71.8885.9478.91
SVFT71.8881.2576.56
Raw Gaussian (ours)0.000.000.00
Unit Gaussian (ours)71.8879.6975.78
QR Random (ours)59.3865.6362.50
SVD(W0W_0) (ours)71.8878.1375.00

Unit Gaussian (75.78%) again performs close to SVD (75.00%). It exceeds VeRA and NOLA and falls below SVFT and LoRA-XS. QR Random reaches 62.50% despite having the highest reasoning mean among our variants on both models in Table R1. This gives a second design takeaway: the basis that works best on static reasoning tasks can differ from the one that works best for interactive task execution.

5. Scaling the learned update on ALFWorld

The coordinate view also lets us test what happens when we change only the learned update. We fixed the QR basis and support mask, then scaled the trained coordinates as N(α)=αNtrainedN(\alpha)=\alpha N_{\mathrm{trained}} without further training.

Update scale α\alphaSeen (%)Unseen (%)Average (%)
−0.50.000.000.00
0.00.000.000.00
0.523.4435.9429.69
1.059.3865.6362.50

With the same basis and mask, reversing or removing the update yielded no successful tasks. Applying half the learned update recovered 29.69% success, and applying it in full recovered 62.50%. This experiment isolates the effect of scaling an already learned coordinate update.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.