acceptodds
Under review as a conference paper at ICLR 2027

RaSTA: Random Subspace Tuning Adaptation

Abstract

Parameter-efficient fine-tuning (PEFT) methods such as LoRA specialize a pretrained model by learning a small additive update to its frozen weights. However, LoRA's trainable parameter count is tied to the shape of each adapted matrix, costing at least the sum of its input and output dimensions even at rank one. We propose Random Subspace Tuning Adaptation (RaSTA), which adapts the model within frozen random subspaces shared across layers and learns only how their directions interact, allowing trainable parameter budgets well below rank-one LoRA without storing the frozen directions in the adapter checkpoint. Its two variants trade off the expressivity of the interactions against the number of directions reached under a matched budget. RaSTA-DM learns a dense mixing matrix between the bases, while RaSTA-HS fixes the mixing as a Walsh-Hadamard transform and learns one scaling vector per basis. To our knowledge, this is the first use of the Walsh-Hadamard transform for this purpose. The latter is somewhat more costly but performs better and degrades more gracefully as the budget shrinks. Across mathematical reasoning, commonsense reasoning, and code generation with 7-8B models, as well as natural-language understanding with RoBERTa-large, both variants remain competitive with rank-one LoRA, and sometimes with higher-rank LoRA, at budgets ranging from less than half to about one tenth of its parameter count. Both variants also train faster and use less peak memory than rank-one LoRA, except for RaSTA-HS at its largest considered budget.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
parameter-efficient fine-tuning, low-rank adaptation, random projections, Walsh-Hadamard transform

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_n3GrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper proposes RaSTA (Random Subspace Tuning Adaptation), which fine-tunes the weights by by adapting them within random subspaces shared across the layers and tuning the directions
  • Proposes two variants: using a dense mixing matrix (DM) and mixing using Walsh-Hadamard transform
  • RaSTA calculates ΔW=γBLM~BR⊤\Delta W=\gamma B_L\widetilde{M}B_R^\top, where BL∈Rm×kB_L \in \mathbb{R}^{m \times k} and BR∈Rn×kB_R \in \mathbb{R}^{n \times k} are frozen random matrices shared across layers, and M~\widetilde{M} is the mixing matrix
  • For MD variant, M is a standard trainable matrix
  • For HS variant, M~=diag⁡(sL) HV diag⁡(sR)\widetilde{M} = \operatorname{diag}(s_L)\, H_V\, \operatorname{diag}(s_R), with HV∈RV×VH_V \in \mathbb{R}^{V \times V} being a fixed normalized Walsh-Haramard matrix.
Critical strengths

None.

Critical weaknesses
  • The paper is incremental over VeRA. The MD variant is a simple combination of VeRA and LoRA-XS, and the HS variant simply adds another random matrix.
  • The performance of RaSTA on all the benchmarks (cmmonsense reasoning, math reasoning, code generation, and GLUE) is wither lower than LoRA even when LoRA uses rank=1, or barely matches it. Simply reducing the trainable parameters is not enough justification if the performance degrades, especially since LoRA is an established and extremely practical method for PEFT.
  • [1] has already noted that to come close to LoRA performance, VeRA needs a large dimension for the random matrix. With such large dimensions, VeRA effectively loses all the advantages of LoRA (large matrices lead to large matmuls, leading to increase in GPU memory and training wall time). Hence, VeRA is not a good baseline to use.
  • The wall time numbers for the HS variant reported in Table 7 are suspicious. Since RaSTA-HS also needs the large matmul with the random BLB_L and BRB_R matrices, which use the same dimensions as VeRA.

--

References

[1] Savadikar et al., "WeGeFT: Weight-Generative Fine-Tuning for Multi-Faceted Efficient Adaptation of Large Models", ICML 2025.

Questions for authors
  • What memory does the field Pk512Pk_{512} measure? Does it include activations, optimizer states, as well as parameters?
  • Why is the latency of RaSTA-HS lower than VeRA?
1 comment

Thank you for the review. Shared points are in our general comment (G1-G6).

Wall time of RaSTA-HS (your question: why is its latency lower than VeRA's?). You were right to be suspicious. In the original comparison, VeRA ran through the PEFT library and RaSTA through our own modules, and that gap was an implementation artifact, not a property of the methods. At ρ=1, RaSTA-HS uses a wider basis than VeRA's V=1024 (V=2048 on 4096-wide layers), so it does at least as much adapter computation. With VeRA re-implemented in the same framework and precision as RaSTA, RaSTA-HS is slower than VeRA at ρ=1, about equal at ρ=2, and faster only at ρ=4. We removed the earlier claims. RaSTA-DM, which uses small bases, matches rank-one LoRA in step time.

What the memory field measures (your question). Peak allocated GPU memory over the timed training steps: base weights, adapter parameters and frozen bases, gradients, optimizer states, and activations.

Incremental over VeRA. We agree that RaSTA builds on VeRA and LoRA-XS, and say so. One correction: RaSTA-HS does not add another random matrix. The Hadamard matrix is fixed and deterministic, and the bases are the same kind of frozen Gaussian bases as in VeRA. What changes is what is learned: VeRA learns diagonal scalings with no mixing and a count (m+V) that depends on the layer's output size, while RaSTA-HS lets all V^2 direction pairs interact through the fixed core, with 2V trainable parameters independent of m and n. At the same budget as LoRA-XS's dense k x k core, it reaches V=1024 instead of k=45 directions (4096 x 4096 layer, ρ=2) (see G3).

Accuracy versus rank-one LoRA. We agree that the claim is a trade-off and not an improvement. At ρ=1, RaSTA-HS uses about half of rank-one LoRA's parameters (45-52%). Across the 14 comparisons in Tables 3-6 (GSM8K and MATH, HumanEval and MBPP on three models each, plus the commonsense and GLUE averages) it is ahead of or tied with rank-one LoRA in 7 and behind by at most 1.6 points in the other 7. At ρ=4 (about a tenth of the parameters) it falls further behind (GLUE average 85.8 against 87.6). The gain is storage and optimizer state where many adapters are kept or served, with 295K parameters per adapter at ρ=4 against 2.62M for rank-one LoRA on Mistral-7B (G4). We have clarified this in the abstract and conclusion.

VeRA as a baseline. We include VeRA because it is the closest prior method that shares frozen random bases, and we use V=1024 and V=256 as in its own setting. We agree that VeRA needs a wide basis for good accuracy and that wide bases cost compute and memory; that cost is exactly what our same-precision comparison now measures, and RaSTA-HS at ρ=4 is cheaper than VeRA while RaSTA-DM uses a small basis. VeRA is not our only baseline: we also compare with LoRA at ranks 1, 8 and 64 and with LoRA-XS. Thank you for the pointer to WeGeFT; we will discuss it in the related work if the paper is accepted.

Reviewer_W5HXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Random Subspace Tuning Adaptation, abbreviated as RaSTA, a parameter-efficient fine-tuning method that updates frozen weight matrices using a learned mixing core placed between two frozen random Gaussian bases shared across layers of matching shape. Under a layer-size-aware parameter budget divided by a compression factor and scaled by the square root of that factor, the authors study two variants: RaSTA-DM, which trains a dense core matrix, and RaSTA-HS, which places a fixed normalized Walsh-Hadamard transform between two learned diagonal scaling vectors. Experiments evaluate Llama-2-7B, Mistral-7B, and Llama-3-8B on mathematical reasoning, commonsense reasoning, and code generation, as well as RoBERTa-large on GLUE, supplemented by single-step training time and peak memory profiling.

Critical strengths

Placing a fast Walsh-Hadamard transform between two learned diagonal scaling vectors inside shared frozen Gaussian bases in RaSTA-HS is a clean structural formulation that substantially expands the spanned subspace width under the same trainable parameter budget. Empirically, Figure 2 and Tables 3 to 6 consistently show that expanding the number of reached random directions via RaSTA-HS outperforms learning a dense mixing matrix over fewer random directions in RaSTA-DM across tasks and budgets.

Critical weaknesses
  1. The manuscript contains glaring, unedited LLM placeholder instructions in the bibliography, reflecting careless reliance on generative AI and poor editorial quality control. In the AI Use Statement on page 10, the authors admit using generative AI to design ablations, write code, interpret results, check mathematical statements, draft text, and format references. Remarkably, the authors failed to read or verify the AI-generated bibliography before submission, leaving three literal unedited todo prompts right in the References section on pages 11 and 12: line 542 reads TODO verify eprint/author list, line 623 reads TODO verify eprint/venue, and line 631 reads TODO verify full author list, title, and arXiv id, where the reference has an incomplete author list and no arXiv identifier at all. Submitting a paper to ICLR with raw LLM verification prompts still sitting in the bibliography is unacceptable.

  2. The headline peak-memory and out-of-memory claims in Table 7, Figure 4, and Table 10 are driven by an unoptimized mixed-precision casting artifact in the LoRA and VeRA baselines rather than an intrinsic memory advantage of RaSTA. In Table 7 on page 9 for Mistral-7B at batch size 4, the authors claim that rank-one LoRA consumes 32.5 GiB at sequence length 512 and runs out of memory on an 80 GiB GPU at sequence length 2048, whereas RaSTA-DM uses only 23.0 GiB at sequence length 512 and 51.3 GiB at sequence length 2048, and RaSTA-HS at budget divisor 1 with width 2048 uses 27.7 GiB and 69.3 GiB. From basic memory accounting, this gap cannot come from parameters, gradients, or optimizer states: by Table 2, rank-one LoRA on Mistral-7B has only 2.62 million trainable parameters, meaning its 32-bit parameters, gradients, and AdamW states occupy only 42 megabytes, or 0.04 GiB, in total. Nor can it come from adapter intermediate activations: in rank-one LoRA, the bottleneck activation has width 1, whereas in RaSTA-HS at budget divisor 1, the adapter projects the activation through the right basis into width 2048, applies the right scaling vector, runs the Hadamard transform, and applies the left scaling vector, requiring autograd to save multiple width-2048 intermediate tensors per layer for the backward pass. Why, then, does rank-one LoRA consume 9.5 GiB more memory at sequence length 512 and over 29 GiB more at sequence length 2048 in Table 7? The caption of Table 7 and Appendix D on line 799 reveal the reason: the base model is held in bfloat16 while trainable parameters are held in float32. In PyTorch, when LoRA's first factor matrix is stored in float32 without wrapping the forward matmul in bfloat16 autocast, multiplying the bfloat16 input activation tensor by the float32 LoRA matrix forces PyTorch to upcast the entire input activation tensor to float32 and save an extra float32 copy of the full input activation across all 224 adapted linear layers for the backward pass, which at batch 4 and sequence length 2048 wastes roughly 30 GiB of redundant activation memory. In RaSTA-DM and LoRA-XS, because the first projection matrix is a frozen bfloat16 basis, the large matrix multiplication stays in bfloat16 and only the tiny compressed bottleneck tensor is cast to float32 for the trainable core. When rank-one LoRA computes its forward pass in bfloat16 as in standard mixed-precision training, its true memory footprint is strictly smaller than both RaSTA-HS and RaSTA-DM, and its adapter FLOPs are 2048 times smaller than RaSTA-HS at budget divisor 1.

  3. The mathematical justification for the Walsh-Hadamard transform in RaSTA-HS and the universal update scaling across both variants is incomplete, and all accuracy tables rely on noisy single-seed runs. First, in Section 3 on line 244, the authors argue that the Hadamard matrix is uniquely suited as the fixed core because its rows are orthogonal and have equal-magnitude entries. However, the left and right bases are already independent isotropic Gaussian random matrices with normalized columns. Because the distribution of an isotropic Gaussian matrix is invariant under any orthogonal transformation, multiplying a Gaussian basis by an orthogonal matrix yields another identically distributed Gaussian basis, and the paper provides no ablation comparing the Hadamard matrix against a fixed random orthogonal matrix to show whether the Hadamard structure contributes anything beyond fast kernel execution. Second, in Section 3 on line 257, the authors apply the exact same square-root scaling factor to both RaSTA-HS and RaSTA-DM. While the subspace width in RaSTA-HS scales inversely with the budget divisor, the subspace width in RaSTA-DM scales with the inverse square root of the budget divisor according to Equation 3, and its dense core carries no normalization factor. Applying the same scaling rule to two parameterizations whose subspace dimensions scale with different powers of the budget divisor is mathematically inconsistent, which explains why the loss curves for RaSTA-DM in Figure 3 still separate noticeably across budgets. Finally, all benchmark numbers come from single-seed runs where evaluation noise exceeds method differences; for instance, in Table 5 on Llama-3-8B HumanEval, RaSTA-DM with 274K parameters at budget divisor 4 scores 49.39 percent, beating both 1.12M parameters at divisor 1 with 46.95 percent and 552K parameters at divisor 2 with 46.34 percent.

Questions for authors
  1. Clarification request on the memory profiling artifact in Tables 7 and 10: Why does rank-one LoRA, which has only 42 megabytes of parameter, gradient, and optimizer states and a width-1 bottleneck activation, consume 9.5 GiB more GPU memory than RaSTA-DM at sequence length 512 and over 29 GiB more at sequence length 2048? In your profiling script, did multiplying the bfloat16 input activation by the float32 LoRA weight matrix cause PyTorch to upcast and save a full float32 copy of the input activation tensor at every linear layer, while RaSTA avoided this because its frozen right basis was stored in bfloat16? What are the exact peak memory and step time of LoRA at ranks 1, 8, and 64, VeRA, and RaSTA when standard bfloat16 autocast is enabled for the adapter forward pass?

  2. Clarification request on update scaling and Hadamard ablation: Given that the subspace width scales inversely with the budget divisor in RaSTA-HS but with the inverse square root of the budget divisor in RaSTA-DM, how do the authors mathematically justify applying the exact same square-root update scaling factor to both variants? Furthermore, does the Walsh-Hadamard matrix in RaSTA-HS provide any accuracy advantage over a fixed random orthogonal matrix of the same size, or is its benefit purely computational via the fast Hadamard kernel?

1 comment

Thank you for the careful review. Shared points are in our general comment (G1-G6).

References. You are right, and we apologize: the three "TODO verify" notes were left in by us. We have verified the three entries concerned (FourierFT, Tied-LoRA, SSH) against the publishers' and arXiv pages, completed them, and removed the notes. We disclosed AI assistance in the paper; the failure was in our verification, and we have rechecked the whole list.

Memory profiling. Your accounting is correct: the 9.5 GiB gap between rank-one LoRA and RaSTA-DM cannot come from parameters, gradients or optimizer states. In the original profiling, LoRA and VeRA ran through the PEFT library with fp32 adapter weights, while our RaSTA modules cast their parameters to the activation dtype and computed in bf16. We have not traced PEFT's internals step by step, so we cannot confirm the exact mechanism, but your explanation (a larger fp32 copy of each adapter input kept for the backward pass) is consistent with what we observe: once LoRA and VeRA are re-implemented with the same bf16 compute path as RaSTA (fp32 parameters cast to bf16 at each forward pass, which plays the role of autocast), most of the gap disappears. The original profiling table (previously Table 7), Figure 4 and the full-grid table (previously Table 10) are removed, together with the out-of-memory claims at length 2048.

Answering your questions with the same-precision implementations:

  • LoRA at ranks 1, 8 and 64: step time is essentially independent of rank (within about 1%), and peak memory rises by about 0.3 GiB (r=8) and 3 GiB (r=64) over r=1.
  • VeRA (V=1024): about 20% slower than rank-one LoRA, about 4 GiB more memory.
  • RaSTA-DM: the same step time as rank-one LoRA, about 3 GiB less memory, independent of ρ.
  • RaSTA-HS at ρ = 1, 2, 4: about 50%, 25% and 10% slower than rank-one LoRA; memory about 3 GiB above it at ρ=1, equal at ρ=2, and about 1.5 GiB below at ρ=4. So you are right that rank-one LoRA's adapter cost is far smaller than RaSTA-HS's, and that rank-one LoRA is lighter than RaSTA-HS at small ρ. For RaSTA-DM we measure the opposite: it stays a few GiB lighter than same-precision rank-one LoRA, which we attribute to LoRA having to keep each adapter's input for the backward pass while a frozen basis needs only a small projection. The abstract, introduction, Sec. 4.3 and conclusion are updated to say exactly this.

Hadamard versus a random orthogonal matrix. We ran the ablation you describe (Sec. 4.2, Table 7: RaSTA-HS at ρ=2, RTE, MRPC, CoLA and STS-B, three seeds): a Haar-random orthogonal core gives 82.2 on average against 82.4 for Hadamard, within seed noise. So you are right that on these tasks the Hadamard matrix's benefit is computational (O(V log V), no stored matrix), and the paper now says so. One qualification on rotation invariance: the core sits between the two learned diagonal scalings, and a diagonal matrix does not commute with Q, so Q cannot be absorbed into B_R unless the scaling is constant. With an identity core, only V of the V^2 direction pairs interact (Lemma 1, App. B). We did not run an identity-core control.

Same scaling for both variants. What the scaling must compensate is the norm of the core's change, which depends on the number of trainable entries, not on the basis width. For RaSTA-DM the core has k^2 entries and norm proportional to k; for RaSTA-HS the norm is proportional to sqrt(V). Since k^2 and 2V are both about sqrt(mn)/ρ, both scale as ρ^(-1/2), so the same γ = sqrt(ρ) applies (App. C, derived for the early phase of training). We do not claim it removes all budget dependence: RaSTA-DM remains more budget-sensitive, as Fig. 3 and Sec. 4.2 state.

Single seeds. We agree that evaluation noise is comparable to many method differences. Your HumanEval example (49.39 vs 46.95) is a four-problem difference on 164 problems. Sec. 4 says differences of 1-2 points should be read as noise, and the new ablation is multi-seed. If the paper is accepted, we will add seeds for the main comparison in the camera-ready version.

Reviewer_apRUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Random Subspace Tuning Adaptation (RaSTA), which fine-tunes the model within frozen random subspaces shared across layers. The paper presents two variants of RaSTA, RaSTA-DM and RaSTA-HS. In RaSTA-DM, a dense matrix is the trainable parameter and in RaSTA-HS, the trainable parameters are two vectors. They evaluate their method against other PEFT methods in GLUE, Commonsense reasoning, HumanEval, MBPP, GSM8K, and MATH.

Critical strengths
  1. The method is simple and easy to understand. Figure 1 shows multiple PEFT methods, making it easier to identify how different PEFT methods differ.
Critical weaknesses
  1. The novelty of the methods presented is minimal. RaSTA-DM can be viewed as LoRA-XS with just random initialization. Also, sharing frozen parameters has been proposed in multiple papers, e.g., VeRA and NOLA[1]. RaSTA-HS is a slight modification to LoRA-XS that introduces two additional trainable vectors while freezing the core base matrix. Overall, their methods introduce a significant number of additional non-trainable parameters. However, in the paper, I do not see the non-trainable parameters being reported, so simply comparing trainable parameters with other methods like LoRA is not enough.
  2. Additionally, this choice of initialization (RaSTA-DM) seems to be worse compared to LoRA-XS on the majority of benchmarks tested in the paper (Tables 3, 4, and 6).
  3. The paper uses significantly old benchmarks for comparison. If the proposed PEFT method performs as well as or better than contemporary methods, why did the authors not use more modern benchmarks to measure performance?
  4. In the experiment, the base model's performance (before fine-tuning) is missing, and it is hard to quantify how much each PEFT method learned. Also, while code is evaluated with EvalPlus, GSM8K and MATH are evaluated with a custom pipeline rather than a standard harness such as lm-evaluation-harness. Could the authors justify this decision and describe how answers are extracted?

[1]https://arxiv.org/pdf/2310.02556

Questions for authors

My questions are stated above in the weakness section.

1 comment

Thank you for the review. We respond to each point; points shared with other reviews are in our general comment (G1-G6).

Novelty. We agree that RaSTA-DM is LoRA-XS with shared random bases instead of SVD bases; the paper presents it as a control for the bases, not as the main contribution. RaSTA-HS is different from "LoRA-XS plus two vectors": it replaces LoRA-XS's trainable dense k x k core by a fixed Hadamard matrix between two learned scalings, so the trainable count is 2V rather than k^2. At the same budget this reaches V = 1024 directions instead of k = 45 (4096 x 4096 layer, ρ = 2), with all V^2 direction pairs interacting (Lemma 1, App. B). VeRA learns diagonal scalings without any mixing and its count (m+V) depends on the layer's output size; NOLA mixes whole random matrices. We do not claim that sharing frozen bases across layers is new (G3).

Non-trainable parameters. We agree that the paper did not report them. The frozen bases are shared across all layers of the same shape, are regenerated from seeds and not stored in checkpoints, and are included in our peak-memory measurements (G1). They are a small fraction of the base model's weights. LoRA-XS, by contrast, keeps separate SVD bases for every layer, and VeRA also relies on frozen bases. Trainable parameters still determine optimizer state and adapter checkpoint size, which matter when many adapters are stored, but we agree they are not the whole cost, which is why Sec. 4.3 reports peak memory.

RaSTA-DM versus LoRA-XS. We agree. RaSTA-DM is behind LoRA-XS on commonsense reasoning and GLUE (Tables 4 and 6) and mixed on math and code (Tables 3 and 5). The abstract and introduction now say this. RaSTA-HS is the variant we recommend, and it is on par with or ahead of LoRA-XS on average (Fig. 2).

Benchmarks. We used the benchmarks of the methods we compare against (LoRA, VeRA, LoRA-XS, NOLA) so that results are directly comparable, and the base models include Llama-3-8B. If you have specific newer benchmarks in mind, we would be glad to know which claim they would test.

Base-model results. All methods in our tables are trained and evaluated under identical conditions (data, steps, prompt, decoding), so the comparison among them does not depend on the base score. The base models are not instruction-tuned, so their scores would mostly reflect how well they follow the prompt format, not what an adapter learned. We nonetheless agree that a base-model row helps interpret the tables [and, if the paper is accepted, we will add one with the same prompt and decoding].

Evaluation pipeline for GSM8K and MATH. All models are evaluated zero-shot after merging the adapter into the base weights, with the same Alpaca-format prompt as in training (ending in "Let's think step by step."), greedy decoding, and at most 512 (GSM8K) or 2048 (MATH) new tokens. GSM8K uses the full 1,319-problem test set and MATH uses the standard 500-problem MATH-500 subset.

  • GSM8K: the answer is the number after "The answer is" (the answer format of MetaMathQA, our training data) or, if that phrase is absent, the last number in the output. It is compared numerically with the reference answer after "####".
  • MATH-500: the answer is the content of the last \boxed{} in the output (otherwise the text after "the answer is" or "answer:", otherwise the last line). It counts as correct if it matches the reference \boxed{} answer after normalization or is mathematically equivalent (a symbolic check with sympy, covering expressions, fractions, intervals and finite sets). The same code, prompt and settings apply to every method, and code generation is evaluated with EvalPlus. We wrote the script because the standard harnesses use their own few-shot prompts and answer conventions, which differ from the format our models are trained on, and we wanted training and evaluation to use the same format for all methods. The fallbacks (last number, last line) make scoring slightly lenient for outputs that do not follow the format, but they apply equally to all methods. The script is part of the released code.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): summary of revisions and responses to main concerns

We thank the reviewers and have uploaded a revised PDF. Below: what changed and which concerns each change addresses (C1-C6), then the concerns shared by several reviewers (G1-G2 here, G3-G6 in the next comment).

SUMMARY OF REVISIONS C1. Training-cost comparison redone (Sec. 4.3; abstract, introduction, conclusion). LoRA and VeRA are now implemented in the same framework and precision as RaSTA. The old profiling tables and figure are removed. Addresses: W5HX (memory artifact), n3Gr (RaSTA-HS wall time versus VeRA; what the memory measures), apRU (non-trainable parameters). See G1.

C2. Hadamard core: justification and ablation (Sec. 3; Sec. 4.2 and Table 7; App. B). Two lemmas on why the core must mix and why flat orthogonal cores are the most evenly spread, and an ablation against a Haar-random orthogonal core (RaSTA-HS, ρ=2, four GLUE tasks, three seeds). Addresses: W5HX (no comparison with a random orthogonal matrix; is the benefit only computational). See G2.

C3. Derivation of the update scaling (Sec. 3; App. C). Addresses: W5HX (same scaling for both variants). See G2.

C4. Claims aligned with the tables (abstract, introduction, conclusion, Sec. 4.3; App. A numbers corrected). We now state that RaSTA-DM is behind LoRA-XS on GLUE and commonsense reasoning, that RaSTA-HS is close to but not above rank-one LoRA, and we removed efficiency claims the new comparison does not support. Addresses: apRU (DM versus LoRA-XS), n3Gr (accuracy versus LoRA). See G4.

C5. Multi-seed reporting for the new ablation (Table 7; App. B): mean and standard deviation over three seeds, and a disclosure that our new seed-0 runs differ from Table 6 (CoLA). Addresses: W5HX (single-seed noise), for the new experiment only. See G5.

C6. References corrected (three unfinished entries). Addresses: W5HX. See G6.

G1. Memory and wall-clock (W5HX, n3Gr, apRU) We agree with W5HX and n3Gr. In the original profiling, LoRA and VeRA ran through the PEFT library, while RaSTA and LoRA-XS used our own modules computing in bf16. As W5HX computes, a gap of several GiB between rank-one LoRA and RaSTA-DM cannot come from parameters, gradients or optimizer states.

With LoRA and VeRA implemented in the same framework and precision as RaSTA (bf16 compute, fp32 trainable parameters; parameter counts and initialization checked against PEFT), most of the gap disappears, which is consistent with W5HX's explanation. We did not isolate the exact mechanism inside PEFT. The result (C1):

  • RaSTA-DM has the same step time as rank-one LoRA and uses a few GiB less memory. We attribute this to LoRA keeping each adapter's input for the backward pass.
  • RaSTA-HS needs more compute, because it applies a width-V transform in every layer. It is slower than rank-one LoRA at small budgets and the cost falls as ρ grows; its memory is above LoRA's at ρ=1 and below it at ρ=4. Against same-precision VeRA it is slower at ρ=1, about equal at ρ=2, and faster only at ρ=4.
  • We removed our earlier claims that both variants are faster and lighter than rank-one LoRA, and the out-of-memory claims at length 2048, which were measured with the PEFT versions.
  • What remains is size: both variants train 2-10x fewer parameters than rank-one LoRA, so optimizer state and adapter checkpoints are smaller.

What is measured (n3Gr): peak allocated GPU memory during a training step, covering base weights, adapter parameters, frozen bases, gradients, optimizer states and activations. Non-trainable parameters (apRU): the frozen bases are shared across layers of the same shape, are included in that peak-memory figure, and are a small fraction of the base model's weights. They are regenerated from seeds and not stored in checkpoints. VeRA also relies on frozen bases. We could not produce a full-model table in this window. If the paper is accepted, we will include one in the camera-ready version.

G2. Hadamard core and update scaling (W5HX) Does the Hadamard matrix help accuracy beyond a fixed random orthogonal matrix? Not on average. In the new ablation (C2), a Haar-random orthogonal core gives 82.2 against 82.4 for Hadamard, within seed noise. On these tasks its benefit is computational (O(V log V) time, nothing stored), and the paper now says so. The core is not redundant with the bases, however. It sits between the two learned diagonal scalings, so it cannot be absorbed into a Gaussian basis, and with an identity core only V of the V^2 direction pairs would interact (Lemma 1, App. B). We did not run an identity-core control.

On the shared scaling (C3): what matters is the number of trainable entries, not the basis width. Both variants have about sqrt(mn)/ρ trainable entries (k^2 ≈ 2V), Adam moves each by about the learning rate, and so the size of the update shrinks as ρ^(-1/2) in both (App. C, early training). In Fig. 3 the scaled RaSTA-HS curves are nearly parallel across budgets, and RaSTA-DM's spread is reduced but not removed.

AuthorsOpenReview ↗

General response (2/2): summary of revisions and responses to main concerns

G3. Novelty (apRU, n3Gr) We agree that RaSTA-DM is a small change to LoRA-XS (shared random instead of SVD bases), and the paper presents it as a control for the bases. Our main variant is RaSTA-HS. VeRA's core is diagonal and its parameter count (m+V) depends on the layer's output size. RaSTA-HS instead places a fixed Hadamard matrix between two learned scalings, so that all V^2 left-right direction pairs interact, with 2V trainable parameters independent of m and n (Lemma 1, App. B). LoRA-XS spends its budget on a dense k x k core; at the same budget RaSTA-HS reaches V = 1024 directions instead of k = 45 (4096 x 4096 layer, ρ = 2), and on average it is ahead of both RaSTA-DM and LoRA-XS at equal ρ (Fig. 2). Sharing frozen bases across layers (VeRA, NOLA) is not what we claim as new. To our knowledge, the Walsh-Hadamard transform has not been used as a mixing core between frozen bases. We also introduce a size-aware budget ρ with an update scaling (C3).

G4. Accuracy versus rank-one LoRA (n3Gr, apRU) Our claim is a size-accuracy trade-off, not higher accuracy than LoRA (C4). At ρ = 1, RaSTA-HS uses about 45-52% of rank-one LoRA's parameters. Across the 14 comparisons in Tables 3-6 (GSM8K and MATH, HumanEval and MBPP on three models each, plus the commonsense and GLUE averages), it is ahead of or tied with rank-one LoRA in 7 and behind by at most 1.6 points in the other 7. At ρ = 4 (about a tenth of the parameters) the gap grows (GLUE average 85.8 against 87.6). We agree with apRU that RaSTA-DM is behind LoRA-XS on GLUE and commonsense reasoning, and the abstract and introduction now say so. The practical gain is storage and optimizer state where many adapters are kept or served: at ρ = 4 an adapter has 295K parameters, against 2.62M for rank-one LoRA on Mistral-7B.

G5. Single seeds (W5HX, apRU) The main tables are single-seed, as stated in Sec. 4, and differences of 1-2 points should be read as noise. W5HX's HumanEval example (49.39 vs 46.95) is a four-problem difference on 164 problems (one problem is 0.61 points). The new ablation reports mean and standard deviation over three seeds (C5), and App. B discloses that our new seed-0 runs differ from Table 6 (CoLA 66.5 against 62.8), which illustrates the noise. We could not rerun the main tables in this window. If the paper is accepted, we will add seeds for the main comparison in the camera-ready version.

G6. References (W5HX) The unfinished "TODO verify" notes were our editorial failure, and we apologize. The three entries concerned (FourierFT, Tied-LoRA, SSH) are corrected against the publishers' and arXiv pages, and the notes are removed (C6).

Reviewer apRU's questions on benchmarks, base-model results and the evaluation pipeline, and n3Gr's comments on VeRA as a baseline, are answered in the individual replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.