- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- The paper proposes RaSTA (Random Subspace Tuning Adaptation), which fine-tunes the weights by by adapting them within random subspaces shared across the layers and tuning the directions
- Proposes two variants: using a dense mixing matrix (DM) and mixing using Walsh-Hadamard transform
- RaSTA calculates , where and are frozen random matrices shared across layers, and is the mixing matrix
- For MD variant, M is a standard trainable matrix
- For HS variant, , with being a fixed normalized Walsh-Haramard matrix.
Critical strengths
None.
Critical weaknesses
- The paper is incremental over VeRA. The MD variant is a simple combination of VeRA and LoRA-XS, and the HS variant simply adds another random matrix.
- The performance of RaSTA on all the benchmarks (cmmonsense reasoning, math reasoning, code generation, and GLUE) is wither lower than LoRA even when LoRA uses rank=1, or barely matches it. Simply reducing the trainable parameters is not enough justification if the performance degrades, especially since LoRA is an established and extremely practical method for PEFT.
- [1] has already noted that to come close to LoRA performance, VeRA needs a large dimension for the random matrix. With such large dimensions, VeRA effectively loses all the advantages of LoRA (large matrices lead to large matmuls, leading to increase in GPU memory and training wall time). Hence, VeRA is not a good baseline to use.
- The wall time numbers for the HS variant reported in Table 7 are suspicious. Since RaSTA-HS also needs the large matmul with the random and matrices, which use the same dimensions as VeRA.
--
References
[1] Savadikar et al., "WeGeFT: Weight-Generative Fine-Tuning for Multi-Faceted Efficient Adaptation of Large Models", ICML 2025.
Questions for authors
- What memory does the field measure? Does it include activations, optimizer states, as well as parameters?
- Why is the latency of RaSTA-HS lower than VeRA?
Thank you for the review. Shared points are in our general comment (G1-G6).
Wall time of RaSTA-HS (your question: why is its latency lower than VeRA's?). You were right to be suspicious. In the original comparison, VeRA ran through the PEFT library and RaSTA through our own modules, and that gap was an implementation artifact, not a property of the methods. At ρ=1, RaSTA-HS uses a wider basis than VeRA's V=1024 (V=2048 on 4096-wide layers), so it does at least as much adapter computation. With VeRA re-implemented in the same framework and precision as RaSTA, RaSTA-HS is slower than VeRA at ρ=1, about equal at ρ=2, and faster only at ρ=4. We removed the earlier claims. RaSTA-DM, which uses small bases, matches rank-one LoRA in step time.
What the memory field measures (your question). Peak allocated GPU memory over the timed training steps: base weights, adapter parameters and frozen bases, gradients, optimizer states, and activations.
Incremental over VeRA. We agree that RaSTA builds on VeRA and LoRA-XS, and say so. One correction: RaSTA-HS does not add another random matrix. The Hadamard matrix is fixed and deterministic, and the bases are the same kind of frozen Gaussian bases as in VeRA. What changes is what is learned: VeRA learns diagonal scalings with no mixing and a count (m+V) that depends on the layer's output size, while RaSTA-HS lets all V^2 direction pairs interact through the fixed core, with 2V trainable parameters independent of m and n. At the same budget as LoRA-XS's dense k x k core, it reaches V=1024 instead of k=45 directions (4096 x 4096 layer, ρ=2) (see G3).
Accuracy versus rank-one LoRA. We agree that the claim is a trade-off and not an improvement. At ρ=1, RaSTA-HS uses about half of rank-one LoRA's parameters (45-52%). Across the 14 comparisons in Tables 3-6 (GSM8K and MATH, HumanEval and MBPP on three models each, plus the commonsense and GLUE averages) it is ahead of or tied with rank-one LoRA in 7 and behind by at most 1.6 points in the other 7. At ρ=4 (about a tenth of the parameters) it falls further behind (GLUE average 85.8 against 87.6). The gain is storage and optimizer state where many adapters are kept or served, with 295K parameters per adapter at ρ=4 against 2.62M for rank-one LoRA on Mistral-7B (G4). We have clarified this in the abstract and conclusion.
VeRA as a baseline. We include VeRA because it is the closest prior method that shares frozen random bases, and we use V=1024 and V=256 as in its own setting. We agree that VeRA needs a wide basis for good accuracy and that wide bases cost compute and memory; that cost is exactly what our same-precision comparison now measures, and RaSTA-HS at ρ=4 is cheaper than VeRA while RaSTA-DM uses a small basis. VeRA is not our only baseline: we also compare with LoRA at ranks 1, 8 and 64 and with LoRA-XS. Thank you for the pointer to WeGeFT; we will discuss it in the related work if the paper is accepted.