- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TensoLoRA reshapes weight updates into fourth-order tensors and represents them as a sum of products involving multiple trainable dense planes and two vector factors, before mapping the resulting tensor back into matrix space. By factorizing only the slice coefficients while retaining dense planes, the method aims to learn more components within a limited parameter budget. The paper evaluates parameter efficiency and downstream performance, and further investigates the learned spectral structure through rank truncation of the final update matrices and analyses involving singular-value and singular-direction swapping.
Critical strengths
-
The paper is very well presented, with clear organization and particularly polished and visually appealing figures.
-
The experimental evaluation is comprehensive, covering multiple downstream tasks and including comparisons with LoRA in terms of memory consumption and runtime.
-
Overall, I consider this a relatively mature submission.
Critical weaknesses
-
Insufficient discussion and comparison with recent tensor-based PEFT methods. A considerable number of tensor-based PEFT methods have been proposed over the past two years. However, the paper only selectively discusses and compares against FacT and TeRA. I believe other relevant tensor-based approaches should also be discussed, even if they are not based on Tucker decomposition. A more comprehensive discussion would help better position TensoLoRA within the existing literature and clarify its distinctions from prior work.
-
Limited advantages in memory consumption and runtime compared with LoRA. Based on the current results, TensoLoRA does not appear to outperform LoRA in terms of memory consumption or runtime. Nevertheless, my understanding is that TensoLoRA uses fewer trainable parameters, which could potentially reduce the storage cost of adapters. The authors could consider explicitly discussing and evaluating this advantage. Additionally, it would be interesting to investigate comparisons with quantized variants in future work.
-
More extensive experiments across different models are needed. I would encourage the authors to include additional comparative experiments on a wider range of model architectures in the main paper. This would provide stronger evidence of the method's generalizability and practical effectiveness.
-
The main concern is the clarity and significance of the contribution. My primary concern regarding this work is its contribution relative to existing literature. The general idea of extending matrix-based parameterizations to tensor-based representations is no longer particularly novel, given the growing body of research on tensor-based PEFT methods. Therefore, I believe the authors should more clearly articulate what constitutes the most important contribution of TensoLoRA to the research community, whether from the perspective of tensor-based modeling or PEFT more broadly. In particular, it would strengthen the paper to explain what new insights, capabilities, or advantages TensoLoRA introduces beyond existing tensor-based approaches.
Questions for authors
see above
7 comments
Official Comment by Authors
W2: imited advantages in memory consumption and runtime compared with LoRA. Based on the current results, TensoLoRA does not appear to outperform LoRA in terms of memory consumption or runtime. Nevertheless, my understanding is that TensoLoRA uses fewer trainable parameters, which could potentially reduce the storage cost of adapters. The authors could consider explicitly discussing and evaluating this advantage. Additionally, it would be interesting to investigate comparisons with quantized variants in future work.
- Thank you for your thoughtful feedback. We agree that computational efficiency is an important consideration and acknowledge that TensoLoRA introduces additional computational overhead compared with LoRA. To examine this limitation, we evaluated adapter storage separately from training time and peak GPU memory. Using adapter-only checkpoints in the same safetensors format, TensoLoRA reduces FP32 storage from LoRA's 17.744 MB to 0.772 MB and 2.902 MB at plane-line ranks 4 and 16, corresponding to reductions of 95.6% and 83.6%. Despite these reductions, TensoLoRA improves UCF-101 HM from LoRA's 81.7 to 83.0 and 83.2. At a matched parameter budget, it further improves HM to 83.6 with comparable storage requirements. These results demonstrate substantial storage savings at smaller parameter budgets and improved accuracy at comparable storage budgets.
| Method | Params. (M) | UCF-101 HM | FP32 (MB) | BF16 (MB) | FP32 storage reduction |
|---|---|---|---|---|---|
| LoRA (r=16) | 4.424 | 81.7 | 17.744 | 8.897 | — |
| TensoLoRA (r_tl=4) | 0.177 | 83.0 | 0.772 | 0.418 | 95.6% |
| TensoLoRA (r_tl=16) | 0.710 | 83.2 | 2.902 | 1.483 | 83.6% |
| TensoLoRA (r_tl=100) | 4.435 | 83.6 | 17.806 | 8.935 | −0.3% |
- We also examined whether the performance gains persist under matched wall-clock budgets. TensoLoRA improved HM on HMDB-51 and UCF-101 by 0.5 and 1.8 percentage points under FP32, with corresponding gains of 0.7 and 2.0 points under BF16. On commonsense reasoning, it improved average accuracy from 84.5 to 85.8 within a comparable training time. These results suggest that the additional per-iteration cost need not compromise TensoLoRA's accuracy advantage under a fixed training-time budget.
- To reduce computational overhead, we replaced dense update reconstruction with direct contractions between inputs and compact tensor factors. We revised Table 15 in Appendix G to report measurements from the optimized BF16 implementation across four tasks. The updated results show 0.8–2.4% overhead at the same nominal rank and 4.2–14.0% for larger TensoLoRA configurations. The language results below show comparable peak GPU memory and training time, with modest overhead at matched parameter counts.
| Method | Rank | Params. (M) | Peak Memory (GiB) | Time / 100 iter. (s) | Relative to LoRA |
|---|---|---|---|---|---|
| LoRA | 32 | 75.5 | 24.3 | 16.4 | 1.00× |
| TensoLoRA | 32 | 6.3 | 24.2 | 16.8 | 1.02× |
| TensoLoRA | 384 | 75.5 | 25.9 | 18.7 | 1.14× |
- We also appreciate the suggestion regarding quantization and will consider comparisons with quantized variants in future work.
Official Comment by Authors
W3: More extensive experiments across different models are needed. I would encourage the authors to include additional comparative experiments on a wider range of model architectures in the main paper. This would provide stronger evidence of the method's generalizability and practical effectiveness.
- Thank you for your valuable suggestion. We agree that evaluating TensoLoRA across different model architectures is important for assessing its generalizability. We therefore extended our evaluation to personalized text-to-image generation using SDXL [1] and video base-to-novel generalization using X-CLIP [2].
- On SDXL, TensoLoRA achieves the highest DINO and CLIP-I scores of 0.617 and 0.735 with only 0.238M trainable parameters. On X-CLIP, TensoLoRA achieves HM scores of 84.7 on UCF-101 and 65.3 on HMDB-51 using 0.186M parameters, compared with LoRA's 84.6 and 64.7 using 5.898M parameters. At comparable parameter counts, TensoLoRA further improves HM to 85.4 and 66.4, respectively.
- Together with our existing evaluations, these experiments cover seven pretrained models across language, image classification, video recognition, and image generation, providing broader evidence of TensoLoRA's applicability across architectures and tasks. We report the SDXL results in Table 4 of Section 5.4 and summarize the X-CLIP findings in Section 5.1, with full comparisons provided in Appendix B. [1] Podell et al. "SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis." ICLR, 2024. [2] Ni et al. "Expanding Language-Image Pretrained Models for General Video Recognition." ECCV, 2022.
1. Personalised Text-to-Image Generation (SDXL)
| Method | DINO ↑ | CLIP-I ↑ | CLIP-T ↑ | LPIPS ↑ | CMMD ↓ | Params. (M) |
|---|---|---|---|---|---|---|
| LoRA | 0.463 | 0.640 | 32.39 | 0.309 | 1.275 | 1.452 |
| VeRA | 0.489 | 0.650 | 32.48 | 0.325 | 1.309 | 1.451 |
| AdaLoRA | 0.468 | 0.642 | 32.34 | 0.306 | 1.274 | 1.452 |
| DoRA | 0.572 | 0.709 | 32.92 | 0.372 | 1.052 | 2.117 |
| PiSSA | 0.580 | 0.718 | 32.41 | 1.452 | ||
| LoHa | 0.424 | 0.623 | 32.17 | 0.301 | 1.268 | 2.903 |
| FourierFT | 0.215 | 0.518 | 32.32 | 0.250 | 1.173 | 1.452 |
| LoKr | 0.449 | 0.632 | 0.312 | 1.312 | 1.454 | |
| SHiRA | 0.467 | 0.645 | 32.09 | 0.342 | 1.254 | 1.452 |
| WaveFT | 0.495 | 0.655 | 32.41 | 0.348 | 1.265 | 1.452 |
| TeRA | 0.572 | 0.715 | 32.43 | 0.376 | 1.102 | 0.094 |
| TensoLoRA (r=1) | 0.617 | 0.735 | 32.35 | 0.370 | 1.080 | 0.238 |
| TensoLoRA (r=6) | 0.372 | 1.079 | 1.425 |
Official Comment by Authors