acceptodds
Withdrawn from ICLR 2027

TensoLoRA: Learning Higher-Order Representations of Weight Updates

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter-efficient fine-tuning (PEFT) adapts pretrained models under a limited trainable-parameter budget. Low-rank adaptation (LoRA) parameterizes each weight update with two low-rank factors. However, reducing the number of trainable parameters in LoRA requires a smaller rank, which lowers the matrix-rank bound and limits the expressive capacity of the weight update. At a fixed parameter budget, weight-update parameterization determines the matrix-rank bound and shapes how update magnitude is distributed across singular directions, as characterized by the singular-value (SV) spectrum. We therefore introduce Tensorized Low-Rank Adaptation (TensoLoRA), which learns higher-order representations of weight updates by jointly optimizing all tensor factors within a fixed parameter budget. Each tensor slice is expressed as a linear combination of shared dense planes, with the slice-wise coefficients parameterized by two line factors per plane. Factorizing these coefficients reduces the parameter cost per component relative to dense coefficient matrices, allowing more jointly learned plane-line components within the same budget while retaining dense trainable planes. TensoLoRA learns less concentrated SV spectra than LoRA and retains higher performance even when the final matrix rank is matched, as shown by rank-controlled analysis. Experiments on video base-to-novel generalization, commonsense reasoning, and image classification across diverse architectures show that TensoLoRA remains competitive with fewer trainable parameters and outperforms LoRA at matched parameter budgets.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Parameter-Efficient Fine-Tuning, Tensorized Weight Updates, Plane-Line Factorization, Singular-Value Spectrum

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cS1QOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TensoLoRA reshapes weight updates into fourth-order tensors and represents them as a sum of products involving multiple trainable dense planes and two vector factors, before mapping the resulting tensor back into matrix space. By factorizing only the slice coefficients while retaining dense planes, the method aims to learn more components within a limited parameter budget. The paper evaluates parameter efficiency and downstream performance, and further investigates the learned spectral structure through rank truncation of the final update matrices and analyses involving singular-value and singular-direction swapping.

Critical strengths
  1. The paper is very well presented, with clear organization and particularly polished and visually appealing figures.

  2. The experimental evaluation is comprehensive, covering multiple downstream tasks and including comparisons with LoRA in terms of memory consumption and runtime.

  3. Overall, I consider this a relatively mature submission.

Critical weaknesses
  1. Insufficient discussion and comparison with recent tensor-based PEFT methods. A considerable number of tensor-based PEFT methods have been proposed over the past two years. However, the paper only selectively discusses and compares against FacT and TeRA. I believe other relevant tensor-based approaches should also be discussed, even if they are not based on Tucker decomposition. A more comprehensive discussion would help better position TensoLoRA within the existing literature and clarify its distinctions from prior work.

  2. Limited advantages in memory consumption and runtime compared with LoRA. Based on the current results, TensoLoRA does not appear to outperform LoRA in terms of memory consumption or runtime. Nevertheless, my understanding is that TensoLoRA uses fewer trainable parameters, which could potentially reduce the storage cost of adapters. The authors could consider explicitly discussing and evaluating this advantage. Additionally, it would be interesting to investigate comparisons with quantized variants in future work.

  3. More extensive experiments across different models are needed. I would encourage the authors to include additional comparative experiments on a wider range of model architectures in the main paper. This would provide stronger evidence of the method's generalizability and practical effectiveness.

  4. The main concern is the clarity and significance of the contribution. My primary concern regarding this work is its contribution relative to existing literature. The general idea of extending matrix-based parameterizations to tensor-based representations is no longer particularly novel, given the growing body of research on tensor-based PEFT methods. Therefore, I believe the authors should more clearly articulate what constitutes the most important contribution of TensoLoRA to the research community, whether from the perspective of tensor-based modeling or PEFT more broadly. In particular, it would strengthen the paper to explain what new insights, capabilities, or advantages TensoLoRA introduces beyond existing tensor-based approaches.

Questions for authors

see above

7 comments
AuthorsOpenReview ↗

Official Comment by Authors

W1: Insufficient discussion and comparison with recent tensor-based PEFT methods. A considerable number of tensor-based PEFT methods have been proposed over the past two years. However, the paper only selectively discusses and compares against FacT and TeRA. I believe other relevant tensor-based approaches should also be discussed, even if they are not based on Tucker decomposition. A more comprehensive discussion would help better position TensoLoRA within the existing literature and clarify its distinctions from prior work.

  • Thank you for pointing out the need for a broader comparison with tensor-based PEFT methods. As you noted, tensor-based PEFT encompasses factorization approaches beyond Tucker decomposition. We expanded the discussion in Section 2.2 to include FacT, LoRTA, LoRETTA, and TeRA, covering Tucker, tensor-train (TT), CP, and Tucker-like tensor networks. We also added comparisons with FacT, LoRTA, and LoRETTA_rep in Table 1 of Section 5.1, alongside TeRA. We also report comparisons with TeRA in Tables 2–4 and FacT in Table 3. Additional experiments are underway, and we will update our response as results become available.
  • These methods differ in their factorization structures and in which factors are learned or shared across layers. Unlike approaches that jointly parameterize updates across layers or rely on frozen random factors, TensoLoRA independently tensorizes each weight update and jointly learns dense planes and line factors. Our existing factorization analysis in Figure 5 of Section 5.7 further shows that TensoLoRA achieves higher HM than a four-way CP variant at comparable parameter counts across all four video benchmarks. These findings highlight factorization structure as an important design choice in tensor-based PEFT, beyond tensorization itself.
AuthorsOpenReview ↗

Official Comment by Authors

W2: imited advantages in memory consumption and runtime compared with LoRA. Based on the current results, TensoLoRA does not appear to outperform LoRA in terms of memory consumption or runtime. Nevertheless, my understanding is that TensoLoRA uses fewer trainable parameters, which could potentially reduce the storage cost of adapters. The authors could consider explicitly discussing and evaluating this advantage. Additionally, it would be interesting to investigate comparisons with quantized variants in future work.

  • Thank you for your thoughtful feedback. We agree that computational efficiency is an important consideration and acknowledge that TensoLoRA introduces additional computational overhead compared with LoRA. To examine this limitation, we evaluated adapter storage separately from training time and peak GPU memory. Using adapter-only checkpoints in the same safetensors format, TensoLoRA reduces FP32 storage from LoRA's 17.744 MB to 0.772 MB and 2.902 MB at plane-line ranks 4 and 16, corresponding to reductions of 95.6% and 83.6%. Despite these reductions, TensoLoRA improves UCF-101 HM from LoRA's 81.7 to 83.0 and 83.2. At a matched parameter budget, it further improves HM to 83.6 with comparable storage requirements. These results demonstrate substantial storage savings at smaller parameter budgets and improved accuracy at comparable storage budgets.
MethodParams. (M)UCF-101 HMFP32 (MB)BF16 (MB)FP32 storage reduction
LoRA (r=16)4.42481.717.7448.897—
TensoLoRA (r_tl=4)0.17783.00.7720.41895.6%
TensoLoRA (r_tl=16)0.71083.22.9021.48383.6%
TensoLoRA (r_tl=100)4.43583.617.8068.935−0.3%
  • We also examined whether the performance gains persist under matched wall-clock budgets. TensoLoRA improved HM on HMDB-51 and UCF-101 by 0.5 and 1.8 percentage points under FP32, with corresponding gains of 0.7 and 2.0 points under BF16. On commonsense reasoning, it improved average accuracy from 84.5 to 85.8 within a comparable training time. These results suggest that the additional per-iteration cost need not compromise TensoLoRA's accuracy advantage under a fixed training-time budget.
  • To reduce computational overhead, we replaced dense update reconstruction with direct contractions between inputs and compact tensor factors. We revised Table 15 in Appendix G to report measurements from the optimized BF16 implementation across four tasks. The updated results show 0.8–2.4% overhead at the same nominal rank and 4.2–14.0% for larger TensoLoRA configurations. The language results below show comparable peak GPU memory and training time, with modest overhead at matched parameter counts.
MethodRankParams. (M)Peak Memory (GiB)Time / 100 iter. (s)Relative to LoRA
LoRA3275.524.316.41.00×
TensoLoRA326.324.216.81.02×
TensoLoRA38475.525.918.71.14×
  • We also appreciate the suggestion regarding quantization and will consider comparisons with quantized variants in future work.
AuthorsOpenReview ↗

Official Comment by Authors

W3: More extensive experiments across different models are needed. I would encourage the authors to include additional comparative experiments on a wider range of model architectures in the main paper. This would provide stronger evidence of the method's generalizability and practical effectiveness.

  • Thank you for your valuable suggestion. We agree that evaluating TensoLoRA across different model architectures is important for assessing its generalizability. We therefore extended our evaluation to personalized text-to-image generation using SDXL [1] and video base-to-novel generalization using X-CLIP [2].
  • On SDXL, TensoLoRA achieves the highest DINO and CLIP-I scores of 0.617 and 0.735 with only 0.238M trainable parameters. On X-CLIP, TensoLoRA achieves HM scores of 84.7 on UCF-101 and 65.3 on HMDB-51 using 0.186M parameters, compared with LoRA's 84.6 and 64.7 using 5.898M parameters. At comparable parameter counts, TensoLoRA further improves HM to 85.4 and 66.4, respectively.
  • Together with our existing evaluations, these experiments cover seven pretrained models across language, image classification, video recognition, and image generation, providing broader evidence of TensoLoRA's applicability across architectures and tasks. We report the SDXL results in Table 4 of Section 5.4 and summarize the X-CLIP findings in Section 5.1, with full comparisons provided in Appendix B. [1] Podell et al. "SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis." ICLR, 2024. [2] Ni et al. "Expanding Language-Image Pretrained Models for General Video Recognition." ECCV, 2022.

1. Personalised Text-to-Image Generation (SDXL)

MethodDINO ↑CLIP-I ↑CLIP-T ↑LPIPS ↑CMMD ↓Params. (M)
LoRA0.4630.64032.390.3091.2751.452
VeRA0.4890.65032.480.3251.3091.451
AdaLoRA0.4680.64232.340.3061.2741.452
DoRA0.5720.70932.920.3721.0522.117
PiSSA0.5800.71832.410.375‾\underline{0.375}1.068‾\underline{1.068}1.452
LoHa0.4240.62332.170.3011.2682.903
FourierFT0.2150.51832.320.2501.1731.452
LoKr0.4490.63232.53‾\underline{32.53}0.3121.3121.454
SHiRA0.4670.64532.090.3421.2541.452
WaveFT0.4950.65532.410.3481.2651.452
TeRA0.5720.71532.430.3761.1020.094
TensoLoRA (r=1)0.6170.73532.350.3701.0800.238
TensoLoRA (r=6)0.610‾\underline{0.610}0.728‾\underline{0.728}32.53‾\underline{32.53}0.3721.0791.425
Reviewer_cvL4OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a new PEFT formulation, called TensoLoRA, based on tensor-shaped parameterization of the delta update. In particular, "dense planes" are shared and used across tensor slices to provide a medium for shared parametrization. In addition, a dropout mechanism is used at train time to randomly remove the TensoLoRA components.

Critical strengths

The proposed formulation appears to be a sensible PEFT formulation.

Results across image/video and language domains appear to be promising.

Critical weaknesses

One of the most attractive aspects of the proposed technique is the ability go below-rank-1 (in LoRA terms) in model complexity. However, experimental validation doesn't provide a comparison against strong competitors and meaningful baselines. Checking out the experimental results:

  • Table 1: TensoLoRA at 0.71M and 4.44M parameters. All baselines are around 4.4M parameters, except one at much lower complexity 0.14.
  • Table 2: All at ~75M, one baseline at 0.3M, yet TensoLORA at 6.3M and 75.5M.
  • Table 3: similar. From these comparison tables, it is very hard to make any sense out of these. Especially in low-complexity settings (below rank 1), the comparisons provide much more limited representation, not providing strong competitors.

In fact, Hugging Face PEFT library contains several techniques that deserve to be evaluated in this context, e.g. (as far as I could remember) VERA, SHIRA, WAVEFT, LOKR/LOHA, IA3) provide flexibility in choosing the PEFT complextiy below rank-1-LORA. Such methods are direct "competitors" for the proposed techniques. Some of these works aren't cited at all, some of them are cited but not compared against.

In general, I would have expected seeing a complexity versus success scatter plot analysis; rather than evaluating at just two operating points, and a much more comprehensive evaluation.

Finally, I should note that PEFT plays a major role in personalization of generative image (or video) models and I think an evaluation & comparison against the strong baselines (as those listed above) in this task would provide valuable insights.

Questions for authors

No extra questions beyond my comments above.

7 comments
AuthorsOpenReview ↗

Official Comment by Authors

W1: One of the most attractive aspects of the proposed technique is the ability go below-rank-1 (in LoRA terms) in model complexity. However, experimental validation doesn't provide a comparison against strong competitors and meaningful baselines. Checking out the experimental results.

  • Thank you for highlighting TensoLoRA's ability to operate below the rank-1 LoRA parameter budget. To strengthen the experimental validation, we expanded our comparisons with strong competitors and meaningful baselines across video, language, image classification, and image generation. The tables below present selected results from Tables 1–4 of the revised manuscript, focusing on configurations below each task's rank-1 LoRA parameter budget and on comparable parameter counts where possible.

Rank-1 LoRA reference budgets

TaskRank-1 LoRA budget (M)
video≈ 0.276
language≈ 2.36
image classification≈ 0.0369
image generation1.452

1. video — Section 5.1, Table 1

MethodParams. (M)UCF-101 HM ↑HMDB-51 HM ↑K-400 HM ↑SSv2 HM ↑
IA³0.0979.6 ± 0.263.2 ± 0.269.0 ± 0.09.1 ± 0.1
VeRA0.1778.5 ± 0.162.3 ± 0.168.3 ± 0.18.5 ± 0.1
LoKr0.1782.2 ± 0.667.8 ± 0.769.9‾\underline{69.9} ± 0.116.2 ± 0.3
FacT-TK₃₂0.1681.2 ± 0.668.4‾\underline{68.4} ± 1.468.9 ± 0.217.8 ± 0.5
LoRTA0.1978.1 ± 0.561.4 ± 0.668.9 ± 0.19.5 ± 0.2
TeRA0.1482.9‾\underline{82.9} ± 0.368.6 ± 0.369.5 ± 0.116.8‾\underline{16.8} ± 0.5
TensoLoRA, rₜₗ = 40.1783.0 ± 1.067.8 ± 0.370.2 ± 0.216.3 ± 0.3

2. language — Section 5.2, Table 2

MethodParams. (M)Average accuracy (%) ↑
WaveFT0.578.4‾\underline{78.4} ± 0.9
IA³0.568.9 ± 0.1
TeRA0.375.4 ± 4.5
TensoLoRA, rₜₗ = 20.478.9 ± 0.4

3. Image classification — Section 5.3, Table 3

MethodParams. (M)Natural ↑Specialized ↑Structured ↑Overall (%) ↑
IA³0.02879.8 ± 0.382.5 ± 1.250.0 ± 0.670.8 ± 0.4
VeRA0.03178.9 ± 0.583.8 ± 0.250.5 ± 0.671.1 ± 0.2
LoKr0.03280.3‾\underline{80.3} ± 0.183.6 ± 0.453.3 ± 0.572.4 ± 0.2
FacT-TK₈0.01480.7 ± 0.384.3 ± 0.655.7 ± 0.673.5 ± 0.1
TeRA0.01077.7 ± 0.183.2 ± 0.348.6 ± 1.369.8 ± 0.5
TensoLoRA, rₜₗ = 20.01380.1 ± 0.284.0‾\underline{84.0} ± 0.254.9‾\underline{54.9} ± 0.173.0‾\underline{73.0} ± 0.1

4. image generation — Section 5.4, Table 4

MethodParams. (M)DINO ↑CLIP-I ↑CLIP-T ↑LPIPS ↑CMMD ↓
VeRA1.4510.4890.65032.48‾\underline{32.48}0.3251.309
TeRA0.0940.5720.71532.430.3761.102
TensoLoRA, rₜₗ = 10.2380.6170.73532.350.3701.080‾\underline{1.080}
TensoLoRA, rₜₗ = 61.4250.610‾\underline{0.610}0.728‾\underline{0.728}32.530.372‾\underline{0.372}1.079
  • At nearby budgets, TensoLoRA achieves the highest mean HM among the listed video methods on UCF-101 and K-400 and the highest language average among the 0.3–0.5M configurations. In image classification, it reaches 73.0 at 0.013M, close to FacT's 73.5 at 0.014M. In image generation, it achieves the highest DINO and CLIP-I scores at 0.238M, although no baseline has a closely matched budget. Figure 4 in Section 5.6 further shows that TensoLoRA remains competitive below the rank-1 LoRA budget, while achieving a clearer performance advantage over competing methods at comparable parameter counts above this threshold.
AuthorsOpenReview ↗

Official Comment by Authors

W2: In fact, Hugging Face PEFT library contains several techniques that deserve to be evaluated in this context, e.g. (as far as I could remember) VERA, SHIRA, WAVEFT, LOKR/LOHA, IA3) provide flexibility in choosing the PEFT complextiy below rank-1-LORA. Such methods are direct "competitors" for the proposed techniques. Some of these works aren't cited at all, some of them are cited but not compared against.

  • Thank you for identifying these relevant competitors. We substantially expanded our experimental comparisons to include all the methods you mentioned, namely VeRA, SHiRA, WaveFT, LoKr, LoHa, and IA³, alongside additional high-rank and tensor-based PEFT methods such as RandLoRA, MoRA, HiRA, FacT, LoRTA, and TeRA. We also updated the related work accordingly. The revised results cover video in Table 1 of Section 5.1, language in Table 2 of Section 5.2, and image classification in Table 3 of Section 5.3. We further introduced image generation as a new evaluation task, with results reported in Table 4 of Section 5.4. To assess parameter efficiency, we examined configurations below the rank-1 LoRA budget and compared methods at similar parameter counts where possible. Figure 4 in Section 5.6 provides additional parameter–performance comparisons across a wider range of budgets.
AuthorsOpenReview ↗

Official Comment by Authors

W3: In general, I would have expected seeing a complexity versus success scatter plot analysis; rather than evaluating at just two operating points, and a much more comprehensive evaluation.

  • Thank you for this suggestion. We agree that two operating points provide an incomplete view of the complexity–performance trade-off. We therefore expanded our experiments across multiple parameter budgets and added Figure 4 in Section 5.6. We varied the plane-line rank of TensoLoRA and the available rank or coefficient settings of competing methods. Since the minimum parameter counts and available configurations differ across methods, we report their actual parameter counts rather than enforcing exact budget matching. TeRA and IA³ are included as single reference configurations. IA³ has no rank parameter to vary, while TeRA's standard formulation is bounded by the rank-1 LoRA budget per weight update. The results show that TensoLoRA remains competitive below this threshold and achieves a clearer performance advantage at higher, comparable parameter budgets.
Reviewer_sGBUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TensoLoRA reshapes weight updates into fourth-order tensors and parameterizes them with learned dense planes and two line factors per component. This allows more components within a fixed trainable-parameter budget than a conventional matrix factorization. The paper evaluates video base-to-novel generalization, commonsense reasoning, and image classification, and studies the learned singular-value spectra through rank truncation and interventions on singular values and directions.

Critical strengths
  1. The parameter-efficiency result is supported across several settings. On Llama3-8B, TensoLoRA matches LoRA’s 84.3% commonsense average with 6.3M rather than 75.5M trainable parameters, and reaches 86.0% at the matched budget (Table 2). The video and image experiments provide additional evidence that the construction is useful beyond one backbone or task family.

  2. The spectral analysis goes beyond observing that the update has higher rank. After truncating every adapted TensoLoRA projection to rank 16, UCF-101 HM remains 83.0%, compared with 81.7% for rank-16 LoRA (Table 4). The singular-value and direction swaps also control the update norm (Table 5). These are useful checks against explaining the result solely by the final matrix-rank bound, even though they do not hold the optimization trajectory fixed.

Critical weaknesses
  1. The comparison is stronger against LoRA than against alternative tensor or high-rank adapters. In Table 2, TeRA uses 0.3M trainable parameters, whereas TensoLoRA uses 6.3M or 75.5M. Since TeRA learns scaling vectors over frozen random tensor factors, this comparison changes both the parameter budget and which factors are learned. The paper also discusses RandLoRA, MoRA, and HiRA without experimental comparisons. A common-budget comparison with a close alternative would better establish the value of the plane-line structure itself. TeRA

  2. The storage savings do not translate into a clear training-resource advantage. At matched parameter counts, Table 13 reports 50.2 versus 35.3 seconds per 100 iterations for video and 362.3 versus 294.1 seconds for language, with similar or slightly higher peak memory. The paper acknowledges dense update reconstruction, which is helpful, but the accuracy comparisons do not show whether the gain survives an equal wall-clock budget. This limits the practical efficiency claim, particularly where the accuracy margin is small.

Questions for authors
  1. How does TensoLoRA compare with TeRA or a representative high-rank adapter across overlapping parameter budgets, with comparable hyperparameter tuning? This would help distinguish the benefit of the proposed structure from increased trainable capacity.

  2. Do the matched-parameter gains remain when training time is also matched? Please also clarify the precision reported as “bfloat32” in Table 13 so that the timing comparison can be interpreted and reproduced.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

W1/Q1: The comparison is stronger against LoRA than against alternative tensor or high-rank adapters. In Table 2, TeRA uses 0.3M trainable parameters, whereas TensoLoRA uses 6.3M or 75.5M. Since TeRA learns scaling vectors over frozen random tensor factors, this comparison changes both the parameter budget and which factors are learned. The paper also discusses RandLoRA, MoRA, and HiRA without experimental comparisons. A common-budget comparison with a close alternative would better establish the value of the plane-line structure itself. (Q1: How does TensoLoRA compare with TeRA or a representative high-rank adapter across overlapping parameter budgets, with comparable hyperparameter tuning? This would help distinguish the benefit of the proposed structure from increased trainable capacity.)

  • Thank you for highlighting this important point. We agree that the choice of trainable tensor factors is an important distinction beyond parameter count. TeRA trains only layer-specific scaling vectors while keeping its random core and factor matrices frozen. Theorem 2 of TeRA [1] establishes that its standard formulation stays within the rank-1 LoRA parameter budget per weight update. In contrast, TensoLoRA jointly learns dense planes and line factors and scales its budget by adding plane-line components. We therefore added comparisons at overlapping budgets. In language, TensoLoRA achieves 78.9 at 0.4M parameters versus TeRA's 75.4 at 0.3M. In image classification, it achieves 73.0 at 0.013M versus 69.8 at 0.010M.

[1] Gu et al. “TeRA: Vector-based Random Tensor Network for High-Rank Adaptation of Large Language Models.” ACL, 2026.

  • We expanded our comparisons to high-rank, tensor-based, and other PEFT methods, including RandLoRA, MoRA, HiRA, VeRA, SHiRA, WaveFT, LoKr, LoHa, FacT, and LoRTA. Results are reported for video in Table 1 of Section 5.1, language in Table 2 of Section 5.2, and image classification in Table 3 of Section 5.3. We also added image generation using SDXL [2] as a new evaluation task, with results reported in Table 4 of Section 5.4. Figure 4 in Section 5.6 further examines parameter–performance trade-offs across different budgets.

[2] Podell et al. "SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis." ICLR, 2024.

AuthorsOpenReview ↗

Official Comment by Authors

W2/Q2: The storage savings do not translate into a clear training-resource advantage. At matched parameter counts, Table 13 reports 50.2 versus 35.3 seconds per 100 iterations for video and 362.3 versus 294.1 seconds for language, with similar or slightly higher peak memory. The paper acknowledges dense update reconstruction, which is helpful, but the accuracy comparisons do not show whether the gain survives an equal wall-clock budget. This limits the practical efficiency claim, particularly where the accuracy margin is small. (Q2: Do the matched-parameter gains remain when training time is also matched? Please also clarify the precision reported as “bfloat32” in Table 13 so that the timing comparison can be interpreted and reproduced.)

  • Thank you for your constructive feedback. We conducted additional experiments to assess whether TensoLoRA's performance gains persist under matched wall-clock budgets. For video, we matched trainable parameter counts at 4.42M for LoRA and 4.44M for TensoLoRA, using LoRA's standard 11-epoch training time as the target budget with matched seeds and precision. Under FP32, TensoLoRA improved HM from 68.7 to 69.2 on HMDB-51 and from 81.4 to 83.2 on UCF-101. These gains persisted under BF16, with improvements of 0.7 and 2.0 percentage points, respectively. On commonsense reasoning, TensoLoRA increased average accuracy from 84.5 to 85.8 under BF16 while training for 13,933 seconds, compared with 13,965 seconds for LoRA. These results suggest that TensoLoRA's higher per-iteration cost need not compromise its accuracy advantage under a fixed wall-clock budget. Extending TensoLoRA's training to 15,924 seconds further improved accuracy to 86.4.
DatasetPrecisionLoRA time(s)TensoLoRA time(s)LoRA HMTensoLoRA HMΔHM(pp)
HMDB-51FP32270.6 ± 2.1271.9 ± 2.368.7 ± 0.569.2 ± 0.4+0.5
HMDB-51BF16163.6 ± 5.1164.2 ± 4.768.4 ± 0.769.1 ± 0.2+0.7
UCF-101FP322461.8 ± 19.82465.9 ± 21.081.4 ± 1.283.2 ± 0.5+1.8
UCF-101BF16581.4 ± 3.0582.6 ± 3.281.4 ± 1.383.4 ± 0.7+2.0
DatasetPrecisionLoRA time(s)TensoLoRA time(s)LoRA AccTensoLoRA AccΔAcc(pp)
CommonsenseBF1613,965 (2 epochs)13,933 (1.75 epochs)84.585.8+1.3
CommonsenseBF1613,965 (2 epochs)15,924 (2 epochs)84.586.4+1.9
  • Following the wall-clock experiments reported above, we further optimized the implementation of TensoLoRA. We replaced explicit dense update reconstruction with direct contractions between inputs and compact tensor factors. As reported in Table 15 of Appendix G, the optimized implementation incurs 0.8–2.4% training-time overhead relative to LoRA at the same nominal rank and 4.2–14.0% at higher TensoLoRA parameter budgets across four tasks. These results demonstrate that TensoLoRA retains its compact trainable factorization with limited additional training time. The “bfloat32” label in the original Table 13 was a typo; the measurements were conducted in FP32. The revised Table 14 reports separate BF16 measurements using the optimized implementation.
AuthorsOpenReview ↗

Official Comment by Authors

  • Thank you again for your detailed review, positive assessment, and helpful suggestions! Your feedback has helped us improve our experiments.
  • We wanted to follow up to share the additional results (Appendix I) in our revised manuscript. If any questions remain, we would be happy to discuss them further. Thank you again for your time!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear Reviewers, Area Chairs, Program Chairs, and Senior Area Chairs,

Thank you for taking the time to thoroughly review our paper and provide valuable feedback. We sincerely appreciate the thoughtful and constructive suggestions from Reviewers sGBU (R1), cvL4 (R2), and cS1Q (R3). We have revised the manuscript and expanded our experiments to address their main concerns. The primary updates are summarized below, with all modifications highlighted in blue in the revised version.

  • Additional Baselines (R1, R2, R3). We broadened our comparisons to include recent high-rank and tensor-based PEFT methods, including VeRA, SHiRA, WaveFT, LoKr, LoHa, HiRA, RandLoRA, FacT, LoRTA, and LoRETTA. The revised experiments cover video, language, image classification, and image generation.
  • Computational Cost (R1, R2, R3). We evaluated performance under matched wall-clock budgets and optimized the implementation through direct tensor contractions. Table 15 in Appendix G reports updated BF16 training-time and peak-memory measurements. We also measured adapter-only checkpoint sizes to quantify storage savings separately from training costs.
  • Parameter Efficiency (R2). We expanded comparisons across trainable-parameter budgets, including settings below the rank-1 LoRA budget. Figure 4 presents parameter–performance trade-offs across multiple operating points.
  • Additional Task and Architecture (R2, R3). We extended our evaluation to personalized image generation using SDXL in Section 5.4 and video recognition using X-CLIP, summarized in Section 5.1 with full results in Appendix B. Our evaluation now spans seven pretrained models across four task domains.
  • Clarification of Methodological Contributions (R3). We revised the Introduction and Related Work to clarify TensoLoRA's plane-line factorization, which retains dense trainable planes while factorizing slice-wise coefficients. The factorization and rank-controlled analyses in Figures 2 and 5 and Tables 5 and 6 further demonstrate the importance of this design.

Below, we provide detailed, point-by-point responses to each reviewer’s comments. Thank you again for your thoughtful feedback. We hope that the revisions sufficiently address the concerns raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.