- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes LoopLoRA, which uses recurrent calculation to improve LoRA performance instead of add more parameters. The core idea of this paper is clear and well designed, but during the performance comparison, some baseline results seem not fine-tuned enough and do not include similar computation comparisons. Also, during the inference time comparison, don't consider the merge parameter for basic Lora.
Critical strengths
-
Simple and parameter-efficient design. LoopedLoRA introduces computational depth within the existing low-rank bottleneck using a shared r×r operator. It leaves the pretrained backbone unchanged and adds only 0.22% more trainable parameters than LoRA on Qwen3-4B-Base at r=16. The method is evaluated across both Qwen3 and Llama-3 backbones.
-
Comprehensive component ablations. Table 3 separately examines recurrent depth, blending strategy, normalization, and activation. Performance improves as k increases from 0 to 4 and then largely saturates, while learned blending outperforms recurrent-only and fixed-weight alternatives. Removing RMSNorm or SiLU also reduces performance, supporting the main architectural choices.
Critical weaknesses
Potentially under-tuned baselines. Results on GSM8K, SVAMP, HumanEval, and HumanEval+ appear unstable and sometimes counterintuitive. On Qwen3-4B-Base, increasing the LoRA rank from 16 to 32 changes GSM8K only from 76.48 to 76.57, while reducing HumanEval from 82.32 to 79.27 and HumanEval+ from 74.39 to 69.51 despite doubling the trainable parameters. Moreover, Qwen3-8B underperforms Qwen3-4B on both GSM8K and SVAMP for standard LoRA (73.62 vs. 76.48 and 72.67 vs. 76.06) and LoopedLoRA (86.58 vs. 89.44 and 87.33 vs. 89.83). Since the paper reports neither base-model results under the same protocol nor method-specific hyperparameter tuning, it is unclear whether the reported gains reflect LoopedLoRA’s design or under-optimized baselines. Please provide base-model anchors and learning-rate/rank sweeps for LoRA and the strongest competing methods.
Unfair inference-efficiency comparison. Standard LoRA can be merged into the backbone with no adapter-specific inference overhead, whereas LoopedLoRA is non-mergeable and requires recurrent computation. Moreover, LoopedLoRA uses fused Triton kernels while the implementations are not kernel-matched. Please compare against merged LoRA and report FLOPs, latency, and memory under matched deployment settings.
Questions for authors
Potentially under-tuned baselines. Results on GSM8K, SVAMP, HumanEval, and HumanEval+ appear unstable and sometimes counterintuitive. On Qwen3-4B-Base, increasing the LoRA rank from 16 to 32 changes GSM8K only from 76.48 to 76.57, while reducing HumanEval from 82.32 to 79.27 and HumanEval+ from 74.39 to 69.51 despite doubling the trainable parameters. Moreover, Qwen3-8B underperforms Qwen3-4B on both GSM8K and SVAMP for standard LoRA (73.62 vs. 76.48 and 72.67 vs. 76.06) and LoopedLoRA (86.58 vs. 89.44 and 87.33 vs. 89.83). Since the paper reports neither base-model results under the same protocol nor method-specific hyperparameter tuning, it is unclear whether the reported gains reflect LoopedLoRA’s design or under-optimized baselines. Please provide base-model anchors and learning-rate/rank sweeps for LoRA and the strongest competing methods.
Unfair inference-efficiency comparison. Standard LoRA can be merged into the backbone with no adapter-specific inference overhead, whereas LoopedLoRA is non-mergeable and requires recurrent computation. Moreover, LoopedLoRA uses fused Triton kernels while the implementations are not kernel-matched. Please compare against merged LoRA and report FLOPs, latency, and memory under matched deployment settings.
Missing compute-matched comparison. LoopedLoRA adds few parameters but obtains additional capacity through repeated computation, increasing training time by 27.9% and prefill latency by approximately 34%. Existing comparisons primarily match parameter count rather than computation. Please compare against higher-rank or nonlinear adapter baselines under matched training and inference FLOPs or wall-clock budgets to establish whether recurrence uses the additional computation effectively.