acceptodds
Under review as a conference paper at ICLR 2027

LoopedLoRA: Low-Rank Adaptation with Recurrent Subspace Depth

Abstract

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method. A common way to improve LoRA adaptation is to increase the rank, which also increases the number of trainable parameters; despite many variants, most adapters retain a single-pass mapping through the low-rank bottleneck. We propose LoopedLoRA, which introduces weight-shared recurrent depth within this subspace. LoopedLoRA recurrently processes an -dimensional latent state using a fusion-friendly RMSNorm–SiLU operator, adding only more trainable parameters than LoRA at on Qwen3-4B-Base. To allow the contribution of recurrent processing to vary across tokens, a learned latent blender adaptively combines the direct and recurrent paths, and LoopedLoRA-Light reuses its score to bypass recurrence at inference. Across four Qwen3 and Llama-3 models, LoopedLoRA improves GSM8K by up to 12.96 percentage points over LoRA and outperforms rank-doubled LoRA () and competitive baselines on mathematical reasoning, while remaining competitive on code generation and commonsense reasoning. With fused Triton kernels and CUDA graph replay, full LoopedLoRA maintains decode latency comparable to LoRA, while LoopedLoRA-Light achieves up to faster decoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LoRA, LLM, Looped Architecture

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xf6NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes LoopLoRA, which uses recurrent calculation to improve LoRA performance instead of add more parameters. The core idea of this paper is clear and well designed, but during the performance comparison, some baseline results seem not fine-tuned enough and do not include similar computation comparisons. Also, during the inference time comparison, don't consider the merge parameter for basic Lora.

Critical strengths
  1. Simple and parameter-efficient design. LoopedLoRA introduces computational depth within the existing low-rank bottleneck using a shared r×r operator. It leaves the pretrained backbone unchanged and adds only 0.22% more trainable parameters than LoRA on Qwen3-4B-Base at r=16. The method is evaluated across both Qwen3 and Llama-3 backbones.

  2. Comprehensive component ablations. Table 3 separately examines recurrent depth, blending strategy, normalization, and activation. Performance improves as k increases from 0 to 4 and then largely saturates, while learned blending outperforms recurrent-only and fixed-weight alternatives. Removing RMSNorm or SiLU also reduces performance, supporting the main architectural choices.

Critical weaknesses

Potentially under-tuned baselines. Results on GSM8K, SVAMP, HumanEval, and HumanEval+ appear unstable and sometimes counterintuitive. On Qwen3-4B-Base, increasing the LoRA rank from 16 to 32 changes GSM8K only from 76.48 to 76.57, while reducing HumanEval from 82.32 to 79.27 and HumanEval+ from 74.39 to 69.51 despite doubling the trainable parameters. Moreover, Qwen3-8B underperforms Qwen3-4B on both GSM8K and SVAMP for standard LoRA (73.62 vs. 76.48 and 72.67 vs. 76.06) and LoopedLoRA (86.58 vs. 89.44 and 87.33 vs. 89.83). Since the paper reports neither base-model results under the same protocol nor method-specific hyperparameter tuning, it is unclear whether the reported gains reflect LoopedLoRA’s design or under-optimized baselines. Please provide base-model anchors and learning-rate/rank sweeps for LoRA and the strongest competing methods.

Unfair inference-efficiency comparison. Standard LoRA can be merged into the backbone with no adapter-specific inference overhead, whereas LoopedLoRA is non-mergeable and requires recurrent computation. Moreover, LoopedLoRA uses fused Triton kernels while the implementations are not kernel-matched. Please compare against merged LoRA and report FLOPs, latency, and memory under matched deployment settings.

Questions for authors

Potentially under-tuned baselines. Results on GSM8K, SVAMP, HumanEval, and HumanEval+ appear unstable and sometimes counterintuitive. On Qwen3-4B-Base, increasing the LoRA rank from 16 to 32 changes GSM8K only from 76.48 to 76.57, while reducing HumanEval from 82.32 to 79.27 and HumanEval+ from 74.39 to 69.51 despite doubling the trainable parameters. Moreover, Qwen3-8B underperforms Qwen3-4B on both GSM8K and SVAMP for standard LoRA (73.62 vs. 76.48 and 72.67 vs. 76.06) and LoopedLoRA (86.58 vs. 89.44 and 87.33 vs. 89.83). Since the paper reports neither base-model results under the same protocol nor method-specific hyperparameter tuning, it is unclear whether the reported gains reflect LoopedLoRA’s design or under-optimized baselines. Please provide base-model anchors and learning-rate/rank sweeps for LoRA and the strongest competing methods.

Unfair inference-efficiency comparison. Standard LoRA can be merged into the backbone with no adapter-specific inference overhead, whereas LoopedLoRA is non-mergeable and requires recurrent computation. Moreover, LoopedLoRA uses fused Triton kernels while the implementations are not kernel-matched. Please compare against merged LoRA and report FLOPs, latency, and memory under matched deployment settings.

Missing compute-matched comparison. LoopedLoRA adds few parameters but obtains additional capacity through repeated computation, increasing training time by 27.9% and prefill latency by approximately 34%. Existing comparisons primarily match parameter count rather than computation. Please compare against higher-rank or nonlinear adapter baselines under matched training and inference FLOPs or wall-clock budgets to establish whether recurrence uses the additional computation effectively.

Reviewer_SkiuOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LoopedLoRA and LoopedLoRA-Light, which introduces recurrent depth finetuning into LoRA: A token-wise gate blends the learned and direct paths, while LoopedLoRA-Light reuses the gate to bypass at inference time for speedup. More importantly, this paper accelerates the decoding phase via fused Triton kernels and CUDA graph replay.

Critical strengths
  1. strong performance in math. LoopedLoRA achieves the highest score in math reasoning tasks, outperforming other baselines with same and double ranks.
  2. detailed ablation study on recurrent depth, recurrent operator.
  3. practical engineering methods for speedup at decoding. This paper addresses the overhead via triton kernel fusion and CUDA graph replay, keeping the decoding latency comparable with LoRA. Combining system optimization is a good try in PEFT.
Critical weaknesses
  1. lack of exposure of how to tune hyper-parameters of baseline methods. In Table 2, the performance of MiSS is significantly lower than other methods, AuroRA gets only 34.95 in GSM8k in Qwen3-8B-base. Further details of configurations is needed to make the results more convincing.
  2. Lack of multi-seed results. The gain of LoopedLoRA is within standard deviation in commonsense reasoning task. Lack of multi-seed results on this experiment makes the method and results limited.
  3. The evaluation doesn't convince me of applicable deployment advantage. The latency is limited to batch size 1, and limited input and output sequences. Meanwhile, LoopedLoRA also has ~28% higher training time and 34% higher prefill latency. These additional cost can't be neglected. Broader measurements across batch sizes and diverse output lengths are important. Additionally, the method should compare with LoRA varaints with similar system optimizations.
Questions for authors
  1. Can the authors provide some insights about why the gain in commonsense reasoning is less compared with that of math reasoning? Could additional analyses of recurrent depth and gate behavior across domains help explain this discrepancy?
  2. Can the authors provide any experiment results when using LoopedLoRA on new architectures with MoE, e.g., OLMoE?
  3. While Table 3 shows that multiple loops perform better than a single non-linear transform, could authors compare LoopedLoRA with an equally deep, FLOPs matched adapter using untied weights? Such a comparison would help clarify whether the gain is from recurrent weight sharing or additional nonlinear compute Additional questions can be viewed as in weakness section. Overall, I think I can raise the score if the authors can provide fair comparisons and more insights in rebuttal phase.
Reviewer_eGA3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Standard LoRA passes each token through its low-rank bottleneck once. The usual way to get more out of LoRA is to raise the rank, which grows the parameter count linearly.

Within this manuscript, the work adds depth instead. It projects the input to an r-dimensional latent state z_0 = Ax. It then refines that state k times with one shared matrix R. Finally it decodes the result. Because R is reused at every step, adding loops adds no parameters.

The experiment reported on Qwen3-4B/8B and Llama-3.2-3B/3.1-8B, with defaults r = 16 and k = 4. GSM8K improves by up to +12.96 points over LoRA, and SVAMP by up to +14.7. The gains on Llama are, +2 points on the 3B model and +8 on the 8B. Per Code and commonsense, roughly +0.4 to +1.9 points on commonsense averages and small changes on HumanEval.

Critical strengths

Within the manuscript, the proposed method is easy to use, well-engineered design with a training-free compute knob. The method adds only (r+1)^2 parameters per projection and reduces exactly to LoRA when k = 0 or the loop is bypassed. The Light variant reuses the learned gate as an inference-time switch, giving a graded accuracy–compute trade-off from a single checkpoint without retraining (Table 7). At τ = 0.1 it skips 18–40% of loop computations. The fused Triton kernel brings decode latency to parity with LoRA.

Critical weaknesses

Within the experiment result reported by this manuscript, the number looks wired to me. For example, the LoRA on QWen8B against GSM8K, as 73.6, is significant lower than the 4B version, which is 76.4. Besides, within original Qwen technical report, the 4B model without LoRA has already achieved 87.79, which looks like similar result compared with LoopedLoRA reported.

Theoretically, my understanding for looped mechanism shall be most suitable for reasoning task, not only match but also coding. Per reported result on HumanEval+ of LoopedLoRA is worse than orignal LoRA with same rank, 16, I do think this is worth for author to give a deep thinking. Intuitively, it doesn't make sense. Even just considering regarding the parameter size from learning perspective, this may mean the depth factor within LoopedLoRA is not really functioning on HumanEval+.

Questions for authors
  1. I would suggest authors to first align the reported prior arts result with other reference and citation, or indicate the testing protocol difference.

  2. I suggest authors to provide a reasonable analysis regarding why LoopedLoRA doesn't achieve significant gain compared other arts, within this target downstream task.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.