- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Mixture of Latent Recursions (MoLR), which adds adaptive recursion to the latent space of LoRA. A shared matrix and a fixed sinusoidal encoding refine the down-projected latent state over multiple steps. A threshold router decides the recursion depth per token, and an auxiliary-loss-free update keeps the depth usage balanced. Because the transformation is shared, the parameter count does not grow with the maximum depth . Experiments on language understanding, language reasoning, and image classification show that MoLR outperforms representative LoRA variants at a similar parameter budget.
Critical strengths
Placing adaptive recursion inside the LoRA bottleneck is a novel idea. The threshold update separates load balancing from task learning, which is a clean design. The evaluation covers 21 datasets on six backbones across three task types, with seed variance reported for all results. Eq. 6 gives a compact polynomial view of the recursion, making the method easy to follow. The practical gains are also notable: MoLR at beats LoRA at , and adding recursion improves three other LoRA variants.
Critical weaknesses
W1. The drop at large is not explained. In Fig. 2b, accuracy at falls to about 57% at , below LoRA at 59.21%. The paper attributes this to "interference" but gives no evidence. Eq. 6 contains , and no norm constraint is applied, so the latent state may explode or vanish. This weakens the claim that deeper recursion can scale LoRA.
W2. The ablation lacks a depth-matched baseline. In Tab. 6a, "Fixed" runs one step, while MoLR runs up to two. Without a fixed 2-step baseline, the gain cannot be attributed to routing. The gain also comes mainly from one task: HumanEval improves by 2.44 points with a std of 1.83, while the other six tasks improve by only 0.47 points on average.
W3. Parameter sharing shows no clear benefit. In Tab. 6c, the step-specific variant scores 61.39%, slightly above MoLR at 61.21%. At , it adds only 128 parameters per module.
W4. Inference cost is high. In Tab. 5, per-token latency is 0.69 ms, about 4× that of LoRA at 0.18 ms. This overhead is large relative to the accuracy gain.
W5. The motivation is weak. LoRA assumes that weight updates have low intrinsic rank, and most follow-up work shares this view. The paper does not explain why richer latent computation is needed under this premise. Its own results also show limited headroom: in App. Tab. 11, raising the LoRA rank from 8 to 32 improves the NLR average by only 1.1 points.
Questions for authors
Q1. Could the authors report the spectral norm of and the norm of at each step for and ? Does spectral normalization or an orthogonality constraint remove the drop at large ?
Q2. Could the authors report a fixed 2-step baseline without routing on all seven NLR tasks? Does MoLR still outperform it, with and without HumanEval?
Q3. Could the authors run the step-specific variant at and ? Does parameter sharing help at larger ?
5 comments
W4. In Tab. 5, we profile methods without runtime optimization to enable controlled comparisons and minimize optimization-induced confounding effects. To better reflect practical deployment, we additionally measure per-token latency (ms) with commonly used inference optimizations, including KV caching, compilation, and CUDA Graphs. With these general-purpose optimizations alone, MoLR already approaches the latency of merged LoRA, supporting its practical deployment. Specialized kernels could further reduce the remaining overhead. We leave their development to future work.
| Method | Eager | Optimized | Avg. |
|---|---|---|---|
| LoRA (merged) | 0.18 | 0.07 | 59.21 |
| MoLR | 0.69 | 0.09 | 61.21 |
W5. The low-rank premise motivates a compact adaptation space but does not imply that LoRA’s direct mapping from latent representations to output fully exploits its potential. Consequently, improving adaptation performance and narrowing the gap with full-parameter fine-tuning under a limited parameter budget remains an active and central research direction (L86). MoLR pursues this objective by implementing adaptive latent recursion, enriching latent computation without sacrificing parameter efficiency. The performance gains brought by increasing LoRA rank are task- and backbone-dependent.
Dear Reviewer D6yV,
As the discussion phase draws to a close, we would like to sincerely thank you for your time and thoughtful feedback. We hope that our rebuttal and additional experimental results have adequately addressed your concerns and provided further clarity on our contributions.
If you find that your previous concerns have been sufficiently resolved, we would be grateful if you could kindly consider reflecting these clarifications in your final evaluation. Of course, we would also be happy to address any remaining questions.
Thank you again for your valuable time and consideration.
Best regards, The Authors
We thank the reviewer for the thoughtful feedback.
W1: Interference. By interference, we refer to potential competition between recursion steps for a shared latent transformation, as each step operates on a different latent-state distribution. The improvement from step-specific transforms over the parameter-sharing variant in Tab. 6c is consistent with this interpretation.
W1 & Q1: Spectral Norm of Wlatent. The table below shows the [min, max] of spectral norms of Wlatent across adapted modules in each selected layer. We observe no systematic change as T increases from 2 to 8. This suggests that the performance drop is unlikely to be driven by growing weight spectral norms.
W1 & Q1: Norm of zt. The table below shows the [min, max] of the norm ratio ∥z0∥2∥zt∥2 across adapted modules in each selected layer. We observe norm expansion of zt as T increases.
Following the ablation study setup in Sec. 4.5, we investigate whether the performance drop at T=8 can be addressed by mitigating this norm expansion via:
Scaled recursion mitigates the performance degradation at large T, although it yields no clear performance gain compared with T=2. We leave determining when deeper recursion is necessary and how to exploit it effectively as valuable directions for future work.
W2 & Q2. Tab. 6a compares different latent recursion designs when all inputs receive the same expected number of recursion steps. Following the ablation study setup in Sec. 4.5, we compare MoLR with the required fixed 2-step baseline. The results below show that MoLR outperforms this baseline with and without HumanEval.
W3 & Q3. Parameter sharing allows:
In contrast, for the step-specific variant, each step-specific residual ΔWt can only receive supervision from tokens reaching that step. Later step-specific residuals thus receive fewer training signals, potentially hindering convergence, particularly when fine-tuning data and updates are limited. We extend the ablation of step conditioning to T=4 and T=6 following the experimental setup in Sec. 4.5. These results favor parameter sharing at larger T, consistent with the potential benefit of training the shared transformation using supervision from all recursion steps.