- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In this work, the authors combine LoRA fine-tuning with recurrent Transformer computations. The results compare shared versus depth-wise LoRA adaptation for repeated layer groups across different backbones and tasks. The manuscript suggests an advantage of LoRA-adapted recurrent computations for certain combinations of mathematical reasoning tasks and backbones.
Critical strengths
Many PEFT baselines and datasets are considered, demonstrating the advantage of PEFT tuning for recursive Transformer layers.
Critical weaknesses
- The combination of LoRA and recurrent Transformers is not particularly new (see, for example, Fu, T., et al. (2026). Think-at-Hard: Dynamic Looped Transformers for Improved Reasoning. Proceedings of the 43rd International Conference on Machine Learning, in Proceedings of Machine Learning Research), and the proposed method is rather a modest extension of previous recurrent architectures.
- The analysis is limited to a fixed depth and does not quite match the title, which might suggest computational depth K as an adaptive variable rather than a fixed hyper-parameter.
Questions for authors
- The results in Table 4 are worse for the standard LoRA-tuned baseline than in Table 1 for the same backbone. Could you explain the reason for this mismatch?
- What do you mean by "computational depth as an adaptation variable" if the depth is a fixed hyper-parameter in the proposed setup?
- Could you show a table with non-tuned backbone, non-tuned recurrent layers, and tuned recurrent layers comparison?
4 comments
1. Distinction from Think-at-Hard (Fu et al., 2026) and Recurrent LoRA (W1)
Our problem formulation and architecture differ from Think-at-Hard in three key respects:
-
No routing modules or attention changes: Think-at-Hard trains an auxiliary neural decider for dynamic token-level skipping and modifies cross-depth attention (duo-causal attention). RLR adds zero routing parameters and preserves standard attention via residual damping ().
-
Strictly matched trainable budgets: Rather than adding trainable capacity with depth, RLR matches the exact trainable parameter budget of single-pass PEFT, isolating computational depth from trainable parameter count.
-
PEFT parameterization & post-tuning damping: Beyond standard LoRA, we show that recurrent gains depend strongly on the PEFT parameterization across LoRA, DoRA, PiSSA, and MiSS (Table 2) and that residual damping remains necessary after optimization (Table 4).
2. Meaning of "Computational Depth as an Adaptation Variable" (W2, Q2)
Here, "adaptation" refers to model fine-tuning (as in Low-Rank Adaptation), not input-adaptive dynamic halting at inference time. Calling computational depth an adaptation variable means treating recurrent depth (and window width ) as controllable structural design choices during PEFT (analogous to rank or target module selection), evaluated across and (Tables 6 and 11) under a fixed trainable-parameter budget.
3. Apparent Mismatch Between Table 4 and Table 1 (Q1)
There is no numerical mismatch between Table 4 and Table 1. For Llama-3.2-1B, the Baseline row in Table 4 (PIQA 69.70, ARC-E 35.52, ARC-C 30.80, HS 56.19, WG 54.54, CSQA 19.25, OBQA 35.60, Avg 43.09, GSM8K 14.48) is identical across all columns to the Dense row in Table 1. The lower scores directly beneath it (Avg 42.32, GSM8K 13.27) belong to the undamped Additive recurrent ablation.
4. Non-Tuned Backbone vs. Non-Tuned Recurrent vs. Tuned Recurrent (Q3)
Below we compare the untrained backbone (), untrained damped recurrent model (Untrained RLR, , zero-initialized adapters), tuned single-pass LoRA (), and tuned RLR-I / RLR-S () on Llama-3.2-1B and Qwen3-4B (The tuned rows represent the optimal checkpoints identified from our expanded learning rate sweep within the range of 2e-5 to 3e-4):
| Model | Method | GSM8K Strict | GSM8K Flex | Minerva MV | Minerva EM | General Avg |
|---|---|---|---|---|---|---|
| Llama-3.2-1B | Untrained Base | |||||
| Untrained RLR | ||||||
| LoRA | ||||||
| RLR-I | ||||||
| RLR-S | ||||||
| Qwen3-4B | Untrained Base | |||||
| Untrained RLR | ||||||
| LoRA | ||||||
| RLR-I | ||||||
| RLR-S |
-
Recurrence without adaptation does not account for the gains: Repeating layers without tuning slightly degrades
Llama-3.2-1Bacross tasks, and onQwen3-4Blowers GSM8K while partially stabilizing zero-shot Minerva formatting (). -
PEFT on recurrent depth drives consistent improvements: After adaptation,
RLR-IandRLR-Ssubstantially outperform both untrained recurrence and tuned single-passLoRAacross GSM8K, Minerva EM, and General reasoning.
I thank the authors for their response. I agree, there is no mismatch between Table 4 and Table 1, that was my oversight. My core concerns are addressed, I am willing to increase my score.
Thank you for taking the time to read our response and for raising your score. We are glad that our explanation resolved your concerns.
We thank Reviewer bCVt for the time and constructive feedback on our submission. We appreciate your positive assessment of our paper's soundness and our evaluation across diverse PEFT baselines and datasets. Below, we address each of your comments regarding related work, terminology, the baseline numbers in Table 4, and the comparison with untrained recurrent models.