- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Loop Dropout, a regularization method for adapting looped language models with shared LoRA updates. During training, the adapter is disabled at each loop while the backbone remains frozen and active. Retained updates use standard inverted-dropout rescaling and all updates are on for inference. The paper motivates this approach through late-loop bias and evaluates it through downstream benchmarks, loop-level diagnostics, and controlled ablations.
Critical strengths
-
This method is simple and easy to use. Loop Dropout changes the training procedure without any trainable parameters or inference overhead. This is a nice design that allows to regularize adaptation without removing the underlying computation, preserving the frozen backbone at every loop.
-
The gains in mathematical reasoning are significant. For MATH-500, the method surpasses LoRA, achieving 38.87 -> 47.53 on Ouro-1.4B and 42.87 -> 47.20 on Ouro-2.6B. Improvements also appear to hold on the GSM8K, with the main comparisons reported over multiple seeds and with independently selected learning rates.
-
The experimental investigation is extensive. Comparisons to LoRA+, CoTo, and LoRA Dropout provide meaningful evidence beyond a basic baseline comparison. Control for update magnitude, masking granularity, and alternative noise distributions. The rank and parameter sharing experiments also help to determine when the method is most useful.
Critical weaknesses
-
The methodology is incremental in its novelty. Stochastic adapter deactivation exists already in CoTo [1]. Dropout for recurrent computation has a longer history [2]. The paper recognizes these links. Its unique contribution is that it applies coherent masks to repeated applications to a shared adapter and shows their effectiveness in looped language models. This is a useful contribution, but the masking operation and inverse keep probability rescaling is not themselves fundamentally new mechanisms. The paper would be stronger with a clearer explanation of why this particular masking structure is especially effective for shared latent-loop adaptation and what new insight it provides beyond just applying an existing regularization principle that is used in recurrent architectures, to this setting of recurring depth.
-
The rescaling ablation does not prove the need for rescaling during training. The unscaled baseline uses binary masks in training, but during inference all adapter applications are executed fully. Thus its worse performance may indicate a mismatch in the update strength between training and inference. The paper itself notes that the scaling correction can be incorporated into the adapter at inference instead. A comparison against this corrected baseline, with separately tuned learning rates, is needed to distinguish the benefit of consistent scaling from any benefit specific to applying the correction during training.
-
The evidence does not support the conclusion that the reduction in late-loop bias is responsible for the gains in performance. Unlike normal inference, which would evaluate the model with all adapter applications, the single-loop experiments remove most adapter applications, and are thus more similar to the perturbations used during Loop Dropout training. Better performance under this intervention therefore shows robustness to adapter removal but does not, by itself, show that late loop specialization hurts the full model. Moreover, the unscaled version achieves better single-loop losses at the cost of degrading downstream task performance. This disjuncture leaves the paper’s initial mechanistic explanation without answer.
-
The benefit for shared adapters is not well established. Table 6 shows that masking causes substantial improvements for independent per-loop adapters. The independent rank-4 adapters achieve better performance than the shared rank-16 adapters under the same parameter budget on GSM8K. The MATH results support the shared configuration, but the evidence is mixed. This leaves open the question of whether Loop Dropout solves a problem specific to shared updates or provides a more general adapter-regularization benefit.
[1] Zhuang, Z., et al. Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation. ICML, 2025.
[2] Semeniuta, S., Severyn, A., and Barth, E. Recurrent Dropout without Memory Loss. COLING, 2016.
Questions for authors
-
How does Loop Dropout perform compared to binary masking during training and multiplying the learned adapter by the keep probability during inference, with separately tuned learning rates?
-
If one adapter application is removed at a time from the complete model, does the resulting performance change support the same late-loop-bias explanation? How do the authors fit this explanation with the unscaled variant with greater single-loop losses but weaker downstream accuracy?
-
What evidence separates a benefit that is specific to shared adapters from a general benefit of masking? Do the sharing-by-masking interaction in Table 6 replicate across seeds and tasks?
-
Are the instruction-tuning gains and LoopUS/Huginn gains replicable with extra seeds and similar hyperparameter selection?
4 comments
W3: The evidence does not support the conclusion that the reduction in late-loop bias is responsible for the gains in performance. The single-loop experiments remove most adapter applications, and are thus more similar to the perturbations used during Loop Dropout training. Moreover, the unscaled version achieves better single-loop losses at the cost of degrading downstream task performance.
Re: We appreciate this perceptive analysis. We address the single-loop evaluation setting and the unscaled model's loss/accuracy comparison separately.
(1) Gain in performance also appears with all adapter applications enabled and in the leave-one-out test. The table reports single-loop losses and standard-inference accuracy (accuracy in %, mean ± SD over three seeds):
| Method | Loss: only loop 1 | Loss: only loop 4 | GSM8K accuracy | MATH-500 accuracy |
|---|---|---|---|---|
| LoRA | 0.744 | 0.571 | 85.34 ± 0.64 | 38.87 ± 1.67 |
| Loop Dropout | 0.585 | 0.543 | 86.53 ± 0.44 | 47.53 ± 2.61 |
Loop Dropout lowers loss at all four positions in Table 28 and Table 1 shows higher accuracy on both benchmarks in every seed when all adapter applications are enabled.
The leave-one-out experiment in Q2 keeps three adapter applications and all four backbone loops active. Loop Dropout still has higher MATH-500 accuracy after every removal and avoids the collapse seen in one LoRA seed when the final application is removed. These results complement the normal-inference gains above, extending the evidence beyond the single-loop test.
(2) Better single-loop loss but worse downstream accuracy: the role of scale consistency. As already explained in Appendix K.1, the unscaled model's lower single-loop loss shows effective adaptation: the first-loop loss is 0.531, compared with LoRA's 0.744. However, its expected training update is per loop, while standard inference applies . Loop Dropout addresses this mismatch during training through rescaling.
The baseline tested in W2 obtains 87.54 ± 0.73 on GSM8K and 44.07 ± 1.81 on MATH-500 after multiplying the learned adapter by at inference. This supports the scale-consistency explanation. Loop Dropout incorporates this consistency during training and directly supports standard LoRA deployment, with no extra inference cost.
W4: The benefit for shared adapters is not well established. Table 6 leaves open the question of whether Loop Dropout solves a problem specific to shared updates or provides a more general adapter-regularization benefit.
Re: Thank you for this comment. Loop Dropout improves both shared and independent adapters on both benchmarks. Each row of Table 6 compares the same adapter configuration with and without Loop Dropout. The improvement holds in all three training seeds for every row.
We present the comparison with three training seeds below (accuracy in %, mean ± SD; rank is per loop):
| Adapters | Rank | Params | Without Loop Dropout: GSM8K | Without Loop Dropout: MATH-500 | With Loop Dropout: GSM8K | With Loop Dropout: MATH-500 |
|---|---|---|---|---|---|---|
| Shared | 16 | 15.1M | 85.34 ± 0.64 | 38.87 ± 1.67 | 86.53 ± 0.44 | 47.53 ± 2.61 |
| Independent | 4 | 15.1M | 84.99 ± 0.84 | 39.40 ± 2.11 | 87.74 ± 0.16 | 45.60 ± 1.20 |
| Independent | 16 | 60.6M | 84.94 ± 0.42 | 37.33 ± 1.79 | 86.58 ± 0.20 | 43.87 ± 0.90 |
Shared adapters preserve parameter sharing and allow one merged weight to be reused at every loop. This follows the looped backbone's natural reuse of the same weights. At the same per-loop rank of 16, the shared adapter trains 15.1M parameters, compared with 60.6M for independent adapters. With Loop Dropout, the shared configuration also achieves the highest MATH-500 accuracy, 47.53%.
Our focus is improving shared-adapter fine-tuning while retaining this parameter sharing and straightforward inference. Further optimizing independent adapters and investigating whether larger parameter budgets can improve their performance are directions for future work. Their gains with Loop Dropout already demonstrate that the training rule is compatible with independent per-loop updates as well.
We have extended Table 6 to three training seeds and revised Section 4.5 accordingly.
Q1: How does Loop Dropout perform compared to binary masking during training and multiplying the learned adapter by the keep probability during inference, with separately tuned learning rates?
Re: Thank you for this precise question. Please refer to our response to W2 above. The proposed baseline obtains 87.54 ± 0.73 on GSM8K and 44.07 ± 1.81 on MATH-500. Loop Dropout incorporates scale consistency during training and directly produces a standard LoRA adapter, which can be merged into the backbone weights with no extra inference cost.
Q2: If one adapter application is removed at a time from the complete model, does the resulting performance change support the same late-loop-bias explanation? How do the authors fit this explanation with the unscaled variant?
Re: Thank you for proposing this test. We remove one adapter application at a time while all four backbone loops run (accuracy in %, mean ± SD over three training seeds):
| Application removed | LoRA GSM8K | LoRA MATH-500 | Loop Dropout GSM8K | Loop Dropout MATH-500 |
|---|---|---|---|---|
| Loop 1 | 86.10 ± 0.19 | 41.33 ± 1.42 | 87.31 ± 0.36 | 48.93 ± 1.30 |
| Loop 2 | 85.90 ± 0.72 | 43.47 ± 1.60 | 86.13 ± 0.60 | 49.27 ± 1.60 |
| Loop 3 | 86.38 ± 0.49 | 45.47 ± 1.62 | 86.00 ± 0.16 | 46.33 ± 1.53 |
| Loop 4 | 77.84 ± 12.85 | 38.80 ± 9.54 | 85.90 ± 0.77 | 47.40 ± 2.20 |
- Loop Dropout has higher MATH-500 accuracy after every removal, by 0.87 to 8.60 points.
- Loop Dropout avoids LoRA's collapse when the last application is removed. One of the three LoRA models falls to 63.00% on GSM8K and 27.80% on MATH-500, producing the SDs of 12.85 and 9.54 in the last row. Loop Dropout obtains 85.90 ± 0.77 and 47.40 ± 2.20 under the same removal.
For the version without rescaling, our W3 response explains why useful single-loop adaptation can coexist with a mismatch when all applications are enabled. Correcting the inference scale substantially improves its accuracy (W2); Loop Dropout incorporates this consistency during training.
Q3: What evidence separates a benefit that is specific to shared adapters from a general benefit of masking? Do the sharing-by-masking interaction in Table 6 replicate across seeds and tasks?
Re: Thank you for these questions. Please refer to our response to W4 above. The gains replicate across three seeds for both shared and independent adapters on both benchmarks. On MATH-500, the gain is largest for the shared adapter: 8.67 points, compared with 6.20 and 6.53 for independent rank-4 and rank-16 adapters. The shared configuration achieves the highest MATH-500 accuracy while retaining one merged weight for all loops.
Q4: Are the instruction-tuning gains and LoopUS/Huginn gains replicable with extra seeds and similar hyperparameter selection?
Re: Thank you for this question.
- Loop Dropout improves instruction-tuning averages across three training seeds. We would like to kindly note that Tables 2 and 11 report three training seeds for each method, with identical training settings and, for same-seed runs, the same adapter initialization and data order (Appendix F.3). Loop Dropout raises the nine-metric average from 62.62 ± 0.78 to 63.20 ± 0.14, and the code average from 69.88 ± 1.21 to 71.00 ± 0.46.
- The Huginn results already average three training seeds. The Huginn entries of Table 9 are means over three training seeds (Appendix D.2): Loop Dropout raises GSM8K from 59.89 to 60.12 and MATH-500 from 13.00 to 13.80.
- Loop Dropout improves both benchmarks on LoopUS-Qwen3-4B across three seeds. We have extended the evaluation to three training seeds for each method. LoRA and Loop Dropout obtain 84.91 ± 0.62 and 85.49 ± 0.34 on GSM8K, and 37.87 ± 2.83 and 40.93 ± 1.80 on MATH-500: Loop Dropout is higher by 0.58 and 3.07 points.
We have updated Table 9 with the three-seed results.
Summary of Response to Reviewer cXDL
Dear Reviewer cXDL,
Thank you again for your thoughtful review. As the discussion period is drawing to a close, we summarize our responses below for your convenience.
-
Novelty and the masking structure (W1). We identify that sharing adapter parameters does not make the update equally effective at every loop. Loop Dropout switches all adapter modules off together at independently selected loops, while every backbone loop remains active; the comparison with CoTo and the masking-granularity ablations support this design (Tables 3 and 5; clarified in Appendix A.3 and the introduction).
-
Rescaling and the proposed baseline (W2, Q1). We evaluated the proposed baseline with inference-time scaling over three seeds; the results support the scale-consistency explanation in our paper. Loop Dropout builds this consistency into training and directly supports standard merged-LoRA inference with no extra inference cost (Appendix C.2).
-
Evidence beyond the single-loop diagnostic (W3, Q2). We reported gains with all adapter applications enabled and ran the requested leave-one-out test: Loop Dropout has higher MATH-500 accuracy after every removal and avoids the collapse observed in one LoRA seed.
-
Shared and independent adapters (W4, Q3). With Table 6 extended to three seeds, Loop Dropout improves both shared and independent adapters on both benchmarks in every seed, and the shared configuration achieves the highest MATH-500 accuracy with one merged weight reused at every loop (Section 4.5).
-
Reproducibility across seeds (Q4). The instruction-tuning and Huginn results already average three training seeds, and we extended LoopUS-Qwen3-4B to three seeds: Loop Dropout improves both GSM8K and MATH-500 (Table 9).
We would greatly appreciate your feedback before the discussion phase ends on whether our responses have addressed your concerns. If any questions remain, we would be glad to discuss and resolve them.
Best,
Authors of submission-23385
We sincerely appreciate your careful and insightful review, and would like to respond to the raised points.
Re: We appreciate this thoughtful comment. This work identifies late-loop bias in shared-adapter fine-tuning and introduces Loop Dropout to address it through masking at loop positions and training-time rescaling.
(1) Loop Dropout trains earlier adapter applications without always relying on later ones. As discussed in Appendix A.3, CoTo switches adapters at individual Transformer layers and progressively increases their activation probability. In our looped implementation, a layer's switch is reused across loops. Loop Dropout instead switches all LoRA modules off together at independently selected loops, while every backbone loop remains active.
Switching off all LoRA modules at one loop removes the adapter's entire contribution at that position. Earlier applications are trained to remain useful when later ones are absent; later applications also encounter different inputs as earlier applications are switched on or off. The shared adapter therefore learns to work across different combinations of loop positions. Rescaling retained updates keeps their expected strength during training consistent with standard inference (see W2).
Table 3 compares Loop Dropout with CoTo under the same tuning protocol (accuracy in %, mean ± SD over three training seeds):
Table 5 further tests the masking unit: independently switching individual LoRA modules gives 41.13% on MATH-500, compared with 44.00% for Loop Dropout. This supports switching all LoRA modules together at selected loops.
(2) New insight: sharing adapter parameters does not make the update equally effective at every loop. As Figure 1b already shows, the same LoRA update reduces loss by 11.9% at the first loop and 32.4% at the last. Loop Dropout raises these reductions to 30.8–35.9% across all four positions. This motivates training the shared update to work at different positions in the recurrence, beyond choosing which parameters to fine-tune.
We have clarified the distinction from CoTo and the joint role of masking and rescaling in Appendix A.3, and state this insight explicitly in the introduction.
Re: Thank you for this precise comment and for proposing this comparison. Loop Dropout matches the expected training update to ordinary LoRA inference. Masking alone gives an expected training update of qΔ per loop, while standard inference applies Δ. Rescaling retained applications by 1/q makes the expected training update Δ as well (Section 3.2).
We also tried the baseline you propose: binary masking during training followed by multiplying the learned adapter by q=0.5 at inference. Accuracy is in %, mean ± SD over three seeds; Average is the mean of the two displayed benchmark means.