acceptodds
Under review as a conference paper at ICLR 2027

Loop Dropout: Regularizing Shared Updates in Looped Language Models

Abstract

Looped language models separate computational depth from parameter count by repeatedly applying the same transformer block. Adapting these models requires a shared update that remains effective as hidden states evolve throughout the recurrent computation. Our empirical analysis reveals a pronounced late-loop bias in standard low-rank adaptation (LoRA): the shared update is more effective at later loop positions. This imbalance motivates training shared updates under varying combinations of their applications. Randomly omitting adapter applications alone, however, does not improve task performance; it reduces expected update strength during training while leaving inference unchanged. We introduce Loop Dropout, which couples stochastic masking of adapter applications with inverse-survival rescaling to preserve expected update strength and promote effective adaptation across loops. Extensive experiments demonstrate improved mathematical reasoning across model sizes, adapter ranks and training recipes, with benefits extending to general instruction tuning and code generation. Loop Dropout outperforms existing LoRA variants and adapter regularizers, while further analysis shows stronger early-loop adaptation. Every backbone loop remains active, and inference applies the adapter at all loops using standard LoRA without additional trainable parameters or inference computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Language Models, Parameter-Efficient Fine-Tuning, Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cXDLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Loop Dropout, a regularization method for adapting looped language models with shared LoRA updates. During training, the adapter is disabled at each loop while the backbone remains frozen and active. Retained updates use standard inverted-dropout rescaling and all updates are on for inference. The paper motivates this approach through late-loop bias and evaluates it through downstream benchmarks, loop-level diagnostics, and controlled ablations.

Critical strengths
  1. This method is simple and easy to use. Loop Dropout changes the training procedure without any trainable parameters or inference overhead. This is a nice design that allows to regularize adaptation without removing the underlying computation, preserving the frozen backbone at every loop.

  2. The gains in mathematical reasoning are significant. For MATH-500, the method surpasses LoRA, achieving 38.87 -> 47.53 on Ouro-1.4B and 42.87 -> 47.20 on Ouro-2.6B. Improvements also appear to hold on the GSM8K, with the main comparisons reported over multiple seeds and with independently selected learning rates.

  3. The experimental investigation is extensive. Comparisons to LoRA+, CoTo, and LoRA Dropout provide meaningful evidence beyond a basic baseline comparison. Control for update magnitude, masking granularity, and alternative noise distributions. The rank and parameter sharing experiments also help to determine when the method is most useful.

Critical weaknesses
  1. The methodology is incremental in its novelty. Stochastic adapter deactivation exists already in CoTo [1]. Dropout for recurrent computation has a longer history [2]. The paper recognizes these links. Its unique contribution is that it applies coherent masks to repeated applications to a shared adapter and shows their effectiveness in looped language models. This is a useful contribution, but the masking operation and inverse keep probability rescaling is not themselves fundamentally new mechanisms. The paper would be stronger with a clearer explanation of why this particular masking structure is especially effective for shared latent-loop adaptation and what new insight it provides beyond just applying an existing regularization principle that is used in recurrent architectures, to this setting of recurring depth.

  2. The rescaling ablation does not prove the need for rescaling during training. The unscaled baseline uses binary masks in training, but during inference all adapter applications are executed fully. Thus its worse performance may indicate a mismatch in the update strength between training and inference. The paper itself notes that the scaling correction can be incorporated into the adapter at inference instead. A comparison against this corrected baseline, with separately tuned learning rates, is needed to distinguish the benefit of consistent scaling from any benefit specific to applying the correction during training.

  3. The evidence does not support the conclusion that the reduction in late-loop bias is responsible for the gains in performance. Unlike normal inference, which would evaluate the model with all adapter applications, the single-loop experiments remove most adapter applications, and are thus more similar to the perturbations used during Loop Dropout training. Better performance under this intervention therefore shows robustness to adapter removal but does not, by itself, show that late loop specialization hurts the full model. Moreover, the unscaled version achieves better single-loop losses at the cost of degrading downstream task performance. This disjuncture leaves the paper’s initial mechanistic explanation without answer.

  4. The benefit for shared adapters is not well established. Table 6 shows that masking causes substantial improvements for independent per-loop adapters. The independent rank-4 adapters achieve better performance than the shared rank-16 adapters under the same parameter budget on GSM8K. The MATH results support the shared configuration, but the evidence is mixed. This leaves open the question of whether Loop Dropout solves a problem specific to shared updates or provides a more general adapter-regularization benefit.

[1] Zhuang, Z., et al. Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation. ICML, 2025.

[2] Semeniuta, S., Severyn, A., and Barth, E. Recurrent Dropout without Memory Loss. COLING, 2016.

Questions for authors
  1. How does Loop Dropout perform compared to binary masking during training and multiplying the learned adapter by the keep probability during inference, with separately tuned learning rates?

  2. If one adapter application is removed at a time from the complete model, does the resulting performance change support the same late-loop-bias explanation? How do the authors fit this explanation with the unscaled variant with greater single-loop losses but weaker downstream accuracy?

  3. What evidence separates a benefit that is specific to shared adapters from a general benefit of masking? Do the sharing-by-masking interaction in Table 6 replicate across seeds and tasks?

  4. Are the instruction-tuning gains and LoopUS/Huginn gains replicable with extra seeds and similar hyperparameter selection?

4 comments

We sincerely appreciate your careful and insightful review, and would like to respond to the raised points.

W1: The methodology is incremental in its novelty. Stochastic adapter deactivation exists already in CoTo. Dropout for recurrent computation has a longer history. The paper would be stronger with a clearer explanation of why this particular masking structure is especially effective and what new insight it provides.

Re: We appreciate this thoughtful comment. This work identifies late-loop bias in shared-adapter fine-tuning and introduces Loop Dropout to address it through masking at loop positions and training-time rescaling.

(1) Loop Dropout trains earlier adapter applications without always relying on later ones. As discussed in Appendix A.3, CoTo switches adapters at individual Transformer layers and progressively increases their activation probability. In our looped implementation, a layer's switch is reused across loops. Loop Dropout instead switches all LoRA modules off together at independently selected loops, while every backbone loop remains active.

Switching off all LoRA modules at one loop removes the adapter's entire contribution at that position. Earlier applications are trained to remain useful when later ones are absent; later applications also encounter different inputs as earlier applications are switched on or off. The shared adapter therefore learns to work across different combinations of loop positions. Rescaling retained updates keeps their expected strength during training consistent with standard inference (see W2).

Table 3 compares Loop Dropout with CoTo under the same tuning protocol (accuracy in %, mean ± SD over three training seeds):

MethodGSM8KMATH-500
CoTo-on-Ouro85.60 ± 0.7637.80 ± 1.06
Loop Dropout86.53 ± 0.4447.53 ± 2.61

Table 5 further tests the masking unit: independently switching individual LoRA modules gives 41.13% on MATH-500, compared with 44.00% for Loop Dropout. This supports switching all LoRA modules together at selected loops.

(2) New insight: sharing adapter parameters does not make the update equally effective at every loop. As Figure 1b already shows, the same LoRA update reduces loss by 11.9% at the first loop and 32.4% at the last. Loop Dropout raises these reductions to 30.8–35.9% across all four positions. This motivates training the shared update to work at different positions in the recurrence, beyond choosing which parameters to fine-tune.

We have clarified the distinction from CoTo and the joint role of masking and rescaling in Appendix A.3, and state this insight explicitly in the introduction.

W2: The rescaling ablation does not prove the need for rescaling during training. A comparison against the corrected baseline, with separately tuned learning rates, is needed to distinguish the benefit of consistent scaling from any benefit specific to applying the correction during training.

Re: Thank you for this precise comment and for proposing this comparison. Loop Dropout matches the expected training update to ordinary LoRA inference. Masking alone gives an expected training update of qΔq\Delta per loop, while standard inference applies Δ\Delta. Rescaling retained applications by 1/q1/q makes the expected training update Δ\Delta as well (Section 3.2).

We also tried the baseline you propose: binary masking during training followed by multiplying the learned adapter by q=0.5q = 0.5 at inference. Accuracy is in %, mean ± SD over three seeds; Average is the mean of the two displayed benchmark means.

TrainingInferenceGSM8KMATH-500Average
LoRAdirect85.32 ± 0.9240.13 ± 1.6262.73
Loop Dropout w/o rescalingdirect84.71 ± 0.1939.33 ± 1.7062.02
Loop Dropout w/o rescaling×q\times q87.54 ± 0.7344.07 ± 1.8165.81
Loop Dropoutdirect86.53 ± 0.4447.53 ± 2.6167.03
  1. Loop Dropout incorporates scale consistency into training. The baseline result supports the scale-mismatch explanation in our paper. Appendix C.2 shows that the constant can be applied during training or folded into the adapter at inference. Loop Dropout places it in training, so the learned adapter directly uses the unit gate of ordinary LoRA inference.
  2. Loop Dropout has no extra inference cost. The learned update can be merged into the backbone weights and reused at every loop. Inference needs no masks or post-training scale correction, and rescaling introduces no trainable parameters or additional model passes.

W3: The evidence does not support the conclusion that the reduction in late-loop bias is responsible for the gains in performance. The single-loop experiments remove most adapter applications, and are thus more similar to the perturbations used during Loop Dropout training. Moreover, the unscaled version achieves better single-loop losses at the cost of degrading downstream task performance.

Re: We appreciate this perceptive analysis. We address the single-loop evaluation setting and the unscaled model's loss/accuracy comparison separately.

(1) Gain in performance also appears with all adapter applications enabled and in the leave-one-out test. The table reports single-loop losses and standard-inference accuracy (accuracy in %, mean ± SD over three seeds):

MethodLoss: only loop 1Loss: only loop 4GSM8K accuracyMATH-500 accuracy
LoRA0.7440.57185.34 ± 0.6438.87 ± 1.67
Loop Dropout0.5850.54386.53 ± 0.4447.53 ± 2.61

Loop Dropout lowers loss at all four positions in Table 28 and Table 1 shows higher accuracy on both benchmarks in every seed when all adapter applications are enabled.

The leave-one-out experiment in Q2 keeps three adapter applications and all four backbone loops active. Loop Dropout still has higher MATH-500 accuracy after every removal and avoids the collapse seen in one LoRA seed when the final application is removed. These results complement the normal-inference gains above, extending the evidence beyond the single-loop test.

(2) Better single-loop loss but worse downstream accuracy: the role of scale consistency. As already explained in Appendix K.1, the unscaled model's lower single-loop loss shows effective adaptation: the first-loop loss is 0.531, compared with LoRA's 0.744. However, its expected training update is qΔq\Delta per loop, while standard inference applies Δ\Delta. Loop Dropout addresses this mismatch during training through rescaling.

The baseline tested in W2 obtains 87.54 ± 0.73 on GSM8K and 44.07 ± 1.81 on MATH-500 after multiplying the learned adapter by qq at inference. This supports the scale-consistency explanation. Loop Dropout incorporates this consistency during training and directly supports standard LoRA deployment, with no extra inference cost.

W4: The benefit for shared adapters is not well established. Table 6 leaves open the question of whether Loop Dropout solves a problem specific to shared updates or provides a more general adapter-regularization benefit.

Re: Thank you for this comment. Loop Dropout improves both shared and independent adapters on both benchmarks. Each row of Table 6 compares the same adapter configuration with and without Loop Dropout. The improvement holds in all three training seeds for every row.

We present the comparison with three training seeds below (accuracy in %, mean ± SD; rank is per loop):

AdaptersRankParamsWithout Loop Dropout: GSM8KWithout Loop Dropout: MATH-500With Loop Dropout: GSM8KWith Loop Dropout: MATH-500
Shared1615.1M85.34 ± 0.6438.87 ± 1.6786.53 ± 0.4447.53 ± 2.61
Independent415.1M84.99 ± 0.8439.40 ± 2.1187.74 ± 0.1645.60 ± 1.20
Independent1660.6M84.94 ± 0.4237.33 ± 1.7986.58 ± 0.2043.87 ± 0.90

Shared adapters preserve parameter sharing and allow one merged weight to be reused at every loop. This follows the looped backbone's natural reuse of the same weights. At the same per-loop rank of 16, the shared adapter trains 15.1M parameters, compared with 60.6M for independent adapters. With Loop Dropout, the shared configuration also achieves the highest MATH-500 accuracy, 47.53%.

Our focus is improving shared-adapter fine-tuning while retaining this parameter sharing and straightforward inference. Further optimizing independent adapters and investigating whether larger parameter budgets can improve their performance are directions for future work. Their gains with Loop Dropout already demonstrate that the training rule is compatible with independent per-loop updates as well.

We have extended Table 6 to three training seeds and revised Section 4.5 accordingly.

Q1: How does Loop Dropout perform compared to binary masking during training and multiplying the learned adapter by the keep probability during inference, with separately tuned learning rates?

Re: Thank you for this precise question. Please refer to our response to W2 above. The proposed baseline obtains 87.54 ± 0.73 on GSM8K and 44.07 ± 1.81 on MATH-500. Loop Dropout incorporates scale consistency during training and directly produces a standard LoRA adapter, which can be merged into the backbone weights with no extra inference cost.

Q2: If one adapter application is removed at a time from the complete model, does the resulting performance change support the same late-loop-bias explanation? How do the authors fit this explanation with the unscaled variant?

Re: Thank you for proposing this test. We remove one adapter application at a time while all four backbone loops run (accuracy in %, mean ± SD over three training seeds):

Application removedLoRA GSM8KLoRA MATH-500Loop Dropout GSM8KLoop Dropout MATH-500
Loop 186.10 ± 0.1941.33 ± 1.4287.31 ± 0.3648.93 ± 1.30
Loop 285.90 ± 0.7243.47 ± 1.6086.13 ± 0.6049.27 ± 1.60
Loop 386.38 ± 0.4945.47 ± 1.6286.00 ± 0.1646.33 ± 1.53
Loop 477.84 ± 12.8538.80 ± 9.5485.90 ± 0.7747.40 ± 2.20
  • Loop Dropout has higher MATH-500 accuracy after every removal, by 0.87 to 8.60 points.
  • Loop Dropout avoids LoRA's collapse when the last application is removed. One of the three LoRA models falls to 63.00% on GSM8K and 27.80% on MATH-500, producing the SDs of 12.85 and 9.54 in the last row. Loop Dropout obtains 85.90 ± 0.77 and 47.40 ± 2.20 under the same removal.

For the version without rescaling, our W3 response explains why useful single-loop adaptation can coexist with a mismatch when all applications are enabled. Correcting the inference scale substantially improves its accuracy (W2); Loop Dropout incorporates this consistency during training.

Q3: What evidence separates a benefit that is specific to shared adapters from a general benefit of masking? Do the sharing-by-masking interaction in Table 6 replicate across seeds and tasks?

Re: Thank you for these questions. Please refer to our response to W4 above. The gains replicate across three seeds for both shared and independent adapters on both benchmarks. On MATH-500, the gain is largest for the shared adapter: 8.67 points, compared with 6.20 and 6.53 for independent rank-4 and rank-16 adapters. The shared configuration achieves the highest MATH-500 accuracy while retaining one merged weight for all loops.

Q4: Are the instruction-tuning gains and LoopUS/Huginn gains replicable with extra seeds and similar hyperparameter selection?

Re: Thank you for this question.

  • Loop Dropout improves instruction-tuning averages across three training seeds. We would like to kindly note that Tables 2 and 11 report three training seeds for each method, with identical training settings and, for same-seed runs, the same adapter initialization and data order (Appendix F.3). Loop Dropout raises the nine-metric average from 62.62 ± 0.78 to 63.20 ± 0.14, and the code average from 69.88 ± 1.21 to 71.00 ± 0.46.
  • The Huginn results already average three training seeds. The Huginn entries of Table 9 are means over three training seeds (Appendix D.2): Loop Dropout raises GSM8K from 59.89 to 60.12 and MATH-500 from 13.00 to 13.80.
  • Loop Dropout improves both benchmarks on LoopUS-Qwen3-4B across three seeds. We have extended the evaluation to three training seeds for each method. LoRA and Loop Dropout obtain 84.91 ± 0.62 and 85.49 ± 0.34 on GSM8K, and 37.87 ± 2.83 and 40.93 ± 1.80 on MATH-500: Loop Dropout is higher by 0.58 and 3.07 points.

We have updated Table 9 with the three-seed results.

AuthorsOpenReview ↗

Summary of Response to Reviewer cXDL

Dear Reviewer cXDL,

Thank you again for your thoughtful review. As the discussion period is drawing to a close, we summarize our responses below for your convenience.

  1. Novelty and the masking structure (W1). We identify that sharing adapter parameters does not make the update equally effective at every loop. Loop Dropout switches all adapter modules off together at independently selected loops, while every backbone loop remains active; the comparison with CoTo and the masking-granularity ablations support this design (Tables 3 and 5; clarified in Appendix A.3 and the introduction).

  2. Rescaling and the proposed baseline (W2, Q1). We evaluated the proposed baseline with inference-time scaling over three seeds; the results support the scale-consistency explanation in our paper. Loop Dropout builds this consistency into training and directly supports standard merged-LoRA inference with no extra inference cost (Appendix C.2).

  3. Evidence beyond the single-loop diagnostic (W3, Q2). We reported gains with all adapter applications enabled and ran the requested leave-one-out test: Loop Dropout has higher MATH-500 accuracy after every removal and avoids the collapse observed in one LoRA seed.

  4. Shared and independent adapters (W4, Q3). With Table 6 extended to three seeds, Loop Dropout improves both shared and independent adapters on both benchmarks in every seed, and the shared configuration achieves the highest MATH-500 accuracy with one merged weight reused at every loop (Section 4.5).

  5. Reproducibility across seeds (Q4). The instruction-tuning and Huginn results already average three training seeds, and we extended LoopUS-Qwen3-4B to three seeds: Loop Dropout improves both GSM8K and MATH-500 (Table 9).

We would greatly appreciate your feedback before the discussion phase ends on whether our responses have addressed your concerns. If any questions remain, we would be glad to discuss and resolve them.

Best,

Authors of submission-23385

Reviewer_53M2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes LoopDropout, a method for training LoRA adapters specifically for looped transformers. Instead of simply applying LoRA, LoRA is applied at various depths (i.e. the preceeding and following forward passes are not adapted) with scaling applied.

Critical strengths
  • the method is simple and easy to understand
  • the method is efficient
  • looped transformers are an interesting and emerging direction
Critical weaknesses
  • unclear passage in methods: "perturbation controls fix a common rate"
  • the paper only evaluates one particular looped transformer and only at two scales
  • the paper does not show resutls on visual models
  • the papers shows limited LoRA variants and does not show compatibility with more parameter-efficient or (modern/better) LoRA methods
  • there's no baselines of full-finetuning
  • results are reported only with a single rank=16.
  • the gains are small, e.g. on average only + 0.6% in table 2, or ~+1% in table 1 and 3 for GSM8k
Questions for authors
  • how does Fig1b continue when applying more loops?
  • what performance can you get out of plain LoRA by tuning learnign rate, alpha and rank?
  • why is only Ouro evaluated? how is the performance on Huginn, loopcoder, nanbeige?
  • does this work for looped visual transformers like Raptor?
  • does this work for BERT-style transformers like ALBERT?
  • How does this work with more modern LoRA methods like DoRA or VeRA (they're cited in the appendix, but not compared/used)?
5 comments

We sincerely appreciate your careful review and your recognition of the simplicity and efficiency of the method, and would like to respond to the raised points.

W1: Unclear passage in methods: "perturbation controls fix a common rate".

Re: We apologize for the confusion and appreciate the opportunity to clarify. The sentence means that the training variants of Table 5, including our Loop Dropout, all use a fixed learning rate to ensure a fair comparison. Meanwhile, for the main comparisons (Tables 1 and 3, and the rank and sharing studies), we tune the learning rate of all methods separately over five learning rates.

We have rewritten the sentence in Section 4.1 accordingly.

W2: The paper only evaluates one particular looped transformer and only at two scales.

Re: Thank you for this comment. We would like to kindly note that, in addition to Ouro-1.4B and Ouro-2.6B, the paper evaluates LoopUS-Qwen3-4B (eight loops) and Huginn (32 loops): the results are introduced in Section 4.2 and reported in Appendix F.1 (Table 9).

Additionally, following your Q3, Nanbeige4.2-3B-Base and LoopCoder-V2 are added under the protocol of Table 1 (MetaMath-GSM recipe, rank-16 adapters, learning rate tuned for each method over {1.25,2.5,5,10,20}×10−5\lbrace 1.25, 2.5, 5, 10, 20 \rbrace \times 10^{-5} on the development split). The table below lists all six models (accuracy in %, mean over three training seeds):

ModelScaleLoopsGSM8K: LoRAGSM8K: Loop DropoutMATH-500: LoRAMATH-500: Loop Dropout
Ouro-1.4B1.4B485.3486.5338.8747.53
Ouro-2.6B2.6B487.6288.7342.8747.20
LoopUS-Qwen3-4B4B884.9185.4937.8740.93
Huginn3.5B3259.8960.1213.0013.80
Nanbeige4.2-3B-Base3B284.3386.6841.6738.80
LoopCoder-V27B285.8287.9541.6745.20

Overall, across the six looped language models and two benchmarks, Loop Dropout has the higher mean in 11 of the 12 comparisons.

W3: The paper does not show results on visual models.

Re: Thank you for this suggestion. The paper focuses on looped language models; following your questions, we also ran LoRA and Loop Dropout on Raptor here. For both methods we tune the learning rate over {1,2,5,10,20}×10−4\lbrace 1, 2, 5, 10, 20 \rbrace \times 10^{-4}, with three seeds, on four classification datasets (test top-1 accuracy in %, mean ± SD, at the best learning rate of each method):

DatasetLoRALoop Dropout
CIFAR-10092.46 ± 0.1992.62 ± 0.14
Flowers-10299.51 ± 0.0999.58 ± 0.02
Oxford-IIIT Pets95.62 ± 0.1095.98 ± 0.11
DTD82.66 ± 0.2883.44 ± 0.20

The accuracy of LoRA is already high on these datasets (for example, 99.51% on Flowers-102), yet Loop Dropout has the higher mean at every one of the five learning rates on every dataset (20 of 20 comparisons).

W4: The paper shows limited LoRA variants and does not show compatibility with more parameter-efficient or (modern/better) LoRA methods.

Re: We appreciate this suggestion. Loop Dropout improves MATH-500 performance when combined with both DoRA and VeRA. Following your Q6, we evaluated each method with and without Loop Dropout under the MetaMath-GSM recipe.

Loop Dropout switches off the method's entire adapter update at selected loops with p=0.5p = 0.5 and rescales the retained updates. Every backbone loop remains active, and inference follows the original adapter method.

DoRA uses rank 16, and we tune the learning rate over {1.25,2.5,5,10,20}×10−5\lbrace 1.25, 2.5, 5, 10, 20 \rbrace \times 10^{-5}. VeRA uses r=1024r = 1024 and trains 0.69M parameters, and we tune the learning rate over {2.5,5,10,20,40,80}×10−4\lbrace 2.5, 5, 10, 20, 40, 80 \rbrace \times 10^{-4}. Each method receives the same tuning budget with and without Loop Dropout; configurations are selected on the development split. Accuracy is in %, mean ± SD over three training seeds:

MethodGSM8KMATH-500
DoRA85.39 ± 0.3138.07 ± 1.62
DoRA + Loop Dropout86.73 ± 0.3542.80 ± 1.56
VeRA86.50 ± 0.2744.53 ± 1.15
VeRA + Loop Dropout86.48 ± 0.4949.00 ± 0.53
  • Loop Dropout improves DoRA on both benchmarks in every seed. The mean gains are 1.34 points on GSM8K and 4.73 points on MATH-500.
  • VeRA + Loop Dropout reaches 49.00% on MATH-500 with 0.69M trained parameters. Loop Dropout adds 4.47 points on MATH-500, with an improvement in every seed, while GSM8K stays on par (86.50 and 86.48).

We have added these results in Appendix M.

W5: There's no baselines of full-finetuning.

Re: Thank you for this helpful suggestion. We have added full fine-tuning as an upper reference (accuracy in %, mean ± SD over three training seeds):

MethodTrained parametersGSM8KMATH-500
Full fine-tuning1.4B85.52 ± 0.2748.00 ± 0.53
LoRA15.1M85.34 ± 0.6438.87 ± 1.67
Loop Dropout15.1M86.53 ± 0.4447.53 ± 2.61

With about 1% of the parameters trained, Loop Dropout exceeds full fine-tuning on GSM8K and is within 0.5 points on MATH-500.

We have added the full fine-tuning reference in Appendix N.

W6: Results are reported only with a single rank=16.

Re: We appreciate this comment. We would like to kindly note that results for ranks 4, 16, 64 and 128 are included in Section 4.6 (Figure 3) and Appendix J.1 (Table 20); the learning rate is tuned separately for each method at every rank. For convenience, we reproduce Table 20 below (accuracy in %, mean ± SD over three training seeds).

RankGSM8K: LoRAGSM8K: Loop DropoutMATH-500: LoRAMATH-500: Loop Dropout
485.82 ± 0.2086.38 ± 0.5039.00 ± 1.2546.07 ± 1.17
1685.34 ± 0.6486.53 ± 0.4438.87 ± 1.6747.53 ± 2.61
6485.44 ± 1.1285.87 ± 0.4837.87 ± 1.7040.73 ± 1.42
12886.10 ± 0.2986.48 ± 0.2239.27 ± 2.0041.60 ± 1.39

Loop Dropout has the higher mean on both benchmarks at every rank, with MATH-500 gains of 7.07, 8.67, 2.87 and 2.33 points at ranks 4, 16, 64 and 128.

W7: The gains are small for GSM8K.

Re: Thank you for raising this point. We answer it directly with the numbers.

  1. Loop Dropout reduces GSM8K errors by 8–9% at both model sizes. LoRA is already at 85–88% accuracy; Loop Dropout adds 1.19 and 1.11 points on Ouro-1.4B and Ouro-2.6B, with an improvement in every seed at both sizes. When adapters trained at four loops are run at more loops, the gain grows from 1.90 points at four loops to 4.75 points at eight (Figure 4).
  2. Loop Dropout adds no trainable parameters or inference computation. With about 1% of the parameters trained, Loop Dropout exceeds full fine-tuning on GSM8K and is within 0.5 points on MATH-500 (our response to W5).
  3. Loop Dropout improves instruction-tuning averages and reduces variation across seeds. The six-benchmark average in Table 2 rises from 60.98 to 61.51, and the nine-metric average of Table 11 rises from 62.62 to 63.20, with a much smaller spread across seeds (SDs of 0.09 and 0.14, compared with 0.66 and 0.78).

Q1: How does Fig. 1b continue when applying more loops?

Re: Thank you for this question. Late-loop bias persists at eight loops, and Loop Dropout improves adaptation at every position. We repeated the Figure 1b diagnostic after training adapters at four, six and eight loops with the direct GSM8K recipe, using three seeds per method.

The table shows the eight-loop results. Only the indicated adapter application is enabled, while all eight backbone loops run. We report the reduction in final-loop teacher-forced loss relative to the frozen model run at the same depth (in %, mean over three training seeds):

MethodLoop 1Loop 2Loop 3Loop 4Loop 5Loop 6Loop 7Loop 8
LoRA10.014.516.011.411.012.716.527.9
Loop Dropout18.329.031.533.234.434.634.333.9

LoRA's last application remains its most effective, reducing loss by 27.9%, compared with 10.0% at the first loop. Loop Dropout gives a larger loss reduction at all eight positions in every seed. The six-loop experiment shows the same pattern: LoRA is most effective at the last loop, and Loop Dropout is higher at every position in every seed.

We have added the diagnostics at four, six and eight loops to Appendix K.

Q2: What performance can you get out of plain LoRA by tuning learning rate, alpha and rank?

Re: Thank you for this detailed question. Our main comparisons give every method the same learning-rate tuning budget and development-set selection procedure. Each method receives five learning-rate candidates. The rank study likewise evaluates LoRA and Loop Dropout at the same four ranks, with the same five-rate grid at each rank.

As already reported in Section 4.6 and Appendix J.1 (Table 20), at each rank in {4,16,64,128}\lbrace 4, 16, 64, 128 \rbrace, we tune the learning rate over {1.25,2.5,5,10,20}×10−5\lbrace 1.25, 2.5, 5, 10, 20 \rbrace \times 10^{-5} and report three training seeds. Across these ranks, tuned LoRA obtains 85.34–86.10 on GSM8K and 37.87–39.27 on MATH-500.

Rank-4 Loop Dropout exceeds LoRA at every tested rank on both benchmarks. It obtains 86.38 / 46.07 with a quarter of the parameters of the rank-16 adapter; our W6 response reproduces the full rank comparison.

In our early experiments, varying α\alpha led to less stable training, so we use the same scaling convention, α/r=2\alpha / r = 2, for LoRA and Loop Dropout.

The paper also evaluates rsLoRA, which uses α/r\alpha / \sqrt{r} instead of α/r\alpha / r. Under the direct GSM8K recipe, it obtains 79.8 ± 0.9, compared with 79.9 ± 1.0 for LoRA and 81.8 ± 0.5 for Loop Dropout (Table 24).

Q3: Why is only Ouro evaluated? How is the performance on Huginn, LoopCoder, Nanbeige?

Re: Thank you for these questions. Please refer to our response to W2 above. The paper also evaluates Huginn and LoopUS-Qwen3-4B (Table 9), and we add LoopCoder-V2 and Nanbeige4.2-3B-Base in our response to W2. Loop Dropout has the higher mean in 11 of the 12 comparisons across the six models and two benchmarks.

Q4: Does this work for looped visual transformers like Raptor?

Re: We appreciate this question. Please refer to our response to W3 above, where we fine-tune Raptor on four classification datasets: Loop Dropout has the higher mean at every learning rate on every dataset, covering all 20 comparisons.

Q5: Does this work for BERT-style transformers like ALBERT?

Re: Thank you for this question. As stated in the paper, this work focuses on looped language models, which are modern generative language models. ALBERT is a small encoder outside this generative setting and is not designed for generation tasks such as GSM8K. As requested, we also tested ALBERT on GLUE with LoRA and with Loop Dropout, and the basic conclusion is that the two methods are on par (83.34 vs 83.17). Our conjecture is that, for a model of such a small scale, parameter-efficient fine-tuning (PEFT) methods such as LoRA may offer less benefit: ALBERT-base has only 12M parameters in total, fewer than the 15.1M parameters of our rank-16 adapter on Ouro-1.4B.

Q6: How does this work with more modern LoRA methods like DoRA or VeRA (they're cited in the appendix, but not compared/used)?

Re: Thank you again for this suggestion. Please refer to our response to W4 above, where Loop Dropout is combined with DoRA and with VeRA: Loop Dropout improves MATH-500 by 4.73 points with DoRA and by 4.47 points with VeRA, with an improvement in every seed.

Reviewer_SL57OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at how to fine-tune looped language models with shared LoRA adapters. Motivated by an empirical analysis of adaptation across loop positions, the authors propose Loop Dropout, which couples stochastic masking of adapter applications with inverse-survival rescaling to preserve expected update strength and encourage effective adaptation across loops. The paper also investigates several other factors including adapter regularizers, masking controls, recurrence depths and etc.

Critical strengths
  1. The method is easy to understand and seems straightforward to implement. It adds no trainable parameters or inference computation, which makes the gains useful in practice.
  2. Extensive experiments demonstrate the effectiveness of proposed method on mathematical and coding tasks.
  3. The paper makes a reasonable effort to understand why the method helps. The experimental analysis is fairly thorough on the mathematical tasks, especially the rank/depth studies and the breakdown of MATH-500 gains.
Critical weaknesses
  1. The paper evaluates non-mathematical tasks after instruction tuning. The results are encouraging for code generation, while improvements on knowledge, truthfulness, and instruction following are smaller and mixed.
  2. The non-mathematical results suggest that the method may help on other tasks, but they do not really explain why it works or whether the same explanation applies across tasks.
  3. I am less convinced by the explanation linking better early-loop adaptation to the final accuracy gains, an established conclusion is expected, rather than a plausible explanation. In addition, the role of rescaling could be clearer. It is unclear that how much of the benefit comes from correcting the inference scale and how much comes from different training dynamics.
Questions for authors

More conclusive analysis is expected, especially on non-mathematical tasks.

3 comments

We sincerely appreciate your careful reading and encouraging review, and would like to respond to the raised points.

W1: The paper evaluates non-mathematical tasks after instruction tuning. The results are encouraging for code generation, while improvements on knowledge, truthfulness, and instruction following are smaller and mixed.

Re: Thank you for this careful reading of the instruction-tuning study. In this study, we tuned models on a 100k-example sample from Tülu 2 and evaluated on nine metrics, with three training seeds per method.

  1. Loop Dropout is higher on seven of the nine metrics, including TruthfulQA MC2 (47.32 →\rightarrow 48.19), as shown in Table 11.
  2. Loop Dropout is more stable across training seeds: its SD is lower on seven of the nine metrics and on all three averages (0.14 for the nine-metric average, compared with 0.78 for LoRA).

Even though the improvements of Loop Dropout on knowledge, truthfulness and instruction following are less pronounced than those on mathematics or code generation, this may reflect the role of regularization. Loop Dropout is a regularization method tailored for looped language models: it can help the model reach a better optimum on the basis of its existing capabilities, rather than bringing entirely new knowledge. This may help explain the larger improvements on reasoning-heavier tasks such as mathematics or coding.

W2: The non-mathematical results suggest that the method may help on other tasks, but they do not really explain why it works or whether the same explanation applies across tasks.

Re: Thank you for this constructive suggestion. We reproduced the single-loop diagnostic of Figure 1b on the six instruction-tuned checkpoints of Table 11. The trained adapter is enabled at one loop only while all four backbone loops run, and we report how much the final teacher-forced loss falls relative to the frozen model (in %, mean ± SD over three training seeds):

MethodOnly loop 1Only loop 2Only loop 3Only loop 4
LoRA13.07 ± 0.3018.30 ± 0.1118.20 ± 0.0219.82 ± 0.14
Loop Dropout23.75 ± 0.7629.87 ± 0.4330.23 ± 0.4227.70 ± 0.28

Loop Dropout improves adaptation at every loop after instruction tuning. Applied alone, the LoRA update lowers the loss by 13.07–19.82%, least at the first loop. Loop Dropout is higher at every loop in every seed, by 7.6 to 12.5 points. Across the twelve data sources and four loop positions, Loop Dropout achieves a larger loss reduction in 47 of the 48 comparisons.

We have added this diagnostic, with the breakdown by data source, to Appendix F.3.

W3: The explanation linking better early-loop adaptation to the final accuracy gains is plausible rather than established. In addition, the role of rescaling could be clearer: it is unclear how much of the benefit comes from correcting the inference scale and how much comes from different training dynamics.

Re: We appreciate this perceptive comment, and answer its two parts in turn.

(1) The link between loop-level adaptation and accuracy. As already explained in the introduction, Figure 1b motivates our design: ordinary LoRA reduces loss by 11.9% when used at the first loop, compared with 32.4% at the last.

Loop Dropout trains the adapter to contribute at each loop. We randomly switch off the adapter at different loops while keeping every backbone loop active. This encourages earlier adapter applications to remain useful when later ones are absent. Figure 1b shows larger loss reductions of 30.8–35.9% across all four positions.

Accuracy also improves when all adapter applications are enabled. Table 1 shows gains on both benchmarks in every training seed: GSM8K rises from 85.34 to 86.53 and MATH-500 from 38.87 to 47.53. The improvement therefore also appears when the complete model generates answers normally.

(2) Rescaling provides scale consistency with standard LoRA inference. Masking alone gives an expected update of qΔq\Delta per loop during training, while ordinary LoRA inference applies Δ\Delta. Loop Dropout rescales retained updates by 1/q1/q so that their expectation is also Δ\Delta during training (Section 3.2).

We also tried binary masking during training followed by multiplying the learned adapter by q=0.5q = 0.5 at inference. Accuracy is in %, mean ± SD over three seeds; Average is the mean of the two displayed benchmark means.

TrainingInferenceGSM8KMATH-500Average
LoRAdirect85.32 ± 0.9240.13 ± 1.6262.73
Loop Dropout w/o rescalingdirect84.71 ± 0.1939.33 ± 1.7062.02
Loop Dropout w/o rescaling×q\times q87.54 ± 0.7344.07 ± 1.8165.81
Loop Dropoutdirect86.53 ± 0.4447.53 ± 2.6167.03

Loop Dropout builds scale consistency into training. The baseline result supports the scale-mismatch explanation in our paper. Appendix C.2 shows that the constant can be applied during training or folded into the adapter at inference; Loop Dropout produces an adapter ready for ordinary LoRA inference directly.

Loop Dropout has no extra inference cost. The learned update can be merged into the backbone weights and reused at every loop, with no inference masks or post-training scale correction. Rescaling adds no trainable parameters or additional model passes.

Q1: More conclusive analysis is expected, especially on non-mathematical tasks.

Re: Thank you for raising this point. Please refer to our responses to W1-3 above.

AuthorsOpenReview ↗

Summary of Response to Reviewer SL57

Dear Reviewer SL57,

We sincerely appreciate your thoughtful review and your recognition of the simplicity and effectiveness of Loop Dropout. As the discussion period is drawing to a close, we summarize our responses below for your convenience.

  1. Results after instruction tuning (W1). Over three training seeds, Loop Dropout improves seven of the nine metrics, reduces the standard deviation on seven, and raises the nine-metric average from 62.62 ± 0.78 to 63.20 ± 0.14 (Table 11).

  2. Analysis on non-mathematical tasks (W2, Q1). We added the single-loop diagnostic on the instruction-tuned checkpoints (Appendix F.3): Loop Dropout improves adaptation at every loop in every seed, and in 47 of the 48 comparisons across twelve data sources and four loop positions.

  3. Loop-level adaptation and final accuracy (W3). We clarified how Loop Dropout trains the adapter to contribute at each loop. Alongside the improved single-loop losses, it achieves higher GSM8K and MATH-500 accuracy in every training seed (Table 1).

  4. The role of rescaling (W3). We evaluated binary masking followed by inference-time scaling by the keep probability; the results support the scale-consistency explanation in our paper. Loop Dropout incorporates this consistency during training and directly supports standard LoRA inference with no extra inference cost (Appendix C.2).

We would greatly appreciate your feedback before the discussion phase ends on whether our responses have addressed your concerns. If any questions remain, we would be happy to provide further clarification.

Best,

Authors of submission-23385

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

Dear Reviewers, ACs, SACs, and PCs,

We sincerely thank all reviewers for their time and detailed feedback. We appreciate the recognition of Loop Dropout's simplicity and efficiency (Reviewers SL57, 53M2, and cXDL), and its empirical effectiveness and extensive experimental analysis (Reviewers SL57 and cXDL).

During the discussion period, we responded directly to every weakness and question raised by the three reviewers, with additional experiments and a revised paper. The main points includes:

  • Evaluation on more looped models (Reviewer 53M2). We added LoopCoder-V2 and Nanbeige4.2-3B-Base, for six looped language models in total, and the looped vision model Raptor.
  • Other LoRA methods and full fine-tuning (Reviewer 53M2). We combined Loop Dropout with DoRA and VeRA (Appendix M) and added a full fine-tuning reference (Appendix N).
  • Analysis on non-mathematical tasks (Reviewer SL57). We added the single-loop diagnostic on the instruction-tuned checkpoints (Appendix F.3).
  • Loop-level adaptation and final accuracy (Reviewers SL57 and cXDL). We clarified how Loop Dropout trains the adapter to contribute at each loop, and ran the requested leave-one-out test.
  • The role of rescaling (Reviewers SL57 and cXDL). We evaluated the proposed baseline with inference-time scaling; Loop Dropout builds this scale consistency into training (Appendix C.2).
  • Shared and independent adapters; training seeds (Reviewer cXDL). We extended Table 6 and the LoopUS-Qwen3-4B results in Table 9 to three training seeds.
  • Novelty and the masking structure (Reviewer cXDL). We clarified the insight behind Loop Dropout and its difference from CoTo (introduction and Appendix A.3).

We thank Reviewer 53M2 for their follow-up, for their recognition of the strong gains of Loop Dropout with VeRA, and for increasing their score. As we have not yet heard back from Reviewers SL57 and cXDL, we have posted a summary of our response to each of them, to help the AC quickly see how every point was addressed. We believe our responses and revisions directly resolve their main concerns.

We would be very happy to provide any further clarification. Thank you again for your time and consideration.

Authors of submission-23385

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.