acceptodds
Under review as a conference paper at ICLR 2027

Learning to Loop: Reinforcement Learning for Looped Transformers

Abstract

Looped Transformers can increase computation by repeatedly applying existing layers, but most approaches use fixed repetition patterns across inputs. We ask whether a model can instead learn, for each question, when additional computation is useful and where it should occur. We introduce a lightweight controller that observes the model's hidden states and decides after each layer whether to repeat the layer or proceed forward. The controller is trained by comparing alternative computation routes for the same question using signals from final-answer correctness and the model's likelihood of a reference solution, while the language model itself remains frozen. Unlike fixed-depth looping, our method allows different questions to use different numbers and locations of repetitions under a maximum computation budget, while retaining fixed-depth strategies as special cases. We evaluate the approach across three 7–8B language models and four mathematical reasoning benchmarks: GSM8K and MathQA for in-domain evaluation, and SVAMP and MATH-500 for frozen transfer. Adaptive routing improves reasoning performance over fixed repetition schedules while allocating different amounts and locations of computation across questions. Controlled comparisons with exact-depth routing and shuffled question–route assignments further indicate that the gains arise from learning question-specific computation paths rather than simply executing more layers.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Transformers, Reinforcement Learning, Dynamic Depth, Layer Routing, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_dEDPOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies adaptive repetition of existing Transformer blocks for mathematical reasoning. After a route-aware LoRA adaptation stage, a frozen language-model backbone is paired with a controller that observes updated hidden states and chooses whether to repeat the current block or advance. The controller is trained using on-policy sampled computation routes, a leave-one-out route advantage, and a mixed terminal reward combining answer correctness with the likelihood of a reference solution. Experiments use three 7–8B models, two training/evaluation tasks (GSM8K and MathQA), and two frozen-transfer tasks (SVAMP and MATH-500). The paper compares adaptive routing with fixed routes, exact-depth learned routes, shuffled route assignments, initial-feature routing, and search-supervised alternatives. It reports five-seed results and measures inference latency, KV-cache memory, and controller training costs.

Critical strengths

S1. Controlled evidence for question-specific computation routes. The paper compares the method against exact-depth learned routing, searched fixed schedules, modal routes, and shuffled question-to-route assignments. In particular, shuffled routes reduce accuracy despite preserving route multiset and execution depth, supporting a genuine benefit from question-dependent assignment rather than merely extra block executions.

S2. Reproducible breadth and meaningful cost accounting. The study includes three backbone families, five independent runs, in-domain and frozen-transfer evaluations, component ablations, and measured inference costs. The Qwen2.5-7B/GSM8K free controller reaches 94.9% at mean 2.16L, versus 93.5% for compulsory 3L, and the appendix reports concrete latency and KV-storage reductions relative to 3L routing.

Critical weaknesses

Practical efficiency and generalization remain unresolved. The approach needs a route-aware LoRA adaptation stage, controller imitation initialization, and up to 25,600 route evaluations per actor before deployment. At inference, repeated blocks create additional per-occurrence KV state and latency. For Qwen2.5-7B/GSM8K, free routing improves accuracy from 92.1% to 94.9% but increases measured per-question time from 6.2 s to 13.1 s compared with native execution. Thus, although free routing is more efficient than compulsory 3L, the practical advantage over a non-looped model trained or decoded under comparable total compute is not established. Furthermore, the two transfer datasets remain within mathematical reasoning, and the controller fixes its route before answer generation rather than adapting to intermediate reasoning states.

Questions for authors

Q1. How robust is adaptive routing outside mathematical reasoning, and can the controller adapt its route during decoding rather than committing to a single prompt-derived route?

Q2. The mixed reward requires a reference-solution likelihood scoring pass. Is the 0.8-point Qwen2.5-7B/GSM8K gain over correctness only RL worth the additional 16 GPU-hours per seed reported for reference scoring, and does this trade-off hold across the other models and datasets?

Reviewer_jYJHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how Transformers can adaptively decide which layers to repeat and how many times to repeat them based on the input question. The authors introduce a lightweight controller on top of a LoRA-adapted and subsequently frozen language model. The model executes layers in a fixed monotonic routing order, with the controller deciding after each block execution whether to "REPEAT" the current layer or "ADVANCE" to the next layer. At the final layer, the controller chooses whether to continue repeating or stop. The controller observes the current residual state, the latest block-induced update, the state before entering the current layer, the mean prompt representation, the layer index, and the remaining computation budget. It is trained using a PPO-style policy-gradient method.

During training, multiple complete routes are sampled for the same question. Each route receives a terminal reward combining final-answer correctness with the teacher-forced likelihood of a reference solution, and a leave-one-out reward baseline is constructed using the other routes sampled for the same question. The language model and its LoRA weights remain frozen during controller training. At deployment, a single route is selected based on the prompt and reused throughout the entire answer generation process.

The experiments cover three 7–8B instruction-tuned models (Llama-3.1-8B, Qwen2.5-7B, and Qwen3-8B) and four mathematical reasoning datasets. GSM8K and MathQA are used for source-task training, while SVAMP and MATH-500 are used for frozen transfer evaluation. The authors compare native execution, fixed repetition strategies (Sequence/Cycle), validation-selected fixed routes, exact-depth RL, free-choice RL, and several controlled variants, including modal-route, shuffled-route, initial-feature routing, and search-supervised routing.

Critical strengths
  • A clearly defined routing formulation with a relatively comprehensive set of controlled experiments. The authors explicitly restrict the routing space to monotonic repeat/advance decisions and provide detailed descriptions of the computation budget, action masks, cache handling, and controller inputs. The inclusion of exact-depth, free-choice, modal-route, shuffled-route, initial-feature, and search-supervised comparisons demonstrates a serious effort to disentangle the effects of execution depth, question–route matching, state feedback, and the training procedure. This makes the empirical evaluation more convincing than a comparison against only one fixed repetition baseline.

  • Reasonably broad experimental coverage, including transfer evaluation and inference-cost measurements. The results cover three different 7–8B models, two source tasks, and two frozen-transfer tasks. The paper also reports realized execution depth, end-to-end latency, and occurrence-specific KV-cache requirements. In particular, the shuffled-route experiments and cross-dataset transfer results provide some evidence supporting the claim that the controller is doing more than simply executing additional layers.

Critical weaknesses
  • The core methodological novelty appears incremental relative to existing dynamic layer routing methods, and fair direct comparisons are insufficient. Prior works such as Dr.LLM and PoLar have already investigated input-dependent execution policies that skip, execute, or repeat pretrained layers. The main distinction of this paper is the use of sequential PPO and online route-level rewards, but it is not sufficiently demonstrated whether this difference constitutes a substantial methodological contribution. The reported Dr.LLM reference uses a substituted adapted checkpoint, a different structural depth limit, and a reimplementation, which the authors themselves acknowledge is not a fully controlled comparison. PoLar is not directly compared. Furthermore, the search-supervised baseline lacks sufficient details regarding PUCT allocation, search hyperparameters, and the actual number of candidate routes explored per question. Consequently, it remains unclear whether the improvements originate from PPO, updated-state feedback, the routing space, or differences in baseline implementations. The fixed-route search also evaluates only a limited set of 64 candidates, which may be a relatively weak approximation of the best input-independent routing policy.

  • The contribution of the routing controller is not fully isolated, and the computation comparisons are not sufficiently rigorous. Before controller training, the language model has already undergone LoRA adaptation using randomly sampled repetition routes. Therefore, all reported improvements are obtained on a backbone specifically adapted to tolerate or exploit repeated layer execution. Without ablations involving standard single-depth adaptation, adaptation without route augmentation, or different adaptation-route distributions, it is difficult to determine how much of the improvement should be attributed to the controller rather than the additional adaptation stage. Moreover, comparisons among free-choice, exact-depth, and common-depth policies are primarily based on block execution depth, without strict alignment of generated answer lengths, total token counts, FLOPs, batching effects, or end-to-end throughput in the main results. The reported latency measurements cover only a limited subset of configurations. Many performance improvements are relatively small, approximately 0.5–1.5 percentage points, yet the paper reports only five seeds per setting without paired confidence intervals, significance tests, or per-seed paired differences. Finally, using the likelihood of a single reference solution as an auxiliary reward is not fully aligned with the final greedy-decoding deployment objective, and the route-level terminal reward cannot identify which specific repetition decisions contribute to the observed improvements.

Questions for authors
  1. Could the authors provide direct comparisons against Dr.LLM and PoLar using exactly the same adapted checkpoint, routing depth limit, training questions, route evaluation budget, and inference protocol? If their full implementations cannot be used, please at least clarify how the proposed controller differs from these methods in terms of action space, supervision signals, and search/training budgets, and report a comparison between PPO and search/supervised imitation under an identical routing space.

  2. How much do the results depend on route-augmented LoRA adaptation? Please compare at least the following settings:

  • Adaptation using only the native LL-depth execution path;
  • Adaptation using fixed Sequence/Cycle routes;
  • Adaptation using the proposed random repetition route augmentation;
  • No LoRA adaptation. The results should be reported for both the learned controller and fixed-routing baselines, particularly on the transfer datasets.
  1. Could the authors report per-seed paired accuracy differences, 95% confidence intervals, and statistical significance tests? This is particularly important for the relatively small gains of free-choice routing over exact-depth and common-depth routing. In addition, please report the distributions of generated token counts, total FLOPs per question, or actual throughput, and provide comparisons under matched end-to-end latency or matched total computation budgets.

  2. In the PPO implementation, the layer embeddings are part of the policy parameters while also being included in the observations. During optimization, the current policy uses observations reconstructed with the current layer embeddings, whereas the old-policy log-probabilities were obtained using the previous embeddings. Please clarify how the probability ratio is implemented, whether a distinction is made between the underlying environment state and the observation encoder, whether advantage normalization is applied, and whether entropy or KL regularization is used. An ablation with frozen layer embeddings or fixed one-hot layer indices would help determine whether this implementation detail affects training stability.

  3. For the search-supervised actor, please specify the PUCT search depth, rollout allocation per question, tree-node expansion strategy, treatment of duplicate routes, and reward used during search. Matching only the total number of completed-route evaluations does not necessarily ensure comparable search quality or exploration coverage across questions.

Suggestions for future work, which are not by themselves decisive reasons for rejection in this review:

  • Extend routing decisions to intermediate states during answer generation, rather than selecting a single fixed route based only on prompt prefill.
  • Investigate non-monotonic skip/repeat/return routing, cross-layer sharing, or finer-grained token-level routing to determine whether the current monotonic routing constraint is a major performance bottleneck.
Reviewer_oytrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses RL to train a routing controller for a looped transformer, deciding if to repeat the same computation of a transformer block, up to a manually selected depth cap. The controller is a lightweight one-hidden-layer network that takes an array of engineered features including hidden states (of the question tokens) and routing budget and outputs sequential decisions to maximize a reward of final-answer accuracy and reference solution likelihood combined. Experimental results on four math datasets and three 7–8B language models show consistent gains over other routing heuristics and vanilla models.

Critical strengths
  • Adaptive computation for the Transformer architecture is an interesting and practical topic
  • The proposed method shows consistent gains on several math datasets under both in-domain and transfer settings
Critical weaknesses
  • The main novelty of the paper is to use RL for looped transformer routing as a sequential decision problem. However, the necessity of RL is unclear without for example an expert iteration baseline. The observation of the actor contains an array of hand engineered features without any ablation studies of feature importance.
  • Despite seeing consistent gains on a bunch of datasets, the paper stops short of explaining where they come from. Why does the question statement alone can decide a computational structure for its answering process. Besides some averaged statistics, we don't know much about the behaviors of the RL-learned router. The training dynamics of the RL process is also unclear, does a question leads to diverse rollouts initially? How do they converge? Doing experiments on moderate-to-easy math datasets is also limited.
  • It's unclear how batching is done when routing is decided on a per-question basis. The paper specifies batch size larger than 1 and acknowledges that "varying routes complicates batching" without more explanation. This directly impacts the practical value of the proposed approach.
  • AI-assisted writing left a large number of unnecessary or excessive hedging across the paper, obscuring the paper's main contributions and arguments. See line 76, 455, 470, etc.
Questions for authors
  • How different routes benefit different questions? Is it a function of the question's perceived difficulty or domains? If we use the question plus a few more reasoning sentences, does the route change?
Reviewer_AHUoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a method for adaptive routing for looped transformers. Most standard methods loop each layer for the same number of iterations whereas this method learns how to assign a maximum total loop budget among the model's layers in an input-dependent manner. The idea is that this approach can outperform standard methods by better adapting looping to the specific computational structure of a given problem. Also, the method supports early exiting which means compared to looping for a fixed number of iterations, this approach can save compute. The authors adapt pre-trained (non-looped) models into looped models and learn a simple controller network via online RL to determine loop routes. Their experiments are focused on 7-8B param Qwen/Llama models in math domains. They also show their learned controller can transfer beyond the problem distribution it was trained on.

Critical strengths
  • The method addresses a relevant problem. Looped transformers are becoming more popular as they have several efficiency and generalization benefits over standard models. It seems wasteful to apply the same number of loops for every input so some kind of adaptive input-dependent approach seems important.
  • Paper is easy to read
  • Method is straightforward to understand and seems to work better than standard baselines in the settings they evaluate on.
Critical weaknesses

As the authors acknowledge, there is extensive prior work on adaptive computation and even quite similar work applying these ideas to looped LLMs. Consequently, the author's proposed method seems like a relatively minor and incremental improvement over the prior work. To enhance their contribution, I think the authors could provide more evidence of the generality of their method and a deeper understanding of why and when it works. A few specific things I think should be addressed:

  • Could you benchmark your method and some of the baselines on a wider variety of tasks (beyond Math benchmarks too)? Where is this type of routing most beneficial and when, if ever, is it harmful?
  • The transfer results are still quite in-distribution. Do you have results for out-of-distribution tasks? It seems reasonable the method might not be beneficial there but it seems worth verifying that it isn't harmful either.
  • It seems like a strong limitation that the route is solely based on the prompt versus being more adaptive as the model generates its response. I understand the computational benefits of this but it seems like for more complicated/long multistep responses there could be substantial variation in the optimal route (i.e. some steps might be a lot easier/harder than others). Per token routing might be too much overhead but per reasoning step routing seems more reasonable than a route solely based on the prompt. Have you explored this approach?
  • It seems like experiments are only run for non-looped pre-trained models. Can your method also be applied to pre-trained looped models (that don't have adaptive looping already). If so, it would be good to have experiments for these models to show if your method works for them as well or at least include comparisons with looped models trained from scratch.
  • It would be good to see more analysis about where the benefits are coming from. What types of inputs benefit most from your specific design choices and why?
Questions for authors
  1. It seems interesting that exact-depth RL seems to underperform Free-choice (Table 1). If I understand currently, this would mean additional loops can be harmful sometimes and early-exiting can be beneficial. It would be interesting to understand the kinds of examples where this is the case and why the additional loops are harmful.
  2. How strong is this monotone routing assumption? Where might it fail? It seems like your method could be adapted to the non-monotone case by adding extra actions for the controller. Did you try this? For completeness, it would be good to show results for this as I don't believe monotone-only looping is the standard setting for looped transformers.
  3. It would be useful to show RL learning curves and whether performance has saturated.
  4. It would also be helpful to have results at a few different model scales (even if they're less than 7/8B). This would help establish (at least directionally) how we should expect the performance of these methods to scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.