acceptodds
Under review as a conference paper at ICLR 2027

Training-Free Looped Transformers

10 views

Abstract

We introduce **training-free looped transformers**, in which a lightweight inference-time wrapper loops a contiguous mid-stack block of layers of a frozen checkpoint without additional fine-tuning, continued training, or architectural changes. Unlike prior looped transformer methods that train with the looped structure end-to-end, we retrofit recurrence onto pretrained models at test time. We show that naive block reapplication usually degrades performance, highlighting the importance of the loop application strategy. Motivated by *viewing a pre-norm transformer block as a forward Euler step on an ODE*, we instead treat looping as a refinement of the same approximation, replacing one large update with smaller damped sub-steps. Across seven dense, sparse MoE, and MLA+MoE model families, our training-free looping strategy yields consistent performance improvements over the original frozen checkpoints.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Training-free inference, Looped Transformer, Inference-time computation, Test-time compute, Large Language Model, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ex5gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether off-the-shelf, frozen pretrained Transformer checkpoints can be converted into looped Transformers at inference time without any fine-tuning. Viewing a pre-norm residual block g(x)=x+Fg(x)g(x) = x + F_g(x) as a single forward Euler step (h=1h=1) on an underlying ODE, the authors show that naive looping (h=1h=1 repeated KK times) pushes hidden states out-of-distribution to t=Kt=K, causing performance collapse. Instead, they propose damped sub-stepping (h=1/Kh=1/K) and a front-loaded KK-stage Runge–Kutta blend (x1=βg(x0)+(1−β)FK(x0)x_1 = \beta g(x_0) + (1-\beta)F^K(x_0)), alongside a layer-mode iteration with frozen router logits to prevent routing thrash in MoE models. Evaluated across 6 model families (dense, MoE, MLA) with a fixed recipe (W=4W=4 middle layers, K=6,β=0.5K=6, \beta=0.5), the wrapper yields consistent (though often modest) improvements on multiple-choice benchmarks and ~5 pp gains on open-ended math/coding with Qwen3-30B-A3B.

Overall, the ODE sub-stepping motivation is clean, the MoE layer-mode insight is practical, and the experimental execution is smooth. However, the paper omits explicit FLOPs/latency accounting in the main text, lacks equal-compute baselines (such as comparing loop FLOPs against extra CoT tokens or sampling), and leaves open whether the training-free loop performs genuine multi-step computation or merely acts as a small perturbation around the original representation. I lean toward Weak Accept.

Critical strengths

1.Clear numerical motivation and zero-training applicability: Interpreting naive looping vs. damped sub-stepping as t=Kt=K drift vs. t=1t=1 ODE refinement is intuitive, well-illustrated (Figure 2), and immediately applicable to frozen open-weight checkpoints.

2.Practical engineering for MoE and KV caching: Identifying routing thrash in MoE block-mode and fixing it via layer-mode with pinned router logits, as well as the snapshot/restore FIRST KV-cache protocol in vLLM and HuggingFace, are well-executed.

3.Broad out-of-the-box evaluation: Testing a single committed hyperparameter recipe across 6 model families without per-benchmark tuning demonstrates good discipline.

Critical weaknesses

1.Missing Computational Cost (FLOPs / Latency) Reporting and Trade-off Guidance: Throughout the main paper, the computational overhead of the default recipe (W=4W=4 layers, K=6K=6) is never explicitly quantified alongside the accuracy gains. Let us estimate the actual cost: iterating a 4-layer window K=6K=6 times adds 4×(6−1)=204 \times (6 - 1) = 20 extra layer evaluations per token (or 24 extra layer passes in Algorithm 4 when the FIRST cache stash pass is executed separately). For a 28-layer model (e.g., Qwen3-1.7B or Llama-3.2-3B) or a 36-layer model (Qwen3-4B), this represents a 55% to 85% increase in per-token FLOPs, plus memory-bandwidth overhead from KV cache snapshot/cropping and tensor cloning at every decode step. Indeed, Appendix G.2 (Table 10) quietly notes that decode_mode=full takes 2–5$\times$ longer in wall-clock time (e.g., 8h →\to 21h on Moonlight-16B). Without reporting FLOPs and latency in Tables 2–4, or discussing when paying a 60%+ FLOP / 2–5$\times$ latency penalty for a +0.3 to +1.0 pp accuracy gain is practically worthwhile, the usability of the method is significantly weakened.

2.Unclear Baseline Comparisons Against Explicit CoT / Equal Test-Time Compute: Recent literature often frames latent looping as an implicit counterpart to explicit Chain-of-Thought (CoT) or test-time scaling. Because the models here were never trained to loop, forcing recurrence at test time is akin to asking a model to do latent reasoning without training. If the looped wrapper consumes 1.6$\times$–2$\times$ more FLOPs (and 2–5$\times$ wall-clock time during decoding), a crucial baseline is missing: how does looping compare against spending the exact same extra compute budget on standard test-time scaling (e.g., allowing a slightly longer CoT token budget, or increasing the number of sampled trajectories from avg@16 to avg@26 / maj@K)?

3.Is It Genuine Latent Refinement or Just a Small Output Perturbation? Looking closely at Equation (16) (x1=βg(x0)+(1−β)FK(x0)x_1 = \beta g(x_0) + (1-\beta) F^K(x_0) with β=0.5\beta=0.5) and Appendix H.1, the output is heavily anchored to the unmodified single-step output g(x0)g(x_0). Furthermore, under the FIRST cache strategy (which is required because keeping iterated KV drops accuracy by 2.5 pp, p. 27), subsequent tokens do not even attend to the looped KV states—they attend to the unlooped natural KV! Combined with the fact that all higher-order ODE solvers and fixed-point accelerators fail to beat K=2K=2 damped Euler (Table 7), this strongly suggests that the training-free loop is not executing deep iterative reasoning, but rather injecting a small, regularized residual perturbation (or self-ensembling smoothing) around g(x0)g(x_0). A deeper analysis of how much the final logits or token trajectories actually diverge from g(x0)g(x_0) would clarify what the loop is truly doing.

Questions for authors

1.Can you provide an explicit FLOPs and wall-clock latency/throughput breakdown (tokens/sec in vLLM and HF) for the results in Tables 2, 3, and 4? In Algorithm 4, does c = FIRST require K+1K+1 passes through the window (since xax_a is evaluated in the first iteration k=1k=1, couldn't the KV from k=1k=1 simply be kept instead of running a separate stash pass at line 21)?

2.On AIME and Codeforces (Table 4), what is the wall-clock runtime ratio between the baseline (K=1K=1) and the looped model (K=6K=6)? If the baseline is given the same total FLOPs or wall-clock budget (e.g., by sampling more chains or using a slightly larger model in the same family), does training-free looping still hold an advantage?

3.Given that β=0.5\beta=0.5 anchors half the update to g(x0)g(x_0) and FIRST discards the looped KV states for future tokens, how much of the gain comes from genuine multi-step ODE refinement (K=6K=6) versus a simple 1-step residual perturbation (K=2K=2) or minor logit smoothing?

Reviewer_VK62OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a training free method that repeatedly applies a block of middle layers in a frozen pretrained transformer during inference, without additional training, parameters, or architectural changes. Inspired by the connection between transformer layers and numerical integration, the authors replace naive repetition with smaller, controlled updates and combine the result with the original output. For mixture of experts models, they repeat each layer separately while keeping expert routing fixed to improve stability. The method is evaluated using a fixed configuration of four middle layers, six iterations, and equal weighting of the original and refined outputs across six model families. Experiments on knowledge intensive multiple choice tasks, competition mathematics, and code generation show improvements over the original models and naive looping.

Critical strengths

The method can be applied directly to publicly released models, which is a meaningfully different setting from prior looped transformer work that requires training with the loop structure from the start.

Critical weaknesses

The method adds much more compute per token and keeps the parameter count unchanged, which abandons the usual parameter efficiency motivation of looped transformers. The only remaining justification is trading inference compute for accuracy, but the paper never compares against other ways of spending the same budget, such as a larger model from the same family, majority voting, or longer reasoning. Without such baselines, a gain of a few tenths of a point to two points may well be worse value than simply switching to a bigger checkpoint?

There are no confidence intervals, no paired tests, and no per question data, while several headline gains on GPQA, OpenBookQA, and ARC Challenge are within about one standard error of the benchmark size. The positive threshold of 0.3 points is far below typical benchmark noise.

The paper does not test whether a random perturbation of similar magnitude, looping randomly chosen layers, or simply varying β without extra iterations produces comparable changes. Without these controls, one cannot rule out that the small gains reflect noise or generic perturbation rather than the proposed mechanism. No analysis of hidden states, attention, or per example behavior is offered to support the refinement interpretation.

Questions for authors

How does the method compare with other ways of spending the same extra compute? For example, what is the accuracy of the next larger model in the same family, or of majority voting over several samples, at a matched inference budget? Could you also report actual latency and throughput overhead?

Can you provide confidence intervals and paired significance tests for the main results?

Reviewer_pZDROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to improve pretrained language models by looping their middle layers at inference time without any additional training. Instead of directly repeating the layers, which can hurt performance, the authors propose a damped update strategy motivated by viewing transformer residual updates as Euler integration steps. They also explore several numerical integration strategies and extend the method to both dense and MoE models. Experiments on multiple-choice, math, and coding benchmarks show performance improvements across different model families.

Critical strengths

S1. The method is simple and can be applied to existing pretrained models without any additional training. I think this is a practical way to improve model performance.

S2. The authors test their method on different dense and MoE models, including Qwen, Llama, Moonlight, and DeepSeek, covering both multiple-choice benchmarks and open-ended math and coding tasks.

S3. The authors try to motivate their design from an ODE perspective by interpreting transformer residual updates as Euler integration steps. I think this provides an interesting way to understand and motivate the proposed method.

S4. The authors explore several iteration strategies, including damped Euler, Runge–Kutta, and fixed-point acceleration methods. I think these comparisons provide some useful insights into what works and what doesn't.

Critical weaknesses

W1. The idea of looping the middle layers of a frozen pretrained model has already been explored in prior work. For example,Lys et al. (2026) investigate a very similar approach, and David Noel Ng demonstrated that simply repeating a block of middle layers without modifying any weights could improve model performance. However, the introduction gives the impression that training-free inference-time looping is a novel contribution, without sufficiently distinguishing the proposed approach from existing methods. I think the main novelty lies in the damped residual update, but its differences from and advantages over prior approaches are not clearly established.

W2. In Figure 4, the authors mainly compare their method with naive looping, which performs poorly. I think it would be more useful to compare with existing training-free looping methods, especially the interpolation and stabilization strategies from ,Lys et al. (2026). Although uniform looping is included in the ablation studies, a more systematic comparison under the same inference compute budget is still missing.

Questions for authors

Q1. The paper mentions that "we apply a single out-of-the-box recipe: K-stage Runge–Kutta (Algorithm 3) with K=6 and β=0.5 at the middle four layers, block-mode for dense backbones and layer-mode with a frozen router for MoE backbones, and the FIRST KV-cache strategy, with no per-cell hyperparameter search over position, K, or strategy." From the text, it seems that this design does not involve per-dataset hyperparameter tuning, but did the authors perform hyperparameter tuning across all benchmarks together? If so, how? I didn't see any mention of this.

Q2. How noisy are these improvements? Confidence intervals are not provided. It would be useful to see (at least for a few datasets) what the confidence intervals look like relative to the reported improvements.

Q3.The paper mentions that they use 6 loops of 4 layers, which would involve 24 layer executions in total (20 additional layer executions compared to the original model). What is the resulting inference slowdown? The authors don't report this. Although we could estimate it ourselves based on each model's architecture, I think this should be an important metric in the results table. For many benchmarks, a 1–2 pp gain may not be worth the additional inference compute.

Q4. I think the idea of looping the middle layers of a frozen pretrained model has already been explored in prior work, but the paper's introduction gives the impression that this is a novel contribution. For example, Lys et al. (2026) studied a very similar approach, and David Noel Ng demonstrated that simply repeating a block of middle layers without modifying any weights could improve model performance. Although the authors cite some related work, I don't think they sufficiently discuss how their approach differs from these existing methods. Could the authors clarify the main novelty of their approach beyond the damped residual update and provide a more direct comparison with prior work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.