- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates whether off-the-shelf, frozen pretrained Transformer checkpoints can be converted into looped Transformers at inference time without any fine-tuning. Viewing a pre-norm residual block as a single forward Euler step () on an underlying ODE, the authors show that naive looping ( repeated times) pushes hidden states out-of-distribution to , causing performance collapse. Instead, they propose damped sub-stepping () and a front-loaded -stage Runge–Kutta blend (), alongside a layer-mode iteration with frozen router logits to prevent routing thrash in MoE models. Evaluated across 6 model families (dense, MoE, MLA) with a fixed recipe ( middle layers, ), the wrapper yields consistent (though often modest) improvements on multiple-choice benchmarks and ~5 pp gains on open-ended math/coding with Qwen3-30B-A3B.
Overall, the ODE sub-stepping motivation is clean, the MoE layer-mode insight is practical, and the experimental execution is smooth. However, the paper omits explicit FLOPs/latency accounting in the main text, lacks equal-compute baselines (such as comparing loop FLOPs against extra CoT tokens or sampling), and leaves open whether the training-free loop performs genuine multi-step computation or merely acts as a small perturbation around the original representation. I lean toward Weak Accept.
Critical strengths
1.Clear numerical motivation and zero-training applicability: Interpreting naive looping vs. damped sub-stepping as drift vs. ODE refinement is intuitive, well-illustrated (Figure 2), and immediately applicable to frozen open-weight checkpoints.
2.Practical engineering for MoE and KV caching: Identifying routing thrash in MoE block-mode and fixing it via layer-mode with pinned router logits, as well as the snapshot/restore FIRST KV-cache protocol in vLLM and HuggingFace, are well-executed.
3.Broad out-of-the-box evaluation: Testing a single committed hyperparameter recipe across 6 model families without per-benchmark tuning demonstrates good discipline.
Critical weaknesses
1.Missing Computational Cost (FLOPs / Latency) Reporting and Trade-off Guidance: Throughout the main paper, the computational overhead of the default recipe ( layers, ) is never explicitly quantified alongside the accuracy gains. Let us estimate the actual cost: iterating a 4-layer window times adds extra layer evaluations per token (or 24 extra layer passes in Algorithm 4 when the FIRST cache stash pass is executed separately). For a 28-layer model (e.g., Qwen3-1.7B or Llama-3.2-3B) or a 36-layer model (Qwen3-4B), this represents a 55% to 85% increase in per-token FLOPs, plus memory-bandwidth overhead from KV cache snapshot/cropping and tensor cloning at every decode step. Indeed, Appendix G.2 (Table 10) quietly notes that decode_mode=full takes 2–5$\times$ longer in wall-clock time (e.g., 8h 21h on Moonlight-16B). Without reporting FLOPs and latency in Tables 2–4, or discussing when paying a 60%+ FLOP / 2–5$\times$ latency penalty for a +0.3 to +1.0 pp accuracy gain is practically worthwhile, the usability of the method is significantly weakened.
2.Unclear Baseline Comparisons Against Explicit CoT / Equal Test-Time Compute: Recent literature often frames latent looping as an implicit counterpart to explicit Chain-of-Thought (CoT) or test-time scaling. Because the models here were never trained to loop, forcing recurrence at test time is akin to asking a model to do latent reasoning without training. If the looped wrapper consumes 1.6$\times$–2$\times$ more FLOPs (and 2–5$\times$ wall-clock time during decoding), a crucial baseline is missing: how does looping compare against spending the exact same extra compute budget on standard test-time scaling (e.g., allowing a slightly longer CoT token budget, or increasing the number of sampled trajectories from avg@16 to avg@26 / maj@K)?
3.Is It Genuine Latent Refinement or Just a Small Output Perturbation? Looking closely at Equation (16) ( with ) and Appendix H.1, the output is heavily anchored to the unmodified single-step output . Furthermore, under the FIRST cache strategy (which is required because keeping iterated KV drops accuracy by 2.5 pp, p. 27), subsequent tokens do not even attend to the looped KV states—they attend to the unlooped natural KV! Combined with the fact that all higher-order ODE solvers and fixed-point accelerators fail to beat damped Euler (Table 7), this strongly suggests that the training-free loop is not executing deep iterative reasoning, but rather injecting a small, regularized residual perturbation (or self-ensembling smoothing) around . A deeper analysis of how much the final logits or token trajectories actually diverge from would clarify what the loop is truly doing.
Questions for authors
1.Can you provide an explicit FLOPs and wall-clock latency/throughput breakdown (tokens/sec in vLLM and HF) for the results in Tables 2, 3, and 4? In Algorithm 4, does c = FIRST require passes through the window (since is evaluated in the first iteration , couldn't the KV from simply be kept instead of running a separate stash pass at line 21)?
2.On AIME and Codeforces (Table 4), what is the wall-clock runtime ratio between the baseline () and the looped model ()? If the baseline is given the same total FLOPs or wall-clock budget (e.g., by sampling more chains or using a slightly larger model in the same family), does training-free looping still hold an advantage?
3.Given that anchors half the update to and FIRST discards the looped KV states for future tokens, how much of the gain comes from genuine multi-step ODE refinement () versus a simple 1-step residual perturbation () or minor logit smoothing?