acceptodds
Under review as a conference paper at ICLR 2027

FlashLoop: Fast and Memory-Efficient Looped Transformers via Lazy Updates

Abstract

Looped Transformers have attracted substantial attention as a parameter-efficient approach to increasing computational depth through repeated application of shared Transformer blocks. However, their practical advantages over conventional Transformers remain under debate: each additional loop incurs another Transformer pass and requires caching another set of KV states, causing inference FLOPs and KV-cache memory to grow continuously with loop depth. This overhead becomes particularly severe at large loop counts and long context lengths, preventing the parameter efficiency of Looped Transformers from translating into practical inference efficiency. In this paper, we find that much of the additional computation and storage introduced by looping is redundant. As recurrence proceeds, state changes become increasingly concentrated on a small subset of tokens; attention-output differences are dominated by a sparse and stable subset of key columns; and KV residuals between adjacent loops become progressively more amenable to low-bit quantization. Building on these observations, we introduce FlashLoop, a training-free inference framework that reduces cross-loop redundancy through token-sparse updates, sparse attention, and KV-residual quantization. Across five Ouro and Huginn variants, FlashLoop largely preserves model accuracy while achieving – end-to-end speedup and – KV-cache memory reduction. At a 32K context length, FlashLoop reduces KV-cache memory by while delivering approximately speedup, substantially improving the practicality of scaling Looped Transformers to greater computational depths.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Transformer, Sparse Attention, KV Cache Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fZUkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes 3 ways to speed up inference in looped transformer models.

  1. First, they observe that in deeper loops state changes become increasingly concentrated on only a few tokens (~x %). So tokens which have converged can fully skip later loops (a form of early exit).
  2. Second, cross-loop differences in attention outputs are dominated by a sparse subset of columns whose indices can be reliably predicted from the preceding loop.
  3. Residual KV caches between consecutive loops become more amenable to quantization.

All 3 put together gives us FlashLoop which on the authors' ablations yields up to 1.64 speedup in decoding and up to 6x reduction in KV cache memory.

  1. Later loops rank tokens updated in earlier loops according to their normalized hidden state changes and retain only top-k for further refinement.
  2. Similarly, later loops rank attention columns and only keep top-k for recomputation.
  3. QUantize the KV cache and the KV residual updates of later loops separately.

The authors implement an optimized kernel on GPUs for these changes and test them on models from Ouro family and Huginn family. Across 5 reasoning benchmarks they observe no loss in quality with FlashLoop while observing significant speedups and KV memory savings.

Critical strengths
  1. The proposed changes are all inference time changes and require no re-training of models.
  2. The authors took the effort to implement an optimized GPU kernel for their changes to make them truly see wall-clock time savings.
Critical weaknesses
  1. For a paper claiming quality neutrality of an inference speedup technique, it is important to see more than 5 eval benchmarks. Especially ones spread across diverse domains such as math, coding, memorization intensive tasks and other areas. It is important to understand regressions in any aspect of the model when applying such changes.
Questions for authors
  1. The rationale provided for why KV residuals are easier to quantize than the KV states themselves is a bit unclear. Especially since you end up quantizing the base KVs as well. Why did we need to consider residuals and quantize them separately if the original KV is also being quantized?
  2. Can we breakdown time savings for prefill vs decode separately? will be interesting to see the differences between the two paradigms.
4 comments
AuthorsOpenReview ↗

Response to Reviewer fZUk (1/2)

We sincerely thank the reviewer for recognizing our training-free design and our efficient implementation. We will provide detailed responses to your questions.

W1: For a paper claiming quality neutrality of an inference speedup technique, it is important to see more than 5 eval benchmarks. Especially ones spread across diverse domains such as math, coding, memorization intensive tasks and other areas. It is important to understand regressions in any aspect of the model when applying such changes.

Reply:

We thank the reviewer for the suggestion. In addition to the five benchmarks in Table 1, which cover math (MATH-500, GSM8K) and general reasoning (ARC-C, HellaSwag, WinoGrande), Appendix I (Table 8) of the paper reports long-context results, i.e., WikiText-2 perplexity at 8K, 16K, and 32K and Needle-in-a-Haystack accuracy, where FlashLoop shows no significant degradation. We further add four benchmarks covering coding (HumanEval, MBPP) and memorization-intensive tasks (LongBench v2, RULER).

For coding, we evaluate HumanEval and MBPP, using the same configuration as Table 1:

ModelMethodHumanEvalMBPP
Ouro-1.4BOriginal71.9574.34
Ouro-1.4BFlashLoop69.5171.96
Ouro-2.6BOriginal79.2781.75
Ouro-2.6BFlashLoop78.0580.42

FlashLoop retains most of the coding accuracy of the original models on both benchmarks, with a smaller drop on the larger model.

For memorization-intensive tasks, which we understand as tasks where the model has to retain and retrieve information from a long context, we evaluate Ouro-2.6B on LongBench v2 (32K input) and RULER (16K context), using the same FlashLoop setting as in the paper:

MethodLongBench v2 (%)RULER 16K (%)
Original37.7780.73
FlashLoop37.5780.55

The results show that FlashLoop stays close to the original model on both long-context benchmarks.

Q1: The rationale provided for why KV residuals are easier to quantize than the KV states themselves is a bit unclear. Especially since you end up quantizing the base KVs as well. Why did we need to consider residuals and quantize them separately if the original KV is also being quantized?

Reply:

We thank the reviewer for the question and would like to first clarify what is quantized. FlashLoop does not quantize the full KV states of every loop. A looped Transformer produces and stores a separate KV cache in every loop. In FlashLoop, the full KV states are quantized only once, for the first loop, and serve as the base. For every later loop, the full KV states are never quantized and we quantize only their residual relative to the preceding loop (Eq. (4)).

Why residuals are easier to quantize. We explain this from two perspectives.

(1) Numerical range. The cross-loop KV residuals have a much smaller magnitude than the full KV states (Fig. 3(c, d)), so the same number of quantization levels covers a much narrower value range. As a result, the residuals have a much lower quantization error than the full KV states at the same bit-width (Fig. 2(c)), and replacing residual quantization with direct quantization of the full KV states leads to a noticeably larger accuracy degradation (Table 2).

(2) Cross-loop redundancy. The KV states of adjacent loops are highly similar, since all loops compute keys and values with the same projection weights and the hidden states change only slightly from one loop to the next (Section 3.1). Quantizing the full KV states of every loop stores this shared information repeatedly. FlashLoop instead stores it once in the base, so that the bits of each later loop represent only its loop-specific difference.

AuthorsOpenReview ↗

Response to Reviewer fZUk (2/2)

Q2: Can we breakdown time savings for prefill vs decode separately? will be interesting to see the differences between the two paradigms.

Reply:

We thank the reviewer for the question. Fig. 4 of the paper reports prefill and decode latency separately as the three components are enabled cumulatively. We list the corresponding numbers below (Ouro-1.4B, 8K context, one A100-40GB):

ConfigurationPrefill (ms)Prefill speedupDecode (ms/token)Decode speedup
Original14561.00×721.00×
+ Token-Sparse Updates6452.26×1580.45×
+ Sparse Attention6452.26×980.73×
+ 4-bit KV Residual Quant. (FlashLoop)9211.58×441.63×

The two phases benefit from different components. Prefill savings come from cross-loop token-sparse updates, which recompute only the tokens that continue to change in later loops. Sparse attention has little effect on prefill, and quantization adds work during prefill, but the complete FlashLoop remains faster than the original model. Decode savings come from loop-aware sparse attention and KV residual quantization, which reduce the KV states read for each generated token. Token-sparse updates alone increase decode latency, and sparse attention offsets much of this increase, and quantization brings the decode latency below that of the original model.

Dear Reviewer fZUk,

Thanks for your positive initial review. If our response effectively resolves your issue, we'd be grateful if you consider further improving your score.

Thank you!

Authors

Reviewer_fZUkOpenReview ↗

Thanks for the response

Thank you to the authors' response. The added numbers help alleviate some of my concerns but I will maintain my current rating based on the strengths of the paper.

Reviewer_L6isOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for reducing the time and memory required by Looped Transformers. It takes advantage of three empirical observations: (1) only a small number of tokens change significantly from one iteration of the loop to the next, (2) the set of tokens that have the highest attention weights doesn't change much from one iteration to the next, and (3) the magnitude of the change in keys and values relative to the previous iteration decreases from one iteration to the next. The proposed method, FlashLoop, takes advantage of these observations by caching tokens from the previous iteration, storing the token indices with the highest attention weights from the previous iteration and recomputing only those in the next iteration, and quantizes the change in keys and values from one iteration to the next rather than the original keys and values themselves.

Critical strengths
  • The paper is for the most part well written (though the phrasing is sometimes imprecise).
  • The paper performed CUDA-level optimization and the code as part of the supplementary material.
  • The paper evaluated on five different Looped Transformer models.
Critical weaknesses
  • The only baseline that the paper compared to in terms of latency is the vanilla model - it did not compare to the latency of other KV-cache optimization methods (it only compared to other KV-cache optimization methods in terms of accuracy).
  • The baselines are weak. The vanilla model is a naive baseline, and the two KV-cache optimization methods are old or unpublished (H2O is from NeurIPS 2023, and Last-step KV Reuse is an arXiv preprint from 2025). The paper should compare to the latest published baselines.
  • It seems that the baselines are not tuned. For example, it seems that the 25% retention ratio was chosen arbitrarily. Instead, the authors should tune the retention ratio to achieve a comparable accuracy compared to the proposed method, and then compare the latency of the baselines to that of the proposed method.
  • Some of the empirical observations are not new, e.g., the second observation on the tokens with the highest attention weights not changing much from one layer to the next. This has been observed for Transformers in general (see IceCache from ICLR 2026 for example).
  • Some of the phrasing is imprecise. For example, the description of the second observation seems to be different from what Fig. 3(b) shows. On L202-203, the paper mentions that "we find that cross-loop changes in the attention output are concentrated on a sparse subset of attention columns", but Fig. 3(b) doesn't show changes in the attention output. It instead shows changes in which attention columns are in the top 10%.
  • In different parts of the paper, different words are used to refer to equivalent things. For example, the third observation refers to "KV states", and the second observation refers to "attention columns". But since there is a one-to-one correspondence between an attention column and a key and a value, attention column means the same thing as a key-value pair. The first observation refers to "tokens", each of which produces a key and a value. This change in terminology in the middle can create confusion.
  • Minor: "loops" seem to really mean "iterations of a loop".
Questions for authors
  • Can you clarify how you calculate the "top-10% column turnover" shown in Fig. 3(b)?
  • L249: Does k vary with loop depth in your experiments? If so, how do you vary it?
  • Can you define "attention key ratio" and "loop 3 / loop 4 token update ratio"?
  • What do you do for iterations 1 and 2 of the loop?
  • For reporting results of the proposed method in Tables 1-3, how were the hyperparameters tuned? Were they tuned on held-out data or test data?
6 comments
AuthorsOpenReview ↗

Response to Reviewer L6is (1/3)

We sincerely thank the reviewer for recognizing the clarity of the paper, our efficient implementation, and the evaluation on five models. We will provide detailed responses to your questions.

W1: The only baseline that the paper compared to in terms of latency is the vanilla model - it did not compare to the latency of other KV-cache optimization methods (it only compared to other KV-cache optimization methods in terms of accuracy).

Reply:

We thank the reviewer for this suggestion. We add a latency comparison against H2O and Last-step KV Reuse under the setting of Table 3 (25% retention). For a fair comparison, we re-implemented both methods in our inference engine following their official code. All methods are then measured in this engine on one A100-40GB with 8K inputs. Results on Ouro-1.4B:

MethodKV cache keptAvg. Acc. (%)Prefill speedupDecode speedup
Original100%70.111.00×1.00×
H2O25%62.810.96×1.35×
Last-step KV Reuse25%68.331.00×1.38×
FlashLoop17.1%70.481.58×1.63×

FlashLoop achieves better accuracy and better efficiency than KV-cache optimization baselines. In particular, both baselines only speed up decoding and do not reduce prefill computation.

W2: The baselines are weak. The vanilla model is a naive baseline, and the two KV-cache optimization methods are old or unpublished (H2O is from NeurIPS 2023, and Last-step KV Reuse is an arXiv preprint from 2025). The paper should compare to the latest published baselines.

Reply:

We chose H2O [1] as a representative KV-cache optimization method for non-looped Transformers, whose attention-score-based framework later methods largely follow, and Last-step KV Reuse because it is the loop-specific KV strategy proposed by the Ouro paper [2] for these models. Other methods for looped Transformers modify the architecture or require additional training, as we discuss in Related Work.

Following the reviewer's suggestion, we add comparisons with recent published methods: R-KV [3] (NeurIPS 2025) for token eviction, and KIVI [4] (ICML 2024) and TurboQuant [5] (ICLR 2026) for KV quantization. We apply each method to the KV cache of every loop with the same −75% KV budget as in Table 3. For TurboQuant, we also report its 2.5-bit setting, whose KV-memory reduction (6.4×) is close to that of FlashLoop (5.85×). ARC-C accuracy (%) on Ouro-1.4B:

ModelOriginalH2OLast-stepR-KVKIVI-4TurboQuant-4TurboQuant-2.5FlashLoop
Ouro-1.4B59.9854.0158.3656.3158.1159.9858.5360.67

Recent KV-cache optimization methods such as TurboQuant clearly reduce the accuracy drop of earlier methods. However, at a compression ratio comparable to FlashLoop (TurboQuant-2.5), they still fall behind FlashLoop, because they compress each loop independently and spend much of the budget on information that is redundant across loops. The new comparisons are added to the revision.

W3: It seems that the baselines are not tuned. For example, it seems that the 25% retention ratio was chosen arbitrarily. Instead, the authors should tune the retention ratio to achieve a comparable accuracy compared to the proposed method, and then compare the latency of the baselines to that of the proposed method.

Reply:

We would like to clarify that the 25% retention ratio in Table 3 was not chosen arbitrarily. It is set to align with Last-step KV Reuse, an important loop-specific baseline proposed by the Ouro paper [2], which keeps only the KV cache of the last of Ouro's four loops, i.e., 25%. We use the same ratio for H2O so that both baselines are compared at the same KV budget. In contrast, the default configuration of FlashLoop keeps only 17.1% of the KV cache.

Following the suggestion, we also tuned the baselines for comparable accuracy. For each baseline, we sweep the retention ratio and take the smallest one whose five-benchmark average is close to that of FlashLoop. For Last-step KV Reuse, the ratio is set by the number of trailing loops k whose KV is kept. We then measure latency in the same setting as in our reply to W1. Results on Ouro-1.4B:

MethodKV cache keptAvg. Acc. (%)Prefill speedupDecode speedup
H2O (tuned)50%69.420.96×1.17×
Last-step KV Reuse (tuned, k=2)50%70.021.00×1.19×
FlashLoop17.1%70.481.58×1.63×

To reach comparable accuracy, both baselines must keep a larger part of the KV cache, so their speedup and memory reduction become smaller. FlashLoop remains faster and uses less KV memory than both baselines.

AuthorsOpenReview ↗

Response to Reviewer L6is (2/3)

W4: Some of the empirical observations are not new, e.g., the second observation on the tokens with the highest attention weights not changing much from one layer to the next. This has been observed for Transformers in general (see IceCache from ICLR 2026 for example).

Reply:

We must clarify a misunderstanding: our observations all concern redundancy across loops, e.g. our second observation states that the important attention columns remain stable across adjacent loops, not across adjacent layers. Each loop consists of multiple layers, so the similarity between adjacent layers observed on non-looped Transformers (e.g., IceCache [6]) does not imply that the same layer still behaves similarly after a full loop. Our analysis in Sec. 3 verifies that this cross-loop stability holds.

W5: Some of the phrasing is imprecise. For example, the description of the second observation seems to be different from what Fig. 3(b) shows. On L202-203, the paper mentions that "we find that cross-loop changes in the attention output are concentrated on a sparse subset of attention columns", but Fig. 3(b) doesn't show changes in the attention output. It instead shows changes in which attention columns are in the top 10%.

Reply: We thank the reviewer for the careful reading and would like to clarify that Fig. 3(b) indeed illustrates a different statement from the one quoted. The statement quoted from L202–203, that cross-loop changes in the attention output are concentrated on a small subset of columns, is illustrated by Fig. 3(a): a small set of columns carries most of the attention mass in every loop, and since each column contributes to the attention output in proportion to its attention weight, the cross-loop changes in the output are concentrated on these columns. Fig. 3(b) illustrates the statement that follows it: the importance of these columns remains stable across adjacent loops, so the previous loop can identify the columns that are likely to remain important in the next loop. To make this correspondence clear, we have extended the caption of Fig. 3 to describe what panels (a) and (b) each show.

W6: In different parts of the paper, different words are used to refer to equivalent things. For example, the third observation refers to "KV states", and the second observation refers to "attention columns". But since there is a one-to-one correspondence between an attention column and a key and a value, attention column means the same thing as a key-value pair. The first observation refers to "tokens", each of which produces a key and a value. This change in terminology in the middle can create confusion.

Reply:

We thank the reviewer for the comment and would like to clarify how these terms are used in the paper.

Tokens, attention columns, and KV states are indeed related, but they are not the same entity. The three terms refer to different entities in the attention computation: a token is a sequence position, whose hidden state produces the query, key, and value. KV states are the stored key and value tensors, and an attention column is the set of attention weights that all queries assign to one key/value pair. They should therefore not be referred to by a single term. To make their relation clear, we add a figure to Appendix B (Fig. 8) that illustrates the three terms in one attention computation, and refer to it in Section 3.

W7: Minor: "loops" seem to really mean "iterations of a loop".

Thank you for this suggestion. We agree and have revised the wording as suggested: each pass through the shared block is now referred to as a loop iteration throughout the paper. The changes are marked in blue in the updated version.

AuthorsOpenReview ↗

Response to Reviewer L6is (3/3)

Q1: Can you clarify how you calculate the "top-10% column turnover" shown in Fig. 3(b)?

Reply: Thank you for this question. The top-10% column turnover in Fig. 3(b) is computed as follows. For each layer, attention head, and query position, let $ T_r $ be the set of the top-10% key columns ranked by attention probability at loop $ r $. The turnover at the transition $ r−1→r $ is

Turnover(r−1→r)=1−∣Tr−1∩Tr∣∣Tr∣\mathrm{Turnover}(r-1 \to r) = 1 - \frac{|T_{r-1} \cap T_r|}{|T_r|}

i.e., the fraction of top-10% columns at loop $ r $ that were not among the top-10% columns at loop $ r−1 $. Each point in Fig. 3(b) averages this fraction over all layers, heads and evaluation prompts. We have added this definition to the caption of Fig. 3.

Q2: L249: Does k vary with loop depth in your experiments? If so, how do you vary it?

Reply:

We thank the reviewer for the question. Yes, k varies with loop depth and decreases in later loops. In each loop, k is a fixed fraction of the prompt tokens that does not depend on the input. The value used in each loop is listed in Table 5, e.g., Ouro-1.4B updates all tokens in loops 1–2, 25% in loop 3, and 10% in loop 4.

The value of k for each loop is determined by the calibration described in Appendix G: on 128 WikiText-2 sequences, we select the smallest retention ratio for which the relative reconstruction error of the hidden states remains below 5%.

Q3: Can you define "attention key ratio" and "loop 3 / loop 4 token update ratio"?

Reply:

We thank the reviewer for the question. The attention key ratio is the fraction of attention columns (i.e., keys) recomputed in each sparse loop during decoding, i.e., the Columns entries in Table 5. The Loop 3 / Loop 4 token update ratio is the fraction of prompt tokens recomputed in loop 3 and loop 4 during prefill, i.e., the Tokens entries in Table 5, and a tick label such as 25/10% means that 25% of the prompt tokens are recomputed in loop 3 and 10% in loop 4.

We have changed the label "attention key ratio" in Fig. 5 to "fraction of attention columns (i.e., keys) recomputed" and expanded the descriptions in the caption.

Q4: What do you do for iterations 1 and 2 of the loop?

Reply:

We thank for the question. The only component applied in loops 1 and 2 is KV residual quantization: the KV states of loop 1 are quantized as the base, and those of loop 2 are stored as quantized residuals. Token-sparse updates and sparse attention start from loop 3.

Q5: For reporting results of the proposed method in Tables 1-3, how were the hyperparameters tuned? Were they tuned on held-out data or test data?

Reply:

We thank the reviewer for the question. FlashLoop has two groups of hyperparameters, the sparsity ratios and the quantization setting, and neither is tuned on held-out or test data of the evaluation benchmarks. As described in Appendix G, the sparsity ratios are calibrated on 128 sequences randomly sampled from WikiText-2, which is neither held-out data nor test data. For each model, we select the smallest retention ratio for which the relative reconstruction error of the hidden states or attention outputs remains below 5%. The resulting configuration is calibrated once per model and then kept fixed for all benchmarks. The quantization setting is not tuned: it follows prior work [4] and is the same for all models.

References

[1] Zhang et al. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

[2] Zhu et al. Scaling Latent Reasoning via Looped Language Models.

[3] Cai et al. R-KV: Redundancy-aware KV Cache Compression for Reasoning Models.

[4] Liu et al. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache.

[5] Zandieh et al. TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate.

[6] Mao et al. IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs.

Reviewer_4qTPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a training-free inference framework for pretrained looped (depth-recurrent) Transformers. It rests on three measurements of cross-loop redundancy in Ouro and Huginn models: hidden-state changes concentrate on a shrinking, approximately nested subset of tokens as loops proceed; changes in attention outputs are dominated by a small set of key columns whose identity is stable across adjacent loops; and adjacent-loop KV residuals quantize with lower error than full KV states. FlashLoop exploits each in turn.

Critical strengths
  1. Each component is motivated by a measurement.: The redundancy characterization Is clear.
  2. One calibration per model, fixed across tasks.: Retention ratios are chosen once on 128 WikiText-2 sequences under a 5% reconstruction-error rule (Appendix E) and held fixed for every benchmark, which limits per-task tuning.
  3. Real kernels and end-to-end measurements.: The paper implements packed 4-bit KV, fused sparse decode kernels and gathered GEMMs, and reports how savings scale with context length.
Critical weaknesses
  1. "Lossless" is not supported, and no uncertainty is reported. MATH-500 has 500 items, so one SE is about 2.2 points; the gains attributed to sparse attention alone (+3.2 on MATH-500, Table 2) are within the same noise. Five-task averages hide task-level losses on precisely the reasoning tasks looped models are built for. Per-task confidence intervals or paired tests are needed, and "lossless" should be replaced with a quantified statement.
  2. The accuracy evaluation does not match the regime the method targets. The savings grow with context length and loop count, but three of the five benchmarks (ARC-C, HellaSwag, WinoGrande) are short-prompt log-likelihood tasks on which KV memory is negligible. Standard long-context suites (e.g., RULER, LongBench) and long chain-of-thought generation with the Thinking models are needed, since errors from reused KV and stale column selections can accumulate over many decode steps.
  3. Missing loop-axis baselines. The Huginn paper (Geiping et al.) itself describes zero-shot per-token adaptive exit and KV-cache sharing across recurrence steps, and Ouro has learned exit gates. These are the natural comparisons for token-sparse updates and cross-loop KV reuse, and neither is discussed.
  4. The efficiency measurements are narrow. All timing is on one A100-40GB, the batch size is not stated (apparently one), and the baseline is the authors' own engine with the features switched off rather than an optimized serving stack (vLLM or SGLang with FlashAttention and paged KV). Under batching, gathering irregular per-request token sets and column sets is harder to amortize, and the 1.5–1.6× speedup may shrink.
Questions for authors
  1. Can you report per-task 95% intervals (or paired bootstrap tests against the original model) for Table 1, and revise the "lossless" claim accordingly?
  2. How does FlashLoop perform on RULER or LongBench and on long chain-of-thought generation with the Thinking models at 8K–32K?
  3. What batch size was used, and how do prefill latency, decode latency and throughput change at batch sizes 8–64? Can the baseline be run on vLLM or SGLang?
  4. Is the last token always active? Can a deactivated token be reactivated? Are loop counts fixed, or does Ouro's exit gate remain in use?
  5. How large is the normalization error introduced by freezing m across loops, measured on real decode traces?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4qTP (1/3)

We sincerely thank the reviewer for recognizing our measurement-driven design, the fair calibration, and our efficient implementation. We will provide detailed responses to your questions.

W1: "Lossless" is not supported, and no uncertainty is reported. MATH-500 has 500 items, so one SE is about 2.2 points; the gains attributed to sparse attention alone (+3.2 on MATH-500, Table 2) are within the same noise. Five-task averages hide task-level losses on precisely the reasoning tasks looped models are built for. Per-task confidence intervals or paired tests are needed, and "lossless" should be replaced with a quantified statement.

Q1: Can you report per-task 95% intervals (or paired bootstrap tests against the original model) for Table 1, and revise the "lossless" claim accordingly?

Reply:

Thank you for requesting a more rigorous assessment of performance preservation. We now report per-task paired-bootstrap 95% confidence intervals for the score differences between our method and the original model (FlashLoop − Original, in percentage points):

ModelMATH-500ARC-CGSM8KHellaSwagWinoGrande
Ouro-1.4B−0.40 [−3.80, +3.00]+0.69 [−0.51, +1.88]+0.84 [−1.21, +2.88]+0.09 [−0.27, +0.44]+0.64 [−1.34, +2.53]
Ouro-1.4B-Thinking−0.60 [−4.60, +3.40]−0.34 [−1.71, +1.02]−1.26 [−3.18, +0.68]−0.23 [−0.58, +0.14]−1.94 [−3.47, −0.39]
Ouro-2.6B+0.60 [−2.20, +3.40]−1.03 [−2.30, +0.26]+0.76 [−1.21, +2.81]+0.04 [−0.24, +0.32]−1.03 [−2.68, +0.63]
Ouro-2.6B-Thinking−3.40 [−7.20, +0.40]+1.17 [−0.43, +2.73]+0.47 [−1.29, +2.20]+1.03 [+0.59, +1.47]−1.06 [−2.60, +0.47]
Huginn-3.5B+2.40 [−0.20, +5.00]+0.51 [−1.19, +2.22]−1.82 [−3.87, +0.30]−0.06 [−0.51, +0.39]−0.16 [−1.82, +1.50]

The intervals include zero in 23 of the 25 model–task comparisons, but we agree that this does not establish equivalence or justify a "lossless" claim. In particular, Ouro-1.4B-Thinking shows a decrease on WinoGrande of 1.94 percentage points (95% CI: [−3.47, −0.39]), and some other comparisons have sufficiently wide intervals that meaningful degradation cannot be ruled out.

We have therefore removed the "lossless" claim throughout the manuscript and revised the presentation to "Our method accelerates looped Transformer inference while largely preserving benchmark performance, with task-dependent performance trade-offs".

W2: The accuracy evaluation does not match the regime the method targets. The savings grow with context length and loop count, but three of the five benchmarks (ARC-C, HellaSwag, WinoGrande) are short-prompt log-likelihood tasks on which KV memory is negligible. Standard long-context suites (e.g., RULER, LongBench) and long chain-of-thought generation with the Thinking models are needed, since errors from reused KV and stale column selections can accumulate over many decode steps.

Q2: How does FlashLoop perform on RULER or LongBench and on long chain-of-thought generation with the Thinking models at 8K–32K?

Reply:

We thank the reviewer for this suggestion. For long context, Appendix I (Table 8) of the paper reports WikiText-2 perplexity at 8K, 16K, and 32K and Needle-in-a-Haystack accuracy, where FlashLoop shows no significant degradation. We additionally evaluate Ouro-2.6B on LongBench v2 and RULER (16K context). For LongBench v2, the input length is 32K tokens and the longer inputs are truncated in the middle, keeping the beginning and the end.

MethodLongBench v2 (%)RULER 16K (%)
Original37.7780.73
FlashLoop37.5780.55

The results show that FlashLoop stays close to the original model on both benchmarks, with only a slight drop.

For long chain-of-thought generation, we compare the original model and FlashLoop on Ouro-1.4B-Thinking, using the first 100 problems of MATH-500 without any selection. We use greedy decoding with a generation budget of up to 65K tokens.

SettingCorrectFinished with EOSMean output length (tokens)
Ouro-1.4B-Thinking69/100 (69%)81/10014,902.7
+FlashLoop69/100 (69%)80/10014,703.2

FlashLoop preserves the accuracy of the original model, and we do not observe noticeable error accumulation over these long generations, which exceed ten thousand tokens on average.

AuthorsOpenReview ↗

Response to Reviewer 4qTP (2/3)

W3: Missing loop-axis baselines. The Huginn paper (Geiping et al.) itself describes zero-shot per-token adaptive exit and KV-cache sharing across recurrence steps, and Ouro has learned exit gates. These are the natural comparisons for token-sparse updates and cross-loop KV reuse, and neither is discussed.

Reply:

We thank the reviewer for pointing out these loop-axis mechanisms. In the revision we discuss both baselines in Related Work and Section 5.3.2, and add a direct comparison.

How these mechanisms relate to FlashLoop.

  • Adaptive exit (Huginn stops a token once the KL divergence between two successive recurrence steps falls below a threshold and Ouro's learned gate decides the exit step per token) acts on when a token stops recurring, which is served as a token-level early-exit mechanism.
  • KV-cache sharing (Huginn) keeps a fixed budget of cached recurrence steps and overwrites the oldest step's KV once the budget is exceeded. It is closely related to the Last-step KV Reuse baseline in Table 3, which reuses a single step's KV for every four steps.
  • FlashLoop is complementary to both. An exit is a one-way stopping decision for a token, whereas cross-loop token-sparse updates select, under a fixed budget, which tokens are updated in each loop based on their normalized hidden-state changes. Skipped tokens reuse the hidden states and KV cache of the preceding loop. Loop-aware sparse attention and cross-loop KV residual quantization further reduce which attention columns are read and how each KV entry is stored, two costs that neither early exit nor cache sharing addresses.

Direct comparison. We next compare FlashLoop head-to-head with these loop-axis mechanisms, used alone and combined with FlashLoop. The additional baselines use the settings from their own papers.

ModelMethodARC-C (%)SpeedupKV Memory
Huginn-3.5BOriginal (R=32)43.94-1.00×
Huginn-3.5BAdaptive exit (KL 5e-4)43.601.18×1.32×
Huginn-3.5BKV sharing (budget 8)41.470.99×4.00×
Huginn-3.5BFlashLoop44.451.52×5.18×
Huginn-3.5BFlashLoop + Adaptive exit44.111.71×5.43×
Ouro-1.4BOriginal (R=4)59.98-1.00×
Ouro-1.4BExit gate59.731.12×1.15×
Ouro-1.4BFlashLoop60.671.59×5.85×
Ouro-1.4BFlashLoop + Exit gate60.411.68×5.92×

The results show that, the early-exit mechanisms bring only modest savings in both speed and memory, KV-cache sharing saves memory at an accuracy cost, and FlashLoop reduces both while preserving accuracy. Enabling either exit mechanism on top of FlashLoop adds further speedup with little change in accuracy, so these loop-axis methods are complementary to FlashLoop.

W4: The efficiency measurements are narrow. All timing is on one A100-40GB, the batch size is not stated (apparently one), and the baseline is the authors' own engine with the features switched off rather than an optimized serving stack (vLLM or SGLang with FlashAttention and paged KV). Under batching, gathering irregular per-request token sets and column sets is harder to amortize, and the 1.5–1.6× speedup may shrink.

Q3: What batch size was used, and how do prefill latency, decode latency and throughput change at batch sizes 8–64? Can the baseline be run on vLLM or SGLang?

Reply:

We thank the reviewer for the question. The efficiency results in the paper use batch size 1. We add a batch sweep for Ouro-1.4B (BF16) on an A100 80GB and an H100 80GB. We report batch sizes 8 and 16, as at batch sizes 32 and 64 the baseline runs out of memory on both GPUs. Since stock vLLM and SGLang do not support looped models, we integrated FlashLoop into vllm-rlt, a vLLM-based serving stack for looped Transformers, and run both FlashLoop and the dense baseline there with 4K-token inputs, 32 output tokens, four loops, one warm-up and three measured repetitions:

GPUBatch sizeMethodTTFT (s)TPOT (ms)Output tokens/sPeak memory (GiB)
A1008Original5.784118.7427.1529.481
A1008FlashLoop3.178106.0239.7521.643
A10016Original10.367191.2431.7756.282
A10016FlashLoop6.360175.4543.8840.706
H1008Original3.74090.8239.0529.504
H1008FlashLoop1.95873.2460.5422.387
H10016Original6.342133.4349.4156.305
H10016FlashLoop3.666114.0471.8941.451

FlashLoop remains faster under batching. At batch sizes 8 and 16 on both GPUs, its prefill is 1.6–1.9× faster and its decodes is 1.1–1.2× faster, giving 1.4–1.6× higher output throughput with lower peak memory throughout. The overall throughput gain is close to the 1.5–1.6× end-to-end speedup reported at batch size 1 in Table 1.

AuthorsOpenReview ↗

Response to Reviewer 4qTP (3/3)

Q4: Is the last token always active? Can a deactivated token be reactivated? Are loop counts fixed, or does Ouro's exit gate remain in use?

Reply:

We thank the reviewer for these clarifying questions and answer them in turn.

Is the last token always active? Yes. During prefill, we always retain the last prompt token in the active set at every loop, because its final-loop hidden state is used to predict the first output token. During decoding, token-sparse updates are not applied: the current token remains active and its hidden state is updated at every loop.

Can a deactivated token be reactivated? No. As described in Section 4.1 and Eq. (5) in Appendix E.1, the active sets are nested within one prefill recurrence, so a token that leaves the active set reuses its preceding state for the remaining loops and will not be reactivated.

Are loop counts fixed, or does Ouro's exit gate remain in use? Loop counts are fixed. For both the original model and FlashLoop, we follow the default configuration of the official release, running four loops for Ouro (32 for Huginn) without the exit gate. FlashLoop is nevertheless compatible with the exit gate: as shown in our reply to W3, enabling it on top of FlashLoop adds further speedup with little change in accuracy.

Q5: How large is the normalization error introduced by freezing m across loops, measured on real decode traces?

Reply:

We thank the reviewer for this question. We measured the normalization error directly on real decode traces. For the same selected columns S and the same decode state, we compare the cached mass with the true mass of the current loop: $ m_{\mathrm{true}}^{(r)}=\sum_{j\in S}p_j^{(r)} $, and report the statistics of normalization error |m_cached − m_true| / m_true. where the mass lies in [0, 1]. On Ouro-1.4B, we record 64 decode steps for ten prompts with a 10% column budget. The results are as followed:

Cached → current loopMeanP95
2 → 30.02990.0784
3 → 40.01610.0430
2 → 4 (no intermediate refresh)0.03710.0951

The error is small and becomes smaller in the later loop transition. Even when the loop-2 mass is carried through loop 4 without a refresh, the error grows only moderately.

With these responses, we hope we have resolved your concerns. We sincerely hope you can consider adjusting your score upward.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer 4qTP,

For your queries, we carefully prepared the answers. We look forward to you considering raising your score if our response effectively addressed your question.

Thank you!

Authors

Comments on the paper

2 comments
Area_Chair_gccxOpenReview ↗

Participate in the discussion period

Dear reviewers,

The authors have replied to your comments and the discussion is close to the end. We encourage you to participate in the discussion to see whether your questions and concerns have been answered by the authors. Thank you!

Best, Your AC

We sincerely thank the reviewers for their insightful feedback.

We are encouraged to see that the reviewers acknowledged the paper's contribution in introducing a training-free inference framework whose components are each motivated by a measured cross-loop redundancy [Reviewers fZUk, 4qTP]; implementing optimized GPU kernels that deliver real wall-clock speedup and memory savings [Reviewers fZUk, L6is, 4qTP]; and evaluating on five Looped Transformer models with a single calibration per model that limits per-task tuning [Reviewers L6is, 4qTP].

We greatly appreciate the reviewers' efforts in evaluating our work, and the constructive suggestions are invaluable for further improving the paper, for example, extending the evaluation to coding and long-context benchmarks [Reviewers fZUk, 4qTP], reporting confidence intervals and comparing with loop-axis baselines and batched serving [Reviewer 4qTP], comparing latency with recent and tuned KV-cache baselines [Reviewer L6is], and breaking down the speedup into prefill and decode [Reviewer fZUk]. We have provided detailed point-by-point responses to each of the reviewers' comments in our individual replies.

The constructive suggestions allowed us to improve the paper, which we hope the reviewers will find to satisfactorily address the raised concerns. We uploaded a revised version of the paper incorporating the reviewers' suggestions. All changes are highlighted in blue color. We hope that based on the answers to their questions and the improvements made to the revised version, the reviewers will increase their scores.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.