- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes 3 ways to speed up inference in looped transformer models.
- First, they observe that in deeper loops state changes become increasingly concentrated on only a few tokens (~x %). So tokens which have converged can fully skip later loops (a form of early exit).
- Second, cross-loop differences in attention outputs are dominated by a sparse subset of columns whose indices can be reliably predicted from the preceding loop.
- Residual KV caches between consecutive loops become more amenable to quantization.
All 3 put together gives us FlashLoop which on the authors' ablations yields up to 1.64 speedup in decoding and up to 6x reduction in KV cache memory.
- Later loops rank tokens updated in earlier loops according to their normalized hidden state changes and retain only top-k for further refinement.
- Similarly, later loops rank attention columns and only keep top-k for recomputation.
- QUantize the KV cache and the KV residual updates of later loops separately.
The authors implement an optimized kernel on GPUs for these changes and test them on models from Ouro family and Huginn family. Across 5 reasoning benchmarks they observe no loss in quality with FlashLoop while observing significant speedups and KV memory savings.
Critical strengths
- The proposed changes are all inference time changes and require no re-training of models.
- The authors took the effort to implement an optimized GPU kernel for their changes to make them truly see wall-clock time savings.
Critical weaknesses
- For a paper claiming quality neutrality of an inference speedup technique, it is important to see more than 5 eval benchmarks. Especially ones spread across diverse domains such as math, coding, memorization intensive tasks and other areas. It is important to understand regressions in any aspect of the model when applying such changes.
Questions for authors
- The rationale provided for why KV residuals are easier to quantize than the KV states themselves is a bit unclear. Especially since you end up quantizing the base KVs as well. Why did we need to consider residuals and quantize them separately if the original KV is also being quantized?
- Can we breakdown time savings for prefill vs decode separately? will be interesting to see the differences between the two paradigms.
4 comments
Response to Reviewer fZUk (2/2)
Q2: Can we breakdown time savings for prefill vs decode separately? will be interesting to see the differences between the two paradigms.
Reply:
We thank the reviewer for the question. Fig. 4 of the paper reports prefill and decode latency separately as the three components are enabled cumulatively. We list the corresponding numbers below (Ouro-1.4B, 8K context, one A100-40GB):
| Configuration | Prefill (ms) | Prefill speedup | Decode (ms/token) | Decode speedup |
|---|---|---|---|---|
| Original | 1456 | 1.00× | 72 | 1.00× |
| + Token-Sparse Updates | 645 | 2.26× | 158 | 0.45× |
| + Sparse Attention | 645 | 2.26× | 98 | 0.73× |
| + 4-bit KV Residual Quant. (FlashLoop) | 921 | 1.58× | 44 | 1.63× |
The two phases benefit from different components. Prefill savings come from cross-loop token-sparse updates, which recompute only the tokens that continue to change in later loops. Sparse attention has little effect on prefill, and quantization adds work during prefill, but the complete FlashLoop remains faster than the original model. Decode savings come from loop-aware sparse attention and KV residual quantization, which reduce the KV states read for each generated token. Token-sparse updates alone increase decode latency, and sparse attention offsets much of this increase, and quantization brings the decode latency below that of the original model.
Dear Reviewer fZUk,
Thanks for your positive initial review. If our response effectively resolves your issue, we'd be grateful if you consider further improving your score.
Thank you!
Authors
Thanks for the response
Thank you to the authors' response. The added numbers help alleviate some of my concerns but I will maintain my current rating based on the strengths of the paper.
Response to Reviewer fZUk (1/2)
We sincerely thank the reviewer for recognizing our training-free design and our efficient implementation. We will provide detailed responses to your questions.
Reply:
We thank the reviewer for the suggestion. In addition to the five benchmarks in Table 1, which cover math (MATH-500, GSM8K) and general reasoning (ARC-C, HellaSwag, WinoGrande), Appendix I (Table 8) of the paper reports long-context results, i.e., WikiText-2 perplexity at 8K, 16K, and 32K and Needle-in-a-Haystack accuracy, where FlashLoop shows no significant degradation. We further add four benchmarks covering coding (HumanEval, MBPP) and memorization-intensive tasks (LongBench v2, RULER).
For coding, we evaluate HumanEval and MBPP, using the same configuration as Table 1:
FlashLoop retains most of the coding accuracy of the original models on both benchmarks, with a smaller drop on the larger model.
For memorization-intensive tasks, which we understand as tasks where the model has to retain and retrieve information from a long context, we evaluate Ouro-2.6B on LongBench v2 (32K input) and RULER (16K context), using the same FlashLoop setting as in the paper:
The results show that FlashLoop stays close to the original model on both long-context benchmarks.
Reply:
We thank the reviewer for the question and would like to first clarify what is quantized. FlashLoop does not quantize the full KV states of every loop. A looped Transformer produces and stores a separate KV cache in every loop. In FlashLoop, the full KV states are quantized only once, for the first loop, and serve as the base. For every later loop, the full KV states are never quantized and we quantize only their residual relative to the preceding loop (Eq. (4)).
Why residuals are easier to quantize. We explain this from two perspectives.
(1) Numerical range. The cross-loop KV residuals have a much smaller magnitude than the full KV states (Fig. 3(c, d)), so the same number of quantization levels covers a much narrower value range. As a result, the residuals have a much lower quantization error than the full KV states at the same bit-width (Fig. 2(c)), and replacing residual quantization with direct quantization of the full KV states leads to a noticeably larger accuracy degradation (Table 2).
(2) Cross-loop redundancy. The KV states of adjacent loops are highly similar, since all loops compute keys and values with the same projection weights and the hidden states change only slightly from one loop to the next (Section 3.1). Quantizing the full KV states of every loop stores this shared information repeatedly. FlashLoop instead stores it once in the base, so that the bits of each later loop represent only its loop-specific difference.