- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies KV-cache compression for looped transformers, where recurrent execution causes cache memory to grow with the number of recurrences. The authors observe that recurrent KV states exhibit depth-dependent redundancy: deeper layers tolerate more aggressive cross-recurrence state sharing. Based on this, they propose a depth-aware allocator that jointly selects which recurrent states to retain, their quantization precision, and the coding scheme under a fixed memory budget. Candidate costs are calibrated using output KL divergence and combined through a dynamic-programming allocator. Experiments on Ouro-1.4B and Ouro-2.6B show lower KL divergence than a layer-wise mixed-precision baseline under tight cache budgets, with the learned allocations also transferring from 384-token calibration prompts to 2,048-token prompts.
Critical strengths
S1. The paper identifies a clear and nontrivial depth-dependent structure in recurrent KV redundancy.
The empirical analysis shows that replacing earlier recurrent states with the final recurrence is substantially less harmful in deeper layers, while shallow layers benefit more from retaining the first recurrence. This provides a useful observation specific to looped transformers and motivates going beyond conventional layer-wise KV quantization.
S2. The proposed allocator provides a simple and coherent way to jointly optimize state retention, quantization precision, and coding under a fixed memory budget.
The formulation unifies several compression choices within a common candidate space and reduces the allocation problem to a multiple-choice knapsack problem that can be solved efficiently with dynamic programming. Importantly, calibration only requires evaluating single-group modifications rather than enumerating all joint configurations, making the procedure relatively lightweight and training-free.
Critical weaknesses
W1. The evaluation does not yet establish practical generation quality or inference efficiency.
The main fidelity evaluation in Section 4 is teacher-forced: all methods receive the same ground-truth continuation tokens, and the primary metrics are next-token KL divergence and NLL rather than errors accumulated along self-generated trajectories. The only reported free-generation experiment, on GSM8K in Section 4.2, does not show a statistically resolved accuracy improvement. This is particularly important for recurrent KV compression, since an early cache-induced prediction error can alter subsequent tokens and recurrent states.
A stronger evaluation should include end-to-end autoregressive generation on representative tasks, e.g., GSM8K/MATH-500 for mathematical reasoning, HumanEval/MBPP for code generation, and LongBench/RULER-style long-context tasks if supported by the model. Metrics should include task accuracy/EM or pass@1 in addition to KL/NLL.
Moreover, although the paper is motivated by KV-cache efficiency, it reports storage reduction but no end-to-end system measurements. The evaluation should report peak KV/total memory, maximum batch size under a fixed memory budget, TTFT, TPOT/decode latency, tokens/s or requests/s, and throughput--latency trade-offs, including the runtime overhead of quantization, residual reconstruction, and heterogeneous cache layouts.
W2. Comparisons with closely related KV-compression and recurrent-state-reuse methods are insufficient.
Section 4.1 mainly compares the authors' nested candidate spaces , , and , so the reported improvement primarily establishes the benefit of enlarging their own allocation space rather than superiority over existing KV-compression methods.
At minimum, the evaluation should include LLA [1], which directly targets cross-loop KV compression, and KVTuner [2], which provides a closely related sensitivity-aware mixed-precision allocation baseline. Representative low-bit KV methods such as KIVI [3] and KVQuant [4], as well as reconstruction/rematerialization approaches such as AQUA-KV [5] and XQuant [6], would further clarify the advantage of jointly selecting recurrence retention and precision.
Simple recurrence-sharing baselines, such as first-state reuse, last-state reuse, and fixed first+last retention, should also be included under matched memory budgets. These controls are important for determining whether the improvement comes from the proposed depth-aware allocator or can largely be reproduced by a simple hand-designed sharing policy. Comparisons should preferably use matched total KV bytes/compression ratios, rather than nominal quantization bit widths.
W3. The evidence for generality across looped transformers, recurrence depths, and context lengths is limited.
All main experiments use only Ouro-1.4B and Ouro-2.6B, i.e., two sizes from the same model family, with the same recurrence structure. The longer-context experiment in Section 4.2 extends the prompt length only from 384 to 2,048 tokens. It is therefore unclear whether the central observation—that earlier recurrent states can be more aggressively shared in deeper layers—is specific to Ouro or generalizes to other recurrent architectures.
A particularly useful additional model would be Huginn [7], which provides a different recurrent-depth architecture. Mixture-of-Recursions [8] would further test the method under dynamic recursive depths, where a fixed depth-aware policy may behave differently. MELT [9] and Parallel Loop Transformer [10] could serve as additional architecture-level reference points because they adopt different mechanisms for reducing or reusing looped-model states.
The paper should also vary the recurrence count , where possible, and evaluate several context lengths (e.g., 2K/4K/8K or longer within the supported range). This would show whether the preferred retention pattern remains stable as either recurrence depth or context length increases.
[1] James O'Neill and Fergal Reid. Looped Latent Attention: Cross-loop KV Compression for Looped Transformers. arXiv:2607.15456, 2026.
[2] Xing Li, Zeyu Xing, Yiming Li, et al. KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference. ICML, 2025.
[3] Zirui Liu, Jiayi Yuan, Hongye Jin, et al. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. ICML, 2024.
[4] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, et al. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. NeurIPS, 2024.
[5] Alina Shutova, Vladimir Malinovskii, Vage Egiazarian, et al. Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models. ICML, 2025.
[6] Aditya Tomar, Coleman Hooper, Minjae Lee, et al. XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization. arXiv:2508.10395, 2025.
[7] Jonas Geiping, Sean McLeish, Neel Jain, et al. Scaling Up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. NeurIPS, 2025.
[8] Sangmin Bae, Yujin Kim, Reza Bayat, et al. Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation. NeurIPS, 2025.
[9] Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo, et al. Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models. arXiv:2605.07721, 2026.
[10] Bohong Wu, Mengzhao Chen, Xiang Luo, et al. Parallel Loop Transformer for Efficient Test-Time Computation Scaling. arXiv:2510.24824, 2025.
Questions for authors
Several recent looped-transformer architectures already incorporate cross-recurrence KV reuse, e.g., Mixture-of-Recursions reuses the first-recurrence KV states across later recurrences [8], while MELT [9] and Parallel Loop Transformer [10] introduce alternative mechanisms for sharing/reusing states across loops. In such models, a substantial fraction of the cross-recurrence KV redundancy may already be removed by the architecture itself. How much additional memory--quality or memory--efficiency advantage can the proposed depth-aware retention/allocation scheme still provide on top of these native KV-reuse mechanisms? It would be useful to evaluate whether the observed depth-dependent redundancy remains present, and whether the proposed allocator offers meaningful gains over the model's built-in reuse policy under matched memory budgets.
5 comments
acknowledge of response
Thanks for the rebuttal and additional experiments. These results address several concern. But still some concern remain:
-
the quality improvement show limited generalization. the results of GSM8K and MATH500 only improve with 1.4B model, on 2.6B the result looks similar.
-
The system benefit is unclear. The latency reduced when batch=1, and slower at batch=4, but for real inference service, the batch should be large, so the efficiency advantages is not convincing yet, maybe the implementation still have space to improve.
-
The presentation should be improved. The revised paper have no figure at all and end at page=8. I think there are still some space at lease 1 architecture or overview figure to help reader better understand this work. And the response about efficiency also only put numbers in the sentense, a short table or several comparison figure and help readers to easier find out the improvement or the quality-efficiency tradeoffs.
Follow-up to Reviewer SL2J
Follow-up to Reviewer SL2J
Thank you for the constructive suggestions. We added a larger-model evaluation, improved the higher-batch implementation, and made the quality–efficiency trade-off clearer.
On Ouro-2.6B MATH-500, the frozen RA policy scores 54.6% versus P's 48.8%: a gain of 5.8 percentage points (Holm p=0.0199, separate two-test supplementary family). Fixed first+last scores 51.6%; RA/FS on this task and RA/P on 2.6B GSM8K remain unresolved. Section 4.3 provides the counts, conservative intervals and protocol.
Following your implementation suggestion, tiled key reconstruction loads adjacent packed words while preserving the policy, quantization and rounding. On Ouro-1.4B C1 (4K prompt, 256 teacher-forced decode tokens, shared GPU), five randomized repetitions per configuration show median paired RA latency reductions of 51.3%/55.2% at batches four/eight versus its direct implementation. Both P and RA use this tiled reader below.
| Ouro-1.4B C1 policy | MATH-500 accuracy | Median decode ms/token, B4 / B8 | Total peak GiB, B4 / B8 |
|---|---|---|---|
| Precision-only (P) | 62.0% | 77.40 / 128.23 | 5.90 / 9.08 |
| Retention-aware (RA) | 70.6% | 77.16 / 130.69 | 5.71 / 8.71 |
Peak allocation includes model weights and covers prefill and decode.
RA is essentially tied with P at batch four and 2.1% slower at batch eight, with lower measured peak allocation. Quality and runtime use different reconstruction modes, with parity checked on specified inputs (Appendices D, G and H).
With precision fixed, retention alone reduces latency by 2.3–4.6%, with every repetition favoring retention in each control; byte costs differ. Under a common byte cap, RA uses the saved storage for higher precision. The 1.4B comparison shows the allocation's value: better task accuracy with higher-batch latency close to P.
The revised draft now includes a method overview in Figure 1 and compact quality, memory and runtime comparisons in Tables 6–7.
Response to Reviewer SL2J
Thank you for recognizing the depth observations and allocator, and for identifying the missing practical evidence. Your central concern led us to test whether choosing retention by depth improves generation under a fixed external-codec budget. The new controlled comparisons support this benefit on two 1.4B reasoning tasks, beyond both precision-only allocation and selected fixed sharing.
W1: errors along self-generated trajectories. Each arm generates greedily from its own evolving prefix, with policies frozen at cap C1:
These exact McNemar tests form one Holm family; intervals are descriptive. Caps are 512 tokens on GSM8K and 2,048 on MATH. Capped and unparsed outputs remain in the denominator. GSM8K questions were previously examined. Equal exact/RA MATH accuracy includes 34 harmful and 34 beneficial flips and does not establish equivalence.
W1: measured efficiency. At 4K, batch 1, Ouro-1.4B C1, RA takes 51.53 ms/token versus P's 57.46 under the same cap and optimized implementation: 10.3% lower latency, with all ten pairs favoring RA. Its total allocated peak, including the model, is 3.46 GiB versus P's 3.51 and exact's 5.90. At batch 4, RA is instead 9.4% slower than P (132.38 versus 121.02 ms). Exact remains faster; FS and prepared LLA-R are also faster at batch 1. Section 6 reports all comparisons; the incremental benefit over P is higher task accuracy and lower latency in this batch-one regime.
Fixed-precision controls show that retention itself saves latency at both batches, but higher precision can outweigh that saving. All 240 measurements completed without failure; monitoring at ten-second intervals observed no foreign GPU load. Timings include reconstruction and exclude tokenization, sampling, scoring and compilation. Quality runs retain reconstructed blocks (RA peak: 6.47 GiB at 4K, batch 1). All 16 mode-parity and 18 direct-reconstruction checks are bitwise equal on specified inputs, not every task trajectory. Serving throughput/capacity and the original residual codec's packed runtime remain unmeasured.
W2: could fixed sharing explain the improvement? First-only, last-only and first+last each receive precision allocation with the same codec, calibration inputs, selection procedure and maximum proposal-evaluation allowance as RA. Codes, metadata and exact tails count toward the common KV-payload cap; actual byte use and search consumption can differ. The best fixed rule is chosen on a separate selection set before evaluation, and is first+last in all four cells.
RA/best-fixed KL ratios are 0.75, 0.33, 0.71 and 0.44 (1.4B C1/C2, then 2.6B C1/C2), all Holm-significant. RA/P ratios 0.27, 0.34, 0.34 and 0.55 also pass their separate four-test family.
The 1.4B C1 supplement uses that frozen first+last policy and unchanged prompts, decoding and judges. FS scores 951/1287 (73.9%) on GSM8K and 332/500 (66.4%) on MATH. RA gains 2.7/4.2 pp, with 88/53 and 45/24 RA-only/FS-only correct pairs; both tests pass Holm over two (p=0.0081/0.0154). The predeclared conservative intervals [−0.2,+5.6]/[−1.1,+9.4] are not inverted McNemar intervals and can include zero despite rejection. This extension was specified after comparator outcomes were known; no policy was reselected. These results support selecting retention by depth within the tested pipeline.
W2: LLA and KVTuner. Section 7 reports both with explicit reproduction limits. KVTuner-Ouro omits pruning and clustering; 299/300 initial proposals were infeasible. The feasible-seeded extension (KT-F) is exploratory; RA/KT-F KL is 0.25 [0.21,0.29]. LLA-R is our trained streaming reconstruction, not author code; it omits LLA-2D and on-policy refinement. RA/LLA-R KL is 0.47/0.25 at the two 1.4B caps, and MATH accuracy is 70.6% versus 59.6% in a descriptive comparison. These results characterize the implementations tested, not the full published methods. KVQuant, AQUA-KV and XQuant remain unmeasured.
W3 / your question about native KV reuse. If an architecture already keeps one shared KV state at a layer, the additional retention choice can disappear. A gain from discarding further recurrent states cannot be assumed. Native-reuse gains, Huginn, other recurrence counts, code generation and long-context task accuracy remain untested. Our evidence covers Ouro-1.4B/2.6B at R=4, with fidelity at 2K prompts and systems measurements at 4K.
Sections 4, 6 and 8 now report the controlled comparisons, system costs and architecture limits. Thank you for motivating these tests of the allocation decisions' practical value.