acceptodds
Under review as a conference paper at ICLR 2027

Depth-Aware KV Cache Compression in Looped Transformers

Abstract

Looped transformers have advantages over their counterparts by reusing the same layer parameters across recurrences. Yet they typically store separate state caches for each recurrence step. The naive KV caching scheme would incur linearly growing overhead in terms of the number of recurrences, limiting the efficiency gain from looped transformers. There exist several techniques for mitigating this overhead by layer-wise mixed-precision KV quantization according to sensitivity, as well as predicting/reconstructing the states at runtime. Our investigation tries to discover connections between these two approaches through depth analysis. As observed, replacing cached states from earlier recurrences with the last recurrence's state causes less distortion in deep layers than in shallow layers. Contrarily, retaining all states with residual coding while increasing the precision of the first recurrence and reducing that of the last causes more distortion in deep layers than in shallow layers. Motivated by this finding, we introduce a depth-aware allocator that simultaneously selects which recurrent states to retain, along with quantization and coding schemes under certain capacity constraints. The allocator was calibrated on 16 documents and tested on 256 held-out documents for the models Ouro-1.4B and Ouro-2.6B. On the test set above, our allocator decreased KL divergence by 36% and 24%, respectively, compared with layer-wise mixed-precision allocation under the same tight storage budget. Furthermore, without recalibrating, allocations calibrated using 384-token prompts still reduced KL divergence by 57% and 33% w.r.t. the layer-wise allocation scheme on separate test documents with longer prompts of 2,048 tokens.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Transformers, KV Cache Compression, Mixed-Precision Quantization, Depth-Aware Allocation, Recurrent-State Retention, Calibration-Based Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SL2JOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies KV-cache compression for looped transformers, where recurrent execution causes cache memory to grow with the number of recurrences. The authors observe that recurrent KV states exhibit depth-dependent redundancy: deeper layers tolerate more aggressive cross-recurrence state sharing. Based on this, they propose a depth-aware allocator that jointly selects which recurrent states to retain, their quantization precision, and the coding scheme under a fixed memory budget. Candidate costs are calibrated using output KL divergence and combined through a dynamic-programming allocator. Experiments on Ouro-1.4B and Ouro-2.6B show lower KL divergence than a layer-wise mixed-precision baseline under tight cache budgets, with the learned allocations also transferring from 384-token calibration prompts to 2,048-token prompts.

Critical strengths

S1. The paper identifies a clear and nontrivial depth-dependent structure in recurrent KV redundancy.
The empirical analysis shows that replacing earlier recurrent states with the final recurrence is substantially less harmful in deeper layers, while shallow layers benefit more from retaining the first recurrence. This provides a useful observation specific to looped transformers and motivates going beyond conventional layer-wise KV quantization.

S2. The proposed allocator provides a simple and coherent way to jointly optimize state retention, quantization precision, and coding under a fixed memory budget.
The formulation unifies several compression choices within a common candidate space and reduces the allocation problem to a multiple-choice knapsack problem that can be solved efficiently with dynamic programming. Importantly, calibration only requires evaluating single-group modifications rather than enumerating all joint configurations, making the procedure relatively lightweight and training-free.

Critical weaknesses

W1. The evaluation does not yet establish practical generation quality or inference efficiency.
The main fidelity evaluation in Section 4 is teacher-forced: all methods receive the same ground-truth continuation tokens, and the primary metrics are next-token KL divergence and NLL rather than errors accumulated along self-generated trajectories. The only reported free-generation experiment, on GSM8K in Section 4.2, does not show a statistically resolved accuracy improvement. This is particularly important for recurrent KV compression, since an early cache-induced prediction error can alter subsequent tokens and recurrent states.

A stronger evaluation should include end-to-end autoregressive generation on representative tasks, e.g., GSM8K/MATH-500 for mathematical reasoning, HumanEval/MBPP for code generation, and LongBench/RULER-style long-context tasks if supported by the model. Metrics should include task accuracy/EM or pass@1 in addition to KL/NLL.

Moreover, although the paper is motivated by KV-cache efficiency, it reports storage reduction but no end-to-end system measurements. The evaluation should report peak KV/total memory, maximum batch size under a fixed memory budget, TTFT, TPOT/decode latency, tokens/s or requests/s, and throughput--latency trade-offs, including the runtime overhead of quantization, residual reconstruction, and heterogeneous cache layouts.

W2. Comparisons with closely related KV-compression and recurrent-state-reuse methods are insufficient.
Section 4.1 mainly compares the authors' nested candidate spaces LL, SS, and HH, so the reported improvement primarily establishes the benefit of enlarging their own allocation space rather than superiority over existing KV-compression methods.

At minimum, the evaluation should include LLA [1], which directly targets cross-loop KV compression, and KVTuner [2], which provides a closely related sensitivity-aware mixed-precision allocation baseline. Representative low-bit KV methods such as KIVI [3] and KVQuant [4], as well as reconstruction/rematerialization approaches such as AQUA-KV [5] and XQuant [6], would further clarify the advantage of jointly selecting recurrence retention and precision.

Simple recurrence-sharing baselines, such as first-state reuse, last-state reuse, and fixed first+last retention, should also be included under matched memory budgets. These controls are important for determining whether the improvement comes from the proposed depth-aware allocator or can largely be reproduced by a simple hand-designed sharing policy. Comparisons should preferably use matched total KV bytes/compression ratios, rather than nominal quantization bit widths.

W3. The evidence for generality across looped transformers, recurrence depths, and context lengths is limited.
All main experiments use only Ouro-1.4B and Ouro-2.6B, i.e., two sizes from the same model family, with the same R=4R=4 recurrence structure. The longer-context experiment in Section 4.2 extends the prompt length only from 384 to 2,048 tokens. It is therefore unclear whether the central observation—that earlier recurrent states can be more aggressively shared in deeper layers—is specific to Ouro or generalizes to other recurrent architectures.

A particularly useful additional model would be Huginn [7], which provides a different recurrent-depth architecture. Mixture-of-Recursions [8] would further test the method under dynamic recursive depths, where a fixed depth-aware policy may behave differently. MELT [9] and Parallel Loop Transformer [10] could serve as additional architecture-level reference points because they adopt different mechanisms for reducing or reusing looped-model states.

The paper should also vary the recurrence count RR, where possible, and evaluate several context lengths (e.g., 2K/4K/8K or longer within the supported range). This would show whether the preferred retention pattern remains stable as either recurrence depth or context length increases.

[1] James O'Neill and Fergal Reid. Looped Latent Attention: Cross-loop KV Compression for Looped Transformers. arXiv:2607.15456, 2026.

[2] Xing Li, Zeyu Xing, Yiming Li, et al. KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference. ICML, 2025.

[3] Zirui Liu, Jiayi Yuan, Hongye Jin, et al. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. ICML, 2024.

[4] Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, et al. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. NeurIPS, 2024.

[5] Alina Shutova, Vladimir Malinovskii, Vage Egiazarian, et al. Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Models. ICML, 2025.

[6] Aditya Tomar, Coleman Hooper, Minjae Lee, et al. XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerialization. arXiv:2508.10395, 2025.

[7] Jonas Geiping, Sean McLeish, Neel Jain, et al. Scaling Up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. NeurIPS, 2025.

[8] Sangmin Bae, Yujin Kim, Reza Bayat, et al. Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation. NeurIPS, 2025.

[9] Victor Conchello Vendrell, Arnau Padres Masdemont, Niccolò Grillo, et al. Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Models. arXiv:2605.07721, 2026.

[10] Bohong Wu, Mengzhao Chen, Xiang Luo, et al. Parallel Loop Transformer for Efficient Test-Time Computation Scaling. arXiv:2510.24824, 2025.

Questions for authors

Several recent looped-transformer architectures already incorporate cross-recurrence KV reuse, e.g., Mixture-of-Recursions reuses the first-recurrence KV states across later recurrences [8], while MELT [9] and Parallel Loop Transformer [10] introduce alternative mechanisms for sharing/reusing states across loops. In such models, a substantial fraction of the cross-recurrence KV redundancy may already be removed by the architecture itself. How much additional memory--quality or memory--efficiency advantage can the proposed depth-aware retention/allocation scheme still provide on top of these native KV-reuse mechanisms? It would be useful to evaluate whether the observed depth-dependent redundancy remains present, and whether the proposed allocator offers meaningful gains over the model's built-in reuse policy under matched memory budgets.

5 comments
AuthorsOpenReview ↗

Response to Reviewer SL2J

Thank you for recognizing the depth observations and allocator, and for identifying the missing practical evidence. Your central concern led us to test whether choosing retention by depth improves generation under a fixed external-codec budget. The new controlled comparisons support this benefit on two 1.4B reasoning tasks, beyond both precision-only allocation and selected fixed sharing.

W1: errors along self-generated trajectories. Each arm generates greedily from its own evolving prefix, with policies frozen at cap C1:

TaskRAPrecision-onlyDifference, pp [95% interval]Holm p
GSM8K, 1.4B, n=1,28776.6%72.9%+3.7 [+0.2, +7.2]0.0024
MATH-500, 1.4B, n=50070.6%62.0%+8.6 [+2.6, +14.5]0.000044
GSM8K, 2.6B, n=1,28777.8%76.6%+1.2 [−2.1, +4.5]0.303

These exact McNemar tests form one Holm family; intervals are descriptive. Caps are 512 tokens on GSM8K and 2,048 on MATH. Capped and unparsed outputs remain in the denominator. GSM8K questions were previously examined. Equal exact/RA MATH accuracy includes 34 harmful and 34 beneficial flips and does not establish equivalence.

W1: measured efficiency. At 4K, batch 1, Ouro-1.4B C1, RA takes 51.53 ms/token versus P's 57.46 under the same cap and optimized implementation: 10.3% lower latency, with all ten pairs favoring RA. Its total allocated peak, including the model, is 3.46 GiB versus P's 3.51 and exact's 5.90. At batch 4, RA is instead 9.4% slower than P (132.38 versus 121.02 ms). Exact remains faster; FS and prepared LLA-R are also faster at batch 1. Section 6 reports all comparisons; the incremental benefit over P is higher task accuracy and lower latency in this batch-one regime.

Fixed-precision controls show that retention itself saves latency at both batches, but higher precision can outweigh that saving. All 240 measurements completed without failure; monitoring at ten-second intervals observed no foreign GPU load. Timings include reconstruction and exclude tokenization, sampling, scoring and compilation. Quality runs retain reconstructed blocks (RA peak: 6.47 GiB at 4K, batch 1). All 16 mode-parity and 18 direct-reconstruction checks are bitwise equal on specified inputs, not every task trajectory. Serving throughput/capacity and the original residual codec's packed runtime remain unmeasured.

W2: could fixed sharing explain the improvement? First-only, last-only and first+last each receive precision allocation with the same codec, calibration inputs, selection procedure and maximum proposal-evaluation allowance as RA. Codes, metadata and exact tails count toward the common KV-payload cap; actual byte use and search consumption can differ. The best fixed rule is chosen on a separate selection set before evaluation, and is first+last in all four cells.

RA/best-fixed KL ratios are 0.75, 0.33, 0.71 and 0.44 (1.4B C1/C2, then 2.6B C1/C2), all Holm-significant. RA/P ratios 0.27, 0.34, 0.34 and 0.55 also pass their separate four-test family.

The 1.4B C1 supplement uses that frozen first+last policy and unchanged prompts, decoding and judges. FS scores 951/1287 (73.9%) on GSM8K and 332/500 (66.4%) on MATH. RA gains 2.7/4.2 pp, with 88/53 and 45/24 RA-only/FS-only correct pairs; both tests pass Holm over two (p=0.0081/0.0154). The predeclared conservative intervals [−0.2,+5.6]/[−1.1,+9.4] are not inverted McNemar intervals and can include zero despite rejection. This extension was specified after comparator outcomes were known; no policy was reselected. These results support selecting retention by depth within the tested pipeline.

W2: LLA and KVTuner. Section 7 reports both with explicit reproduction limits. KVTuner-Ouro omits pruning and clustering; 299/300 initial proposals were infeasible. The feasible-seeded extension (KT-F) is exploratory; RA/KT-F KL is 0.25 [0.21,0.29]. LLA-R is our trained streaming reconstruction, not author code; it omits LLA-2D and on-policy refinement. RA/LLA-R KL is 0.47/0.25 at the two 1.4B caps, and MATH accuracy is 70.6% versus 59.6% in a descriptive comparison. These results characterize the implementations tested, not the full published methods. KVQuant, AQUA-KV and XQuant remain unmeasured.

W3 / your question about native KV reuse. If an architecture already keeps one shared KV state at a layer, the additional retention choice can disappear. A gain from discarding further recurrent states cannot be assumed. Native-reuse gains, Huginn, other recurrence counts, code generation and long-context task accuracy remain untested. Our evidence covers Ouro-1.4B/2.6B at R=4, with fidelity at 2K prompts and systems measurements at 4K.

Sections 4, 6 and 8 now report the controlled comparisons, system costs and architecture limits. Thank you for motivating these tests of the allocation decisions' practical value.

Reviewer_SL2JOpenReview ↗

acknowledge of response

Thanks for the rebuttal and additional experiments. These results address several concern. But still some concern remain:

  1. the quality improvement show limited generalization. the results of GSM8K and MATH500 only improve with 1.4B model, on 2.6B the result looks similar.

  2. The system benefit is unclear. The latency reduced when batch=1, and slower at batch=4, but for real inference service, the batch should be large, so the efficiency advantages is not convincing yet, maybe the implementation still have space to improve.

  3. The presentation should be improved. The revised paper have no figure at all and end at page=8. I think there are still some space at lease 1 architecture or overview figure to help reader better understand this work. And the response about efficiency also only put numbers in the sentense, a short table or several comparison figure and help readers to easier find out the improvement or the quality-efficiency tradeoffs.

AuthorsOpenReview ↗

Follow-up to Reviewer SL2J

Follow-up to Reviewer SL2J

Thank you for the constructive suggestions. We added a larger-model evaluation, improved the higher-batch implementation, and made the quality–efficiency trade-off clearer.

On Ouro-2.6B MATH-500, the frozen RA policy scores 54.6% versus P's 48.8%: a gain of 5.8 percentage points (Holm p=0.0199, separate two-test supplementary family). Fixed first+last scores 51.6%; RA/FS on this task and RA/P on 2.6B GSM8K remain unresolved. Section 4.3 provides the counts, conservative intervals and protocol.

Following your implementation suggestion, tiled key reconstruction loads adjacent packed words while preserving the policy, quantization and rounding. On Ouro-1.4B C1 (4K prompt, 256 teacher-forced decode tokens, shared GPU), five randomized repetitions per configuration show median paired RA latency reductions of 51.3%/55.2% at batches four/eight versus its direct implementation. Both P and RA use this tiled reader below.

Ouro-1.4B C1 policyMATH-500 accuracyMedian decode ms/token, B4 / B8Total peak GiB, B4 / B8
Precision-only (P)62.0%77.40 / 128.235.90 / 9.08
Retention-aware (RA)70.6%77.16 / 130.695.71 / 8.71

Peak allocation includes model weights and covers prefill and decode.

RA is essentially tied with P at batch four and 2.1% slower at batch eight, with lower measured peak allocation. Quality and runtime use different reconstruction modes, with parity checked on specified inputs (Appendices D, G and H).

With precision fixed, retention alone reduces latency by 2.3–4.6%, with every repetition favoring retention in each control; byte costs differ. Under a common byte cap, RA uses the saved storage for higher precision. The 1.4B comparison shows the allocation's value: better task accuracy with higher-batch latency close to P.

The revised draft now includes a method overview in Figure 1 and compact quality, memory and runtime comparisons in Tables 6–7.

Reviewer_kQFDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies KV-cache compression for looped Transformers. Matched interventions on Ouro-1.4B and Ouro-2.6B show that state retention and precision choices have depth-dependent fidelity costs. A training-free allocator chooses a complete history scheme per layer group, including retained loops, precision, and direct or residual coding. Costs are calibrated on 16 documents and a multiple-choice knapsack selects schemes under a byte budget. The analysis gives sufficient conditions for calibration gains to transfer despite interaction effects. Held-out tests show lower next-token KL than layer-wise precision allocation at selected tight budgets. A free-generation test on 1,287 GSM8K questions reports unresolved accuracy differences without absolute scores.

Critical strengths
  1. A controlled comparison of allocation choices.

The nested candidate sets share calibration data, cost evaluation, solver, and budget. The layer-wise baseline is included in the expanded search space, making the observed fidelity gains informative about the added retention and coding options. Cost-table construction scales with the sum of candidate counts rather than enumerating all joint combinations.

  1. Careful matching and statistical reporting for the core fidelity experiments.

The depth interventions control storage and precision, while paired tests, Holm correction, and confidence intervals support the main comparisons. The transfer analysis explicitly accounts for group interactions and calibration-to-test discrepancies instead of treating additive calibration as an unconditional guarantee.

Critical weaknesses
  1. The supported fidelity gains have not yet been connected to downstream accuracy gains.

The controlled KL and ΔNLL experiments do provide evidence for the paper’s fidelity objective, including significant tight-budget improvements on both models. Section 4.2 also reports free generation on 1,287 GSM8K questions. However, absolute accuracies and uncertainty are omitted, and differences from uniform residual two-bit coding remain unresolved. This leaves the practical meaning of the KL improvements unclear; unresolved differences do not establish either superiority or equivalence.

  1. External comparisons and architectural generality remain limited.

Most comparisons use the nested L/S/H candidate sets on two models from the Ouro family. Some detailed intervention findings are shown only for Ouro-1.4B, and certain gains are unresolved on Ouro-2.6B. Comparisons with relevant looped-model cache approaches would help establish value beyond the internal allocation baseline. Generic quantizers such as KIVI or KVQuant could also be informative. Their retained-state choices and total storage, including metadata, must match; nominal bit rates alone are insufficient.

Overall assessment: I lean toward weak rejection. The paper offers a well-controlled fidelity study, but the downstream consequences and value relative to independent cache methods remain insufficiently established for my recommendation.

Questions for authors
  1. Please report the existing GSM8K experiment’s absolute accuracies, sample counts, and paired uncertainty. Include the uncompressed model and tested uniform/L/S/H allocations, specifying their budgets.

  2. At matched bytes and state retention, how does the method compare with loop-aware cache methods or adapted quantizers?

  3. Which depth trends and allocation gains hold on both Ouro scales? Which remain specific to one scale or budget? Please distinguish these from hypotheses about other recurrent architectures.

3 comments
AuthorsOpenReview ↗

Response to Reviewer kQFD

Thank you for asking for task statistics, matched comparisons and a clearer account of scope. We agree that the original GSM8K reporting was incomplete. We now report it fully and separately from Linux reruns. The new controlled KIVI study shows task benefits on 1.4B, while the original comparison remains unresolved.

Q1: the existing GSM8K experiment, including its budget. Each model uses 1,287 official test questions (indices 32–1318), three demonstrations, independent greedy generation and a 512-token cap. All questions remain in the denominator; unparsed answers count as incorrect. The allocation fits the uniform residual two-bit byte cap (λ=0). At this cap, L is uniform residual two-bit.

Original experimentOuro-1.4BOuro-2.6B
Exact: correct / 1,287 (accuracy)981 (76.22%)1024 (79.56%)
Uniform residual two-bit980 (76.15%)1018 (79.10%)
Submitted allocation, MPS-calibrated975 (75.76%)1029 (79.95%)
Allocation − uniform, pp [95% interval]−0.39 [−2.94, +2.17]+0.85 [−1.44, +3.14]
Holm-adjusted McNemar p0.69650.5445

Allocation-only/uniform-only correct counts are 50/55 and 47/36. Intervals use the recorded conservative paired Bonferroni Clopper–Pearson construction and are not simultaneous across contrasts; the two tests use Holm correction. Neither superiority nor equivalence is established.

Linux recalibration changed the allocations. Their separate reruns score 965/1287 and 1025/1287, differing from uniform by −1.2 pp [−3.7, +1.3] and +0.5 pp [−1.5, +2.6], adjusted p = 0.33 and 0.46. These do not replace the original numbers above. Generation tested exact, uniform and one allocation per model from the original 12-option menu, which omits the later all-state residual four-bit candidate. We do not have a complete L/S/H task ablation.

The new KIVI study adds the following task evidence, with policies frozen before generation:

Accuracy (%)ExactPrecision-only (P)Retention-aware (RA)RA − P, pp [95% interval]
GSM8K 1.4B76.272.976.6+3.7 [+0.2, +7.2]
GSM8K 2.6B79.676.677.8+1.2 [−2.1, +4.5]
MATH-500 1.4B70.662.070.6+8.6 [+2.6, +14.5]

The three-test Holm-adjusted p values are 0.0024, 0.303 and 0.000044. MATH uses all 500 problems, greedy decoding, a 2,048-token cap and boxed-answer extraction with symbolic checking. GSM8K reuses previously examined questions. This is a separate codec and cap from the original experiment.

Q2: independent methods, retained states and total bytes. We fix the KIVI quantizer, geometry and KV-payload cap, counting codes, scales, zeros and exact tails. P optimizes precision while retaining all states; RA also selects retention. Fixed first-only, last-only and first+last controls each optimize precision with the same procedure. Retention intentionally varies in this control; it tests the added allocation choice under a fixed codec, not quantizer superiority at identical retained states.

The fixed rule is chosen on the selection set before evaluation. RA/P KL ratios are 0.27–0.55 and RA/best-fixed ratios 0.33–0.75 across both models and caps; all eight tests pass their four-test Holm families. Actual byte use can differ within the cap. The completed 1.4B C1 supplement gives FS-first+last 951/1287 and 332/500 correct. RA gains 2.7/4.2 pp (Holm p=0.0081/0.0154 in a new two-test family). Conservative intervals [−0.2,+5.6]/[−1.1,+9.4] include zero; they are not inverted McNemar intervals. This supplement was specified after comparator outcomes were known, without reselection.

KVTuner-Ouro omits precision-pair pruning and layer clustering. Its first search had 299/300 infeasible proposals; the exploratory feasible-seeded variant (KT-F) had 203/297 feasible and selected an initial seed. RA/KT-F KL is 0.25 [0.21,0.29]. Our distilled LLA-R reconstruction omits LLA-2D and on-policy refinement; RA/LLA-R KL is 0.47 [0.39,0.56] and 0.25 [0.20,0.31] at the two 1.4B caps. These are descriptive comparisons, not rankings of the full published methods. Section 7 details the adaptations; Section 6 reports their actual memory and runtime.

Q3: what holds on both scales? Last-state reuse is less harmful at depth on both models, but its deep/shallow KL ratio is 0.061 on 1.4B versus 0.51 on 2.6B. The opposite depth ordering for the precision-shift intervention was tested on 1.4B only. Both scales show tight-budget internal H/L gains, 2K-prompt fidelity gains and external-codec RA gains over P and fixed sharing at both caps. Task superiority over P is resolved only for 1.4B. Internal-codec task gains, a general H-over-S advantage, other architectures and other recurrence counts remain unestablished. Thank you for helping us make these distinctions explicit.

Thank you for the detailed response. The fixed-codec KIVI study addresses my main concerns. Retention-aware allocation lowers KL relative to precision-only allocation and to fixed sharing on both models. On Ouro-1.4B it improves GSM8K and MATH-500 accuracy over precision-only allocation and matches the uncompressed model.

However, task gains are shown only on Ouro-1.4B, and the selected allocations are sensitive to the calibration backend. Exact inference is still faster, and throughput is not measured. Therefore, I will keep my original rating.

AuthorsOpenReview ↗

Follow-up to Reviewer kQFD

Thank you for recognizing the evidence from the fixed-codec study. We would like to highlight two additions in the latest revision that directly address your remaining concerns.

On Ouro-2.6B MATH-500, the frozen RA policy achieves 54.6% versus P's 48.8%, a gain of 5.8 percentage points (Holm-adjusted p=0.0199, separate two-test supplementary family; Section 4.3). This extends the task-level evidence to the larger model; the 2.6B GSM8K comparison remains unresolved.

We also improved packed-key reconstruction while preserving the allocation policy and arithmetic. On Ouro-1.4B C1 at a 4K prompt, RA's median paired latency falls by 51.3%/55.2% at batches 4/8 relative to its direct implementation. With both RA and P using the improved reader, RA has nearly equal latency at batch 4 and is 2.1% slower at batch 8, with lower total peak allocation (Table 6). These shared-GPU measurements support improved quality at comparable higher-batch latency within the compressed pipeline; serving throughput remains unmeasured.

We also retain the calibration-backend sensitivity as a limitation and report the recalibrated results separately. Thank you for helping us clarify the contribution and its scope.

Reviewer_t9MKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies KV cache compression for looped transformers, where each recurrence stores its own KV states and cache grows linearly with recurrence count. The authors observe that the cost of replacing early recurrent states with the last state is depth-dependent: cheaper in deep layers, more costly in shallow layers, while precision shifting under residual coding shows the opposite trend. Based on this, they propose a depth-aware allocator that jointly selects which recurrent states to retain, their quantization precision, and direct/residual coding, formulated as a multiple-choice knapsack solved by dynamic programming over per-group calibrated KL costs. They also provide a finite-change analysis bounding interaction effects across groups. Experiments on Ouro-1.4B/2.6B show 36%/24% KL reduction versus layer-wise mixed-precision allocation under the same budget.

Critical strengths
  1. The depth-dependent observation is nontrivial and well supported. The finding that state-substitution cost and precision-shift cost have opposite depth profiles is counterintuitive and directly motivates the joint allocator. The controlled comparisons (equal layer counts, equal storage, paired documents, Holm-corrected tests) make this convincing and it is the main reason I lean toward acceptance.

  2. The allocation framework is practical and the gains are consistent. Calibrating only 1+Σ(Kg−1) configurations and reusing the allocation at test time is cheap, and the gains hold on held-out documents and at 2,048-token prompts without recalibration. This suggests the method is not overfitting to calibration data.

Critical weaknesses
  1. The evaluation is narrow relative to the claims. All results are on two Ouro models, next-token KL on WikiText-2/CPython, and the GSM8K free-generation result is explicitly unresolved. Since the paper's motivation is inference efficiency for reasoning-capable looped models, the absence of any task-level gain (accuracy, latency, actual memory measurements) makes the practical significance of a 36% KL reduction unclear.

  2. The finite-change analysis is not connected to the method's actual behavior. Proposition 3.1 gives a sufficient condition, but the bounds ηgh and δa,b are never estimated, so the condition is not verified for any selected allocation. The post-hoc decomposition in Table 2 shows only ~55% of single-group gain survives, with an interaction penalty interval crossing zero. This weakens the theoretical contribution, which reads as separate from the empirical pipeline rather than supporting it.

Questions for authors
  1. Can you report actual wall-clock latency and peak memory for compressed vs. uncompressed inference, including the reconstruction overhead of residual coding? KL alone does not establish an efficiency benefit.
  2. At λ=1/2 on Ouro-2.6B the gain is unresolved (KL ratio 1.07). Does the method's advantage vanish as the budget loosens, and if so, what is the operating regime you recommend?
1 comment
AuthorsOpenReview ↗

Response to Reviewer t9MK

Thank you for highlighting the depth observations and asking us to demonstrate their practical value. We now report task accuracy and measured inference costs. The clearest benefit is better use of a constrained cache budget: with KIVI fixed, retention choices improve two 1.4B reasoning tasks over precision-only allocation and provide a measured batch-one latency reduction.

Q1 / W1: actual latency and peak memory, including reconstruction. We measured a packed KIVI-retention adapter using frozen Ouro-1.4B C1 policies, a 4K prompt, 256 decode tokens and ten randomized repetitions after three warm-ups on an RTX 6000 Ada. P and RA use the same optimized reconstruction path.

Batch 1 measurementExactPrecision-only (P)Retention-aware (RA)
Total peak allocated, GiB5.903.513.46
Total peak reserved, GiB7.253.733.77
Prefill forward time, ms631.3678.1662.6
Decode forward time/token, ms34.5857.4651.53

RA reduces decode latency against P by 10.3% under the same cap; all ten paired ratios favor RA (0.893–0.906). At batch 4, RA is 9.4% slower than P (132.38 versus 121.02 ms), while its allocated peak is 5.71 GiB versus exact's 15.56. Exact remains faster at both batches. Holding precision fixed, retention saves about 10%/2% latency at batch 1/4; the higher precision chosen by RA outweighs that saving at batch 4. Section 6 and Appendix G report every adapted baseline and implementation control; FS and prepared LLA-R are faster at batch one.

All 240 measurements completed without failure; monitoring at ten-second intervals observed no foreign GPU load. Timings are synchronized forward passes, including reconstruction but excluding tokenization, sampling, scoring and compilation. Peaks include the model and actual codec parameters, but exclude CUDA-context and external allocations. The submitted residual evaluator reconstructs floating-point tensors; its logical byte count is not a measured memory saving, and its packed runtime remains unmeasured. Serving capacity is also unmeasured.

Quality runs retain reconstructed blocks and use more memory (RA: 6.47 GiB at 4K, batch 1). All 16 mode-parity checks and 18 direct-reconstruction checks pass with bitwise-equal K/V, full logits and predictions on their specified inputs. They do not cover every task trajectory.

For W1's task-level question, frozen 1.4B KIVI policies improve GSM8K from P's 72.9% to RA's 76.6%, and MATH-500 from 62.0% to 70.6%; both RA/P tests pass Holm correction. The fixed-sharing supplement shows RA gains of 2.7/4.2 pp over the already selected first+last rule (Holm p=0.0081/0.0154, separate two-test family). Its conservative intervals, [−0.2,+5.6]/[−1.1,+9.4], are not inversions of the paired tests. This supplement was specified after comparator outcomes were known, without reselection. The 2.6B GSM8K difference against P remains unresolved.

Q2: does the advantage vanish at a looser budget? The submitted 2.6B ratio of 1.07 remains unresolved for that original policy. It cannot establish either an advantage or its absence. Necessary Linux recalibration selected different policies, whose results we label separately:

Internal-codec H/L1.4B2.6B
λ = 1/40.64, significant0.76, significant
λ = 1/20.87, adjusted p = 0.0600.81, significant

Significance uses the six-test Holm family. Under KIVI, RA/P KL is 0.27/0.34 at the tighter cap and 0.34/0.55 at the looser cap for 1.4B/2.6B, with all four tests significant. The supported regime is cache-constrained fidelity improvement on these Ouro workloads, with particularly clear tight-budget evidence. The largest menu is not uniformly best; the measured latency benefit is specific to batch one.

W2: what connects Proposition 3.1 to the actual allocator? We have not estimated uniform η or δ, and therefore have not verified the proposition's sufficient condition for a selected policy. The revised text makes this explicit. The measured decomposition instead diagnoses how much additive gain survives interactions:

×10⁻³ nats, mean [95% interval]1.4B2.6B
Sum of single-group gains11.7 [3.9, 21.8]1.7 [0.8, 2.6]
Interaction penalty4.9 [1.1, 9.7]0.7 [0.1, 1.4]
Joint net gain6.8 [2.5, 12.3]0.9 [0.3, 1.5]

These Linux BF16 measurements condition on policies selected on the same 16 documents; they are not independent validation or a like-for-like rerun of the submitted FP32 decomposition. About 55–60% of the single-group gain survives, but positive observed net gains do not bound worst-case interactions or transfer error.

Sections 4.3 and 6 now report task accuracy and measured costs; Section 3.4 and Appendix A clarify the proposition's conditions. Thank you for helping us make the practical contribution and its limits explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.