- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
LLA compresses the KV cache across loops in weight-tied Transformers. It stores compact latents and reconstructs loop-specific keys and values when needed. The paper also studies how cache errors affect real generation and uses further training to improve stability.
Critical strengths
- The paper identifies cross-loop KV redundancy and shows that it can be effectively compressed.
- The method reduces cache memory while keeping good accuracy at moderate compression, especially on Ouro-2.6B.
- The analysis of cache drift during real generation is valuable. The proposed training changes help close the gap with oracle evaluation.
Critical weaknesses
- The contribution is somewhat incremental, as low-rank KV compression and cross-loop cache sharing have been explored before.
- The paper discusses MELT in the related work but does not compare with it experimentally.
- Prefill is much slower in the current implementation, and generation quality still drops at high compression ratios.
Questions for authors
- Please directly compare LLA with MELT in accuracy, cache memory, and inference speed under comparable training and evaluation settings.
- Can the two-pass prefill be improved, and when do decoding gains outweigh its extra cost?
4 comments
We would be grateful to learn whether our reply and the revised PDF address the reviewer's concerns. In brief: the prompt is now encoded at the teacher's own prefill time (Table 13 of the revision), the end-to-end tables give the crossover (Tables 16–18 and 22), MELT is placed side by side under each paper's published protocol (Table 30), and the contribution rests on the cardinality-matched controls (Table 7) and the Huginn sharing comparison (Table 33). The revision also scores served GSM8K under exactly the configuration the serving tables time (Table 24): the second prefill pass is an option rather than a requirement, and the accuracy tables hold for the decoder the serving tables time. We are happy to answer anything that remains before the discussion closes.
Thank you for the detailed responses and additional experiments. My questions have been addressed. However, given the prior work, I still consider LLA’s novelty limited. I therefore maintain my rating of 2.
[1] Geiping et al., 2025. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach.
[2] Vendrell et al., 2026. Memory-Efficient Looped Transformer.
[3] Blayney et al., 2026. A Mechanistic Analysis of Looped Reasoning Language Models.
We thank the reviewer for confirming that the questions are addressed, and we write this for the area chair as much as for the reviewer, since the one remaining ground is novelty against [1]–[3]. All three are cited and discussed in the submission (Sec. 2; Tables 1, 30 and 33), and none of them contains the paper's mechanism or its finding.
[1] Geiping et al. (2025) is the Huginn architecture, which the paper uses as its second backbone. Its cache-memory contribution is zero-shot KV-cache sharing (its Sec. 6.2): a fixed slot budget per token that later recurrence steps overwrite, with no learned component. The paper measures that exact mechanism on Huginn (Table 33): it is lossless to its floor of 16.5×, beyond which only the codec continues (26.4× composed with it), and on Ouro training-free loop reuse applied to the prompt scores 0.000 GSM8K at 4× where the codec scores 0.800 (Table 1). LLA is not sharing: it is a learned low-rank codec of the per-loop K/V trajectory, and the cardinality-matched controls (Table 7: random and single-loop encoders at the same effective rank and cache budget) fail where the trajectory codec succeeds, which sharing-style reasoning does not predict.
[2] Vendrell et al. (2026), MELT, appeared on arXiv on 8 May 2026 and has since been an ICML 2026 workshop poster; its code and checkpoints are unreleased. The submission names it the closest concurrent work (Sec. 2) and places the published numbers side by side (Table 30). The two compress different objects: MELT fine-tunes the whole looped model (1,000 steps, 320M tokens, about 720 H100-hours) to distil the loop trajectory into a single gated state per layer, which caps the ratio at the loop count (3.98× at T = 4); LLA keeps the backbone frozen (2.3 B200-hours per codec, 101M codec parameters), keeps the trajectory, sets the ratio by (r_k, r_v) and reaches 4–21× on the same base model, composing with token eviction (Tables 28–29) and with sharing (Table 33). Under the ICLR 2027 reviewer FAQ a comparison with a paper available only as a preprint is not required and its absence cannot be a basis for rejection; the paper makes the comparison regardless.
[3] Blayney et al. (2026) is a mechanistic analysis of hidden-state fixed points across loop iterations. It does not study the KV cache (the term does not occur in the paper), proposes no compression, and reports that of the four models it tests Ouro, this paper's main backbone, is the one that does not reach a fixed point. The submission cites it for exactly what it shows, that per-loop computation settles onto low-dimensional trajectories (Sec. 2), and starts where it stops: that the per-loop K and V of a token form a converging trajectory whose stacked spectrum is low-rank, that this is specific to the recurrence axis (matched-budget codecs on the head and layer axes lose more accuracy, Table 7), that K and V converge at different rates (hence r_v > r_k), and that a frozen-backbone codec exploiting it turns into serving capacity on real hardware with the prefill included.
The three facts the paper contributes are therefore in none of [1]–[3]: the low-rank loop-axis structure with its controls; the frozen-backbone codec that reaches 4–21× at 2.3 B200-hours, with 0.782 ± 0.007 GSM8K over three conversion seeds against 0.419 ± 0.183 for the per-loop MLA head codec at the same budget; and the measured conversion of the reduction into serving capacity. [1] is the paper's testbed and baseline, [2] is a concurrent method the paper already compares against, and [3] is an analysis that stops before the cache. We ask that the novelty assessment be weighed against what these three works contain, and we thank the reviewer and the area chair for their time.
We thank the reviewer for the clear summary and for the questions, which led to new measurements.
Prefill slower; "can the two-pass prefill be improved, and when do decoding gains outweigh its extra cost?" Yes, and the answer is now measured (items 1–3 of our top-level response). Encoding the prompt into the latent store now costs the teacher's own prefill: 0.16 / 0.81 / 6.23 / 20.4 s at 4k / 16k / 64k / 128k for the 4× codec against the teacher's 0.15 / 0.76 / 6.01 / 20.2 s, where the submitted path took 1.43 / 9.62 / 112.4 / 423.4 s. The second pass is only needed for the exact first-token logits of the streaming decoder and costs one more prefill; the encode-only variant is the teacher's prefill alone. On when decoding gains outweigh the cost, the end-to-end table gives the crossover directly: at batch 1 the teacher is faster at every prompt length (2.4× per decode step under the Python-driven protocol, 1.2× once both decode steps are replayed from CUDA graphs, 17.7 vs 21.1 ms at 4k, new appendix table); at batch 32 the 4× codec is faster end to end from 4k prompts with ≥ 512 generated tokens (89.5 vs 99.6 s at 512 tokens, 342.7 vs 443.6 s at 2,048), and wherever the teacher's cache does not fit the batch (16k on Ouro-1.4B, 4k on Ouro-2.6B-Thinking) the codec is the only configuration that serves it.
Generation quality drops at high compression. True, and we keep it visible: the served 14.2× codec scores 0.386 on GSM8K against the teacher's 0.794 and the served 4× codec's 0.674 on Ouro-1.4B, and the paper's framing is a frontier (Fig. 1), not a free lunch. What the revision adds is where the trade pays and how stable the operating point is: the 4× point is 0.782 ± 0.007 over three conversion seeds (GSM8K-400) against 0.419 ± 0.183 for the per-loop MLA head codec at the same budget; on Ouro-2.6B-Thinking the served 4× codec reaches 0.844, above its own oracle number; and the end-to-end and concurrency tables show the throughput side of the frontier.
Contribution incremental. We would put the contribution as three facts about weight-tied looped models that had not been measured. (i) The per-loop K/V of a token is a converging trajectory whose stacked spectrum is low-rank, and this is specific to the recurrence axis: matched-budget codecs on the head and layer axes lose more accuracy, and the cardinality-matched controls added in the revision (random and single-loop encoders at the same effective rank and cache budget) fail where the trajectory codec succeeds. (ii) A frozen-backbone codec exploiting it reaches 4–21× KV reduction with 2.3 B200-hours of conversion and 101M codec parameters, against 806M for the per-loop MLA head codec at the same 4× budget and 0.782 ± 0.007 vs 0.419 ± 0.183 GSM8K over three seeds. (iii) The reduction turns into serving capacity on real hardware, now with the prefill included (top-level items 1–4). On cross-loop cache sharing specifically, the revision measures the strongest training-free form of it, Huginn's own step sharing, on the same checkpoint and recurrence: it is lossless to its floor of 16.5×, and composed with a per-slot codec it holds that accuracy to 26.4× end to end (reply to asQG), so sharing and the codec are complementary rather than the same idea. Training-free loop reuse on Ouro is also reported as the honest baseline: decode-only last-step reuse gives 0.690 and loop-averaged reuse 0.752 on 1.4B GSM8K, above our served 4× codec (0.674) at 4×, while the codec is the only option beyond 4× and is ahead at 2.6B (0.844 vs 0.822).
MELT not compared experimentally. A faithful head-to-head is not possible yet: MELT's code and checkpoints are not released (the paper says "to be released soon"; nothing was public as of 10 November), and reproducing it means a full-model fine-tune of Ouro (1,000 steps, 320M tokens, about 720 H100-hours in their report) against our frozen-backbone conversion of 2.3 B200-hours. What can be compared is the published numbers under each paper's own protocol, and the revision adds that as a table with the caveat stated: MELT reaches 3.98× KV reduction at T = 4 (its single gated state per layer caps the ratio at the loop count), max batch 11 vs 3 for its teacher at 32k on an H100, time-to-first-token 0.089 vs 0.111 s, batch-1 decode 8.5 vs 13.5 tokens/s (slower than its teacher, as ours is at batch 1: 47 vs 56 tokens/s with both decode steps replayed from CUDA graphs), and MATH-500 93.4 vs 94.4 on the Thinking models. LLA covers 4–21× on the same base model with the backbone frozen, and the two are complementary: MELT distils the loop trajectory into one state, LLA keeps the trajectory and compresses it. We will run a head-to-head as soon as MELT's checkpoints are available.
We hope the measured prefill, the end-to-end crossover and the clarified contribution address the reviewer's concerns, and would be grateful if they were taken into account in the final assessment.