acceptodds
Under review as a conference paper at ICLR 2027

Looped Latent Attention: Cross-Loop Key-Value Compression For Looped Transformers

Abstract

Looped, weight-tied Transformers reduce parameters by reusing a single block, but decoding still stores a separate key/value (KV) cache for every recurrence step. We show this loop-indexed cache is highly structured: for a fixed token, layer and head, the per-loop K/V vectors trace a short low-rank trajectory across loops, while the head and layer axes stay much flatter. We introduce Looped Latent Attention (), a post-training cache codec that stores compact K and V latents and reconstructs loop-specific vectors only when attention reads them. The codec is initialized from the SVD of teacher activations and refined by logit and attention-output distillation; an -2D variant folds heads into one latent for the extreme-compression regime. At matched cache budget, per-head beats head-axis MLA, cross-layer sharing, KV quantization and zero-parameter final-loop reuse; the last result shows that an unadapted endpoint cannot replace the loop trajectory. The advantage holds on Ouro-2.6B-Thinking and transfers to Huginn-3.5B, where an SVD codec is near-lossless to on decoder-independent fidelity. The same full-RoPE codecs that produce the accuracy results also serve: with a fused latent-read kernel, decodes end to end on one B200 at the teacher's fixed-batch throughput and – its peak throughput while caching – less, with decoder-independent quality unchanged. Serving through the streaming latent store exposes a train/serve mismatch, which we trace to drift of the stored K/V of generated tokens and remove by fitting the codec under the served semantics with an exact 64-token window: served GSM8K rises from to at on Ouro-1.4B (oracle re-forward ) and from to on Ouro-2.6B, matching the oracle, with the same fused kernel. An absorbed variant, LLA-D, reads latents without reconstruction and reaches the teacher's peak throughput. On student-generated prefixes, on-policy refinement raises MATH-500 at from 0.43 to 0.64–0.66 across two runs and reduces no-answer generations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
looped transformers, latent attention, KV cache reduction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AbkYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

LLA compresses the KV cache across loops in weight-tied Transformers. It stores compact latents and reconstructs loop-specific keys and values when needed. The paper also studies how cache errors affect real generation and uses further training to improve stability.

Critical strengths
  1. The paper identifies cross-loop KV redundancy and shows that it can be effectively compressed.
  2. The method reduces cache memory while keeping good accuracy at moderate compression, especially on Ouro-2.6B.
  3. The analysis of cache drift during real generation is valuable. The proposed training changes help close the gap with oracle evaluation.
Critical weaknesses
  1. The contribution is somewhat incremental, as low-rank KV compression and cross-loop cache sharing have been explored before.
  2. The paper discusses MELT in the related work but does not compare with it experimentally.
  3. Prefill is much slower in the current implementation, and generation quality still drops at high compression ratios.
Questions for authors
  1. Please directly compare LLA with MELT in accuracy, cache memory, and inference speed under comparable training and evaluation settings.
  2. Can the two-pass prefill be improved, and when do decoding gains outweigh its extra cost?
4 comments

We thank the reviewer for the clear summary and for the questions, which led to new measurements.

Prefill slower; "can the two-pass prefill be improved, and when do decoding gains outweigh its extra cost?" Yes, and the answer is now measured (items 1–3 of our top-level response). Encoding the prompt into the latent store now costs the teacher's own prefill: 0.16 / 0.81 / 6.23 / 20.4 s at 4k / 16k / 64k / 128k for the 4× codec against the teacher's 0.15 / 0.76 / 6.01 / 20.2 s, where the submitted path took 1.43 / 9.62 / 112.4 / 423.4 s. The second pass is only needed for the exact first-token logits of the streaming decoder and costs one more prefill; the encode-only variant is the teacher's prefill alone. On when decoding gains outweigh the cost, the end-to-end table gives the crossover directly: at batch 1 the teacher is faster at every prompt length (2.4× per decode step under the Python-driven protocol, 1.2× once both decode steps are replayed from CUDA graphs, 17.7 vs 21.1 ms at 4k, new appendix table); at batch 32 the 4× codec is faster end to end from 4k prompts with ≥ 512 generated tokens (89.5 vs 99.6 s at 512 tokens, 342.7 vs 443.6 s at 2,048), and wherever the teacher's cache does not fit the batch (16k on Ouro-1.4B, 4k on Ouro-2.6B-Thinking) the codec is the only configuration that serves it.

Generation quality drops at high compression. True, and we keep it visible: the served 14.2× codec scores 0.386 on GSM8K against the teacher's 0.794 and the served 4× codec's 0.674 on Ouro-1.4B, and the paper's framing is a frontier (Fig. 1), not a free lunch. What the revision adds is where the trade pays and how stable the operating point is: the 4× point is 0.782 ± 0.007 over three conversion seeds (GSM8K-400) against 0.419 ± 0.183 for the per-loop MLA head codec at the same budget; on Ouro-2.6B-Thinking the served 4× codec reaches 0.844, above its own oracle number; and the end-to-end and concurrency tables show the throughput side of the frontier.

Contribution incremental. We would put the contribution as three facts about weight-tied looped models that had not been measured. (i) The per-loop K/V of a token is a converging trajectory whose stacked spectrum is low-rank, and this is specific to the recurrence axis: matched-budget codecs on the head and layer axes lose more accuracy, and the cardinality-matched controls added in the revision (random and single-loop encoders at the same effective rank and cache budget) fail where the trajectory codec succeeds. (ii) A frozen-backbone codec exploiting it reaches 4–21× KV reduction with 2.3 B200-hours of conversion and 101M codec parameters, against 806M for the per-loop MLA head codec at the same 4× budget and 0.782 ± 0.007 vs 0.419 ± 0.183 GSM8K over three seeds. (iii) The reduction turns into serving capacity on real hardware, now with the prefill included (top-level items 1–4). On cross-loop cache sharing specifically, the revision measures the strongest training-free form of it, Huginn's own step sharing, on the same checkpoint and recurrence: it is lossless to its floor of 16.5×, and composed with a per-slot codec it holds that accuracy to 26.4× end to end (reply to asQG), so sharing and the codec are complementary rather than the same idea. Training-free loop reuse on Ouro is also reported as the honest baseline: decode-only last-step reuse gives 0.690 and loop-averaged reuse 0.752 on 1.4B GSM8K, above our served 4× codec (0.674) at 4×, while the codec is the only option beyond 4× and is ahead at 2.6B (0.844 vs 0.822).

MELT not compared experimentally. A faithful head-to-head is not possible yet: MELT's code and checkpoints are not released (the paper says "to be released soon"; nothing was public as of 10 November), and reproducing it means a full-model fine-tune of Ouro (1,000 steps, 320M tokens, about 720 H100-hours in their report) against our frozen-backbone conversion of 2.3 B200-hours. What can be compared is the published numbers under each paper's own protocol, and the revision adds that as a table with the caveat stated: MELT reaches 3.98× KV reduction at T = 4 (its single gated state per layer caps the ratio at the loop count), max batch 11 vs 3 for its teacher at 32k on an H100, time-to-first-token 0.089 vs 0.111 s, batch-1 decode 8.5 vs 13.5 tokens/s (slower than its teacher, as ours is at batch 1: 47 vs 56 tokens/s with both decode steps replayed from CUDA graphs), and MATH-500 93.4 vs 94.4 on the Thinking models. LLA covers 4–21× on the same base model with the backbone frozen, and the two are complementary: MELT distils the loop trajectory into one state, LLA keeps the trajectory and compresses it. We will run a head-to-head as soon as MELT's checkpoints are available.

We hope the measured prefill, the end-to-end crossover and the clarified contribution address the reviewer's concerns, and would be grateful if they were taken into account in the final assessment.

We would be grateful to learn whether our reply and the revised PDF address the reviewer's concerns. In brief: the prompt is now encoded at the teacher's own prefill time (Table 13 of the revision), the end-to-end tables give the crossover (Tables 16–18 and 22), MELT is placed side by side under each paper's published protocol (Table 30), and the contribution rests on the cardinality-matched controls (Table 7) and the Huginn sharing comparison (Table 33). The revision also scores served GSM8K under exactly the configuration the serving tables time (Table 24): the second prefill pass is an option rather than a requirement, and the accuracy tables hold for the decoder the serving tables time. We are happy to answer anything that remains before the discussion closes.

Thank you for the detailed responses and additional experiments. My questions have been addressed. However, given the prior work, I still consider LLA’s novelty limited. I therefore maintain my rating of 2.

[1] Geiping et al., 2025. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach.

[2] Vendrell et al., 2026. Memory-Efficient Looped Transformer.

[3] Blayney et al., 2026. A Mechanistic Analysis of Looped Reasoning Language Models.

We thank the reviewer for confirming that the questions are addressed, and we write this for the area chair as much as for the reviewer, since the one remaining ground is novelty against [1]–[3]. All three are cited and discussed in the submission (Sec. 2; Tables 1, 30 and 33), and none of them contains the paper's mechanism or its finding.

[1] Geiping et al. (2025) is the Huginn architecture, which the paper uses as its second backbone. Its cache-memory contribution is zero-shot KV-cache sharing (its Sec. 6.2): a fixed slot budget per token that later recurrence steps overwrite, with no learned component. The paper measures that exact mechanism on Huginn (Table 33): it is lossless to its floor of 16.5×, beyond which only the codec continues (26.4× composed with it), and on Ouro training-free loop reuse applied to the prompt scores 0.000 GSM8K at 4× where the codec scores 0.800 (Table 1). LLA is not sharing: it is a learned low-rank codec of the per-loop K/V trajectory, and the cardinality-matched controls (Table 7: random and single-loop encoders at the same effective rank and cache budget) fail where the trajectory codec succeeds, which sharing-style reasoning does not predict.

[2] Vendrell et al. (2026), MELT, appeared on arXiv on 8 May 2026 and has since been an ICML 2026 workshop poster; its code and checkpoints are unreleased. The submission names it the closest concurrent work (Sec. 2) and places the published numbers side by side (Table 30). The two compress different objects: MELT fine-tunes the whole looped model (1,000 steps, 320M tokens, about 720 H100-hours) to distil the loop trajectory into a single gated state per layer, which caps the ratio at the loop count (3.98× at T = 4); LLA keeps the backbone frozen (2.3 B200-hours per codec, 101M codec parameters), keeps the trajectory, sets the ratio by (r_k, r_v) and reaches 4–21× on the same base model, composing with token eviction (Tables 28–29) and with sharing (Table 33). Under the ICLR 2027 reviewer FAQ a comparison with a paper available only as a preprint is not required and its absence cannot be a basis for rejection; the paper makes the comparison regardless.

[3] Blayney et al. (2026) is a mechanistic analysis of hidden-state fixed points across loop iterations. It does not study the KV cache (the term does not occur in the paper), proposes no compression, and reports that of the four models it tests Ouro, this paper's main backbone, is the one that does not reach a fixed point. The submission cites it for exactly what it shows, that per-loop computation settles onto low-dimensional trajectories (Sec. 2), and starts where it stops: that the per-loop K and V of a token form a converging trajectory whose stacked spectrum is low-rank, that this is specific to the recurrence axis (matched-budget codecs on the head and layer axes lose more accuracy, Table 7), that K and V converge at different rates (hence r_v > r_k), and that a frozen-backbone codec exploiting it turns into serving capacity on real hardware with the prefill included.

The three facts the paper contributes are therefore in none of [1]–[3]: the low-rank loop-axis structure with its controls; the frozen-backbone codec that reaches 4–21× at 2.3 B200-hours, with 0.782 ± 0.007 GSM8K over three conversion seeds against 0.419 ± 0.183 for the per-loop MLA head codec at the same budget; and the measured conversion of the reduction into serving capacity. [1] is the paper's testbed and baseline, [2] is a concurrent method the paper already compares against, and [3] is an analysis that stops before the cache. We ask that the novelty assessment be weighed against what these three works contain, and we thank the reviewer and the area chair for their time.

Reviewer_gujvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Looped transformers use weight-tied transformers, which reduce the number of parameters but it does not reduce the compute flops or the KV cache size. This paper address the latter by proposing a latent looped attention scheme, which compresses the K and V and therefore reducing the KV cache size. The paper identifies and utilizes an interesting observation that the recurrence axis is the most compressible axis. The paper also proposes initialization with SVD teacher and futher OPD to refine and improve accuracy. They show results on 1.4 - 2.6B models and showcase improved GMS8K/ Math500 results with OPD that can match the non-LLA oracle models.

Critical strengths

The paper identifies a key bottleneck in Looped transformer of increasing KV cache size, while the weights are reduced by looped attention. They propose a novel solution of using LLA by identifying that the recurrence axis is compressible. The experiments are well structured with respect to matching cache comparisons, but the decode/latency results along with implementation for matched kernels for baseline and LLA inference are not clearly differentiated.

Critical weaknesses

The key barrier for practical use of this approach is the hidden cost of prefill and how that will impact overall throughput / latency. The main claims are around decode speed up, and for matched accuracy needs the orcale-trained version to generate the KV, which will add to latency. This is not clearly explained for real serving usecases. The results for decode are also presented for a narrow 32 Batch size usecase, how well does it scale at different concurrencies? Does the kernel implementation also play a part in separation of speed-up claim v/s use of LLA compared to teacher? For long context results, additional RULER benchmarks are needed, the results show decode perf. upto 64/128K but the benchmarks are on smaller context lens.

Questions for authors

Can one support native latent cache generation with post training an existing looped oracle? Current approach of using Looped Oracle teacher suggests viability of inference, but this will be a limiter in real world usecases. As noted in appendix, the prefill latency is a bottleneck that makes this prohibitive for real usecases. How well does this approach scale to longer context lengths, can we add ruler benchmarks? The decode results are presented upto 128K, but accuracy claims are not provided at longer context lengths. Does LLA perform better than traditional approaches for KV cache compression approaches like SnapKV/PyramidKV for looped transformers?

5 comments
AuthorsOpenReview ↗

(1/3) - Reply to Reviewer gujv

We thank the reviewer for a detailed review whose points map directly onto measurements we have now made; we address them in the reviewer's order.

Hidden prefill cost; "the oracle-trained version is needed to generate the KV, which adds latency". Two separate points. (a) Prefill: items 1–3 of our top-level response. The cost in Appendix Table 13 was our research decoder (chunked, unfused, two passes), not the method; the prompt is now encoded inside one ordinary teacher forward at the teacher's own prefill time (4×: +1–7%; 14.2× and 21.3×: at or below the teacher), the store it builds agrees with the submitted one to kernel noise, and the end-to-end table gives request latency across prompt and output lengths. (b) Nothing oracle-related runs at serving time. The served decoder (Table 2 "served" rows, Sec. 7) is the frozen looped model plus the codec; the "oracle" rows are an evaluation protocol (teacher-forced re-forward) that isolates codec fidelity from streaming error, not a component of the system. The 2.6B served GSM8K (0.844) is above its oracle number (0.834). We have rewritten the paragraph in Sec. 7 that allowed the other reading.

Decode results only at batch 32 — scaling across concurrencies. The Table 2 runs recorded the whole batch sweep (1 … 1,024, 16 decode steps per point, same protocol); we had printed only the batch-32 column and the maximum. Aggregate decode throughput (tokens/s) at 4k context on one B200:

batchteacherLLA 4×LLA 14.2×LLA 21.3×
115777
8120525552
16161102110103
32180209223206
48188303336307
64OOM341414412
128OOM438543698
256OOMOOM648879
1024OOMOOMOOM1102

At 16k context the teacher's last feasible batch is 8 (51 tokens/s); LLA reaches batch 48 / 192 / 256 at 4× / 14.2× / 21.3× (122 / 187 / 279 tokens/s). Below batch 16 LLA is slower than the teacher at every context length; the crossover is batch 16–32 at 4k and batch 8 at 16k. The full sweep is now an appendix figure, and Sec. 7 states the crossover instead of the single column.

(continued in the next comment)

AuthorsOpenReview ↗

(2/3) - Reply to Reviewer gujv

Does the kernel, rather than LLA, account for the speed-up? We ran two controls in the same job and on the same GPU: the teacher's own uncompressed K/V served through our fused kernel (identity up-projection, exact window covering the whole context), and LLA through an unfused PyTorch read. Decode step time in ms (4× codec, W = 64):

contextbatchteacher, SDPA (Table 2)teacher's K/V through our fused kernelLLA, fused kernelLLA, unfused PyTorch
4k165122155258
4k32180126162401
16k169129156444
16k8159121160474
16k32OOMOOM3021197

The answer is no, in two parts. (i) The fused kernel is what makes a latent read usable at all: the unfused read is 1.7× (4k, batch 1) to 4× (16k) slower. (ii) The kernel is not what separates LLA from the teacher. The teacher's own K/V through the same kernel takes 126 ms at 4k / batch 32, faster than both the SDPA baseline (180 ms) and the latent read (162 ms), and 121 ms at 16k / batch 8 (SDPA 159, latent read 160). Where both caches fit, the per-step time is therefore set by the kernel, and the 35–40 ms the latent read adds over an uncompressed read through the same kernel is the codec's own cost (up-projection and score). What the codec buys is capacity: the teacher's cache does not fit batch 64 at 4k or batch 16 at 16k under either kernel, while the 4× store decodes batch 32 at 16k in 302 ms (106 tokens/s) and reaches batch 128 at 4k. The throughput advantage of Table 2 and of the sweep is the batch the smaller store admits, not kernel speed. The same control explains the batch-1 gap of Sec. 7: both steps are launch-bound at batch 1 (122 ms even for the teacher's K/V through our kernel, against 65 ms with SDPA).

With the host taken out of both decoders. The protocol of Table 2 drives both decoders from Python through the model's forward, so at small batch both steps are bound by the driver. To remove it from the comparison we captured one decode step of each decoder in a CUDA graph and replayed it: the teacher with the flash attention kernel of its eager path reading its cache in place (transformers' eager SDPA path also copies the K and V cache views at every attention call, which inflates its eager step at large batch), the codec with its store length, slot indices and the merge of the kernel's partials on the device. Greedy continuations through the captured steps reproduce the eager ones token for token for all three decoders (96 tokens; the teacher's with zero logit difference). Decode step in ms, eager → graphed (4× codec W = 64, 14.2× codec W = 128; tokens/s in parentheses where the batch matters):

contextbatchteacherLLA 4×LLA 14.2×
4k178 → 17.7190 → 21.1162 → 19.8
4k881 → 24.9194 → 41.0166 → 33.3
4k32186 → 60.8 (526/s)196 → 81.7 (392/s)167 → 65.3 (490/s)
4k128OOM307 → 243.7 (525/s)257 → 191.7 (668/s)
16k182 → 20.3197 → 26.4160 → 23.4
16k8164 → 40.5 (198/s)196 → 80.4 (100/s)169 → 62.8 (127/s)
16k32OOM310 → 238.6 (134/s)248 → 183.2 (175/s)
16k128OOMOOM730 → 659.1 (194/s)

(The eager step is host-bound and its time depends on the host: on this node the teacher's batch-1 step is 78 ms against the 65 ms of the kernel control above, taken on another node; the graphed columns are GPU-bound, and a second run on the same node, otherwise idle, reproduces every graphed cell within 3.3% and every eager cell within 7.9%.) With the driver removed the batch-1 gap is 1.2–1.3× instead of 2.4×, and per step the teacher's read is the shorter one at every batch its cache fits: the 4× latent moves the same bytes per token as the teacher's K/V (r_k + r_v = 256 = 2 d_h) through a kernel that is slower per byte than flash attention, and at 14.2× the kernel is bound by its per-position work rather than by bytes. What the compression buys is capacity. At 4k the teacher's last batch is 32, where it decodes at 526 tokens/s, while the 4× store serves batch 128 at the same aggregate rate (525 tokens/s) and the 14.2× store at 668 tokens/s; at 16k the teacher's last batch is 8 (198 tokens/s) while the 4× store serves batch 32 (134 tokens/s) and the 14.2× store batch 128 (194 tokens/s). The 2.6× and 5.0× peak-throughput ratios of Sec. 7 are what the Python-driven protocol gives, where the teacher's step also carries the cache copies; the revision states both protocols side by side and attributes the gain to the batch the store admits. Both controls are appendix tables and Sec. 7 now reads accordingly.

(continued in the next comment)

AuthorsOpenReview ↗

(3/3) - Reply to Reviewer gujv

Accuracy at long context: RULER. Agreed that passkey retrieval alone is weak evidence, and the prefill fix made RULER tractable. We ran the lm-eval RULER tasks niah_single_1, niah_single_2, niah_multikey_1, niah_multivalue, variable tracking (vt) and SQuAD QA at 4k, 8k, 16k and 32k (100 items per task and length, greedy, Ouro-1.4B's trained context is 64k) for the teacher and the two served codecs, 4× (W = 64) and 14.2× (W = 128). Accuracy at 4k / 8k / 16k / 32k:

taskteacherLLA 4× (W = 64)LLA 14.2× (W = 128)
niah_single_11.00 / 1.00 / 1.00 / 1.001.00 / 1.00 / 1.00 / 0.990.97 / 0.86 / 0.84 / 0.76
niah_single_21.00 / 1.00 / 1.00 / 0.991.00 / 1.00 / 1.00 / 0.940.92 / 0.72 / 0.57 / 0.53
niah_multikey_10.98 / 0.98 / 0.94 / 0.941.00 / 0.90 / 0.91 / 0.640.78 / 0.53 / 0.24 / 0.13
niah_multivalue1.00 / 0.99 / 0.99 / 0.900.89 / 0.80 / 0.41 / 0.150.53 / 0.32 / 0.16 / 0.04
vt1.00 / 1.00 / 0.70 / 0.360.88 / 0.76 / 0.76 / 0.540.39 / 0.33 / 0.15 / 0.11
qa_squad (128-token budget)0.53 / 0.37 / 0.34 / 0.190.63 / 0.20 / 0.30 / 0.160.55 / 0.30 / 0.19 / 0.06

Reading: at 4× the served codec keeps single-needle retrieval to 32k, multikey retrieval to 16k (0.91 vs 0.94) with a drop at 32k (0.64 vs 0.94), and tracks variables at least as well as the teacher from 16k on; the task it loses is multivalue retrieval, four needles per query, from 16k. At 14.2× every task degrades with length, consistent with that codec's GSM8K; it is the capacity point of the frontier, not the accuracy point. Two protocol notes so the cells are read at the right resolution. (i) 100 items per cell; duplicate runs of three tasks per arm agree within 0.05 on the needle and tracking tasks and within 0.08 on QA, so smaller differences are noise. (ii) Ouro-1.4B opens most of its QA answers with a reasoning preamble that RULER's default 32-token budget truncates (73% of the teacher's QA answers at 4k scored zero for that reason), so every arm is scored with a 128-token budget; about a quarter of the teacher's answers at 4k–16k are still cut off inside the preamble even then, so its QA row is a lower bound. The 4× codec, for its part, returns an empty completion on 10–43% of QA items at 4k–16k, which the teacher and the 14.2× codec never do: on that prompt format the served decoder declines rather than retrieves wrongly, and we report the row as measured and say so in the caption. The table and these notes are in the appendix (Table "RULER under the streaming decoder"), next to the passkey figure of the submission, whose mechanism they confirm: the window-aware codec is exact inside its window and compressed outside it, and multi-needle queries are the first to feel the compression at long range.

SnapKV / PyramidKV. These are token-eviction methods: they shrink the number of cached tokens, while LLA shrinks the bytes per token, so they are orthogonal and compose. The submission already composes LLA with a StreamingLLM-style sink-plus-window eviction (appendix eviction table): under an aggressive 260-token budget LLA 4× stays within one point of the 8-bit K/V control, i.e. the two mechanisms do not interact. The revision says so in the related work and we will keep the eviction table.

Native latent-cache generation by post-training the looped model. We tried it: LoRA adapters on q/o (r = 16) and on q/k/v/o (r = 32) trained jointly with the codec under the served-semantics objective. r = 16 tracked the codec-only run to three digits and r = 32 was worse at every step, so a frozen backbone with the codec fine-tuned under served semantics (Sec. 7, Table 4) was the post-training that worked, and the revision reports the negative result in the appendix. We agree that a model trained from the start with a latent cache is the natural next step; the paper's contribution is to show which axis to compress and to measure the compressed cache on the decoder that serves it.

We hope the prefill fix, the sweep, the kernel controls and the RULER results address the reviewer's concerns, and would be grateful if they were weighed in the final assessment.

Reviewer_asQGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Looped Latent Attention (LLA), which compresses KV states across loops into shared latent representations and reconstructs loop-specific states when needed. The codec is initialized with SVD and trained through distillation while keeping the backbone frozen. Experiments evaluate compression fidelity, task accuracy, and serving efficiency. The paper also investigates streaming generation errors and introduces adaptations to mitigate them.

Critical strengths
  1. The cross-loop analysis and matched-budget comparisons provide useful evidence that recurrence is an effective axis for KV-cache compression. The substantial cache reductions enable larger batch capacity and higher decode throughput. The paper also explicitly investigates and mitigates the gap between reconstruction fidelity and streaming generation quality, strengthening its practical evaluation.
Critical weaknesses
  1. The substantial prefill overhead limits the practicality of the current implementation for latency-sensitive long-context inference. Table 13 reports 109 seconds for LLA 4× versus 6.1 seconds for the teacher at 64k context, and 408.3 versus 20.2 seconds at 128k. These costs substantially increase response startup time, and the reported batched decode throughput gains do not establish an end-to-end latency benefit.
Questions for authors
  1. Could you compare LLA with Huginn’s existing training-free KV-cache sharing [1], using the same checkpoint and recurrence count at comparable KV-cache memory budgets?

  2. For the final window-aware configurations, how does total request latency, including prefill and decoding, compare with the teacher across representative prompt and output lengths under matched serving conditions?

[1] Geiping et al. “Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach.” NeurIPS 2025.

4 comments

We thank the reviewer for a precise review; both questions led to measurements that are now in the paper.

Critical weakness and Q2 — prefill overhead and total request latency under matched serving. The reviewer is right that Table 13 as submitted did not establish an end-to-end benefit. Items 1–3 of our top-level response give the fix and the measurement: the prompt is now encoded inside one ordinary teacher forward, at the teacher's own prefill time (4× codec: 0.16 / 0.81 / 6.23 / 20.4 s at 4k / 16k / 64k / 128k against the teacher's 0.15 / 0.76 / 6.01 / 20.2 s; the submitted path was 1.43 / 9.62 / 112.4 / 423.4 s), and the store it builds agrees with the submitted one to attention-kernel noise, so every accuracy number in the paper stands (served GSM8K on the full test set is 0.695 with the new prefill against 0.672 for the submitted path, paired McNemar p = 0.026 in the new path's favour).

For the final window-aware configurations the end-to-end table (item 3) answers the question as asked, across prompt lengths 1k / 4k / 16k, output lengths 128 / 512 / 2,048 and batch 1 / 32 on one B200. At batch 1 the teacher is faster at every prompt length under the Python-driven protocol of that table: its SDPA decode step is 65 ms against 155 ms for the fused latent read, and both steps are launch-bound there (the teacher's own K/V through our kernel takes 122 ms). We therefore also measured the decode step with the host removed from both decoders: one step of each is captured in a CUDA graph and replayed (the teacher with the flash attention kernel of its eager path reading its cache in place, the codec with its store length, slot indices and merge on the device; greedy continuations through the captured steps reproduce the eager ones token for token). The batch-1 step at 4k is then 17.7 ms for the teacher against 21.1 ms for the served 4× codec (19.8 ms at 14.2×), and a 4k-prompt request with 512 generated tokens takes 9.1 s against 11.1 s (eager decode alone: 33.8 s against 83.8 s). The remaining batch-1 difference is the codec's extra kernels per step (10,678 launches against the teacher's 5,743), not the store read; both measurements are new appendix tables. At batch 32 the ordering is set by the context: the teacher wins with 1k prompts; at 4k LLA 4× is faster end to end once the output is ≥ 512 tokens (89.5 vs 99.6 s at 512 tokens, 342.7 vs 443.6 s at 2,048); at 16k the teacher does not fit batch 32 while the codecs serve it at 47–114 generated tokens/s, and on Ouro-2.6B-Thinking the same happens already at 4k. Section 7 and the new appendix section "Prefill cost, store equivalence and end-to-end requests" now state this, and the appendix also gives the encode-only variant (prefill term halved) for deployments that do not need the exact first-token logits.

(continued in the next comment)

Q1 — Huginn's training-free KV-cache sharing, same checkpoint and recurrence, comparable budgets. We agree this is the right training-free control for the Huginn results and we have measured it. The released model exposes the scheme as a generation option (cache_lookup_strategy="compress-s<N>"): the 4R (step, layer) cache slots of the four-layer recurrent block are mapped to N shared slots, so the 4 + 4R = 132 slots of the full cache become 4 + N, a budget of (4+4R)/(4+N); N = 4, one slot per layer shared by all 32 steps, is the floor of the scheme, since a smaller N would share one slot between different layers. Geiping et al. report it only on MT-Bench. We ran it on GSM8K (5-shot, first 500 test problems, greedy, R = 32, batch 1, the model's own cached generation loop), and then composed it with a codec at the floor:

Huginn-3.5B, GSM8K strict / flexiblecache budget (end to end)accuracy
full cache1×0.266 / 0.266
compress-s166.6×0.298 / 0.300
compress-s811×0.268 / 0.268
compress-s4 (floor of the scheme)16.5×0.288 / 0.292
compress-s4 + rank-2640 latent per slot22× (64× on the recurrent block)0.292 / 0.292
compress-s4 + rank-1320 latent per slot26.4× (128×)0.282 / 0.284
compress-s4 + rank-660 latent per slot29.3× (256×)0.218 / 0.218

Three conclusions. (i) Huginn's own sharing is within noise of the full cache all the way to its floor (n = 500, SE ≈ 0.02), and we state this in the paper as the training-free reference for that model. (ii) The two mechanisms act on different factors of the cache: sharing removes the step axis, a codec shrinks the bytes of each slot, so their budgets multiply. The composed rows are a rank-r PCA latent per (layer, K | V) on the shared final-step K/V, fitted on 174k tokens of GSM8K train text and applied to past tokens while the current token attends to its exact K/V, i.e. the served semantics of the paper's decoder; it holds the full-cache accuracy to 26.4× end to end (128× on the recurrent block) and gives 0.218 at 29.3× (256×). The codec is what continues beyond the sharing floor, and the two are complementary rather than competing. (iii) The codec rows of the submission's Huginn table (SVD, oracle-decoded, n = 200) remain as fidelity numbers; the new table is measured in the model's own generation loop, and the revision says which is which. The new table and paragraph are in the appendix and Section 5 points to them.

We hope these measurements address both concerns and would be grateful if the reviewer would reconsider the score in their light.

Thank you for the additional experiments and the detailed response. The new prefill implementation substantially reduces the overhead reported in the submission, and the end-to-end measurements give a clearer picture of when the smaller cache is useful.

My concern about latency-sensitive inference remains, however. The two-pass configuration used for the accuracy results still takes roughly twice the teacher’s prefill time, and batch-1 requests remain slower even with CUDA graphs. The improvements at larger batches are useful, but I still consider the startup and request latency costs a significant practical limitation.

The Huginn experiments also help clarify the comparison. They show that the existing training-free sharing is a strong baseline and that adding a per-slot PCA codec allows further compression. This supports combining the two approaches, although it does not establish how standalone LLA compares with cache sharing at matched memory budgets.

I appreciate the progress made during the discussion, but the new evidence is not sufficient to change my overall assessment. I will maintain my rating of 2 (Weak rejection). I have no further experimental requests at this stage.

We thank the reviewer for reading the new material and for saying precisely what remains. We take the three points in turn so that the record is exact.

Startup cost. The second prefill pass is not needed to build the store. The encode-only prefill produces the same latents at the teacher's own prefill time (4× codec 0.16 / 0.81 / 6.23 / 20.4 s at 4k / 16k / 64k / 128k against the teacher's 0.15 / 0.76 / 6.01 / 20.2 s; 14.2× at or below the teacher). What the second pass changes is the distribution of the first generated token only: with it, that token is drawn from the model reading the reconstructed prompt K/V, as every later token is; without it, from the teacher's own logits of the same forward. We used the two-pass variant for the accuracy tables because every token of a served answer then reads the store, the stricter protocol for the codec, not because the store differs. We have now scored served GSM8K on the full test set under exactly the configuration whose cost the serving tables report, the encode-only prefill and the single-forward decode step, with the two-pass protocol re-run in the same job: the 4× codec scores 0.697 strict against 0.698 (ahead on 98 problems, behind on 99, McNemar p = 1.00), and the 14.2× codec 0.440 against 0.393 (178 / 116, p < 0.01). The revised appendix reports all four combinations of prefill passes and decode forwards under both GSM8K filters. The startup cost of the method is therefore the teacher's prefill; the appendix states what the optional second pass buys.

Batch-1 decode. Agreed, and the paper does not claim otherwise. At batch 1, with a context whose cache fits the device, the codec costs 1.2× per generated token (21.1 against 17.7 ms at 4k with both decode steps replayed from CUDA graphs) and offers nothing in return, because a smaller store buys capacity, not a shorter read. The claim is the capacity: at 4k the 4× store serves batch 128 at 525 tokens/s where the teacher's last batch is 32 at 526; at 16k the codecs serve batch 32 and 128 against the teacher's 8; on Ouro-2.6B-Thinking the teacher's cache fits neither batch 32 at 4k nor batch 1 at 128k on one B200. Section 7 states the batch-1 cost and both protocols in those terms. The one published method that compresses the loop axis, MELT, reports the same shape at batch 1 (8.5 against its teacher's 13.5 tokens/s at 3.98×; ours 47 against 56 with graphed steps). Whether a 1.2× batch-1 step outweighs a 4–14× smaller cache depends on the deployment; we ask only that the paper be read as claiming the regime it measures.

Sharing at matched memory. The standalone comparison is Table 1, on the model the paper is about. At the same 4× budget over all tokens, Ouro's training-free sharing (final-loop reuse) scores 0.000 GSM8K and 0.443 MC-average against LLA's 0.800 and 0.640; restricted to generated tokens it keeps GSM8K (0.708 / 0.752) but reduces the 5-shot cache by 1.12× and scores 0.20 / 0.30 on MATH-500 against 0.57 for the served 4× codec. On Huginn the reviewer is right that the picture differs, and the revision says so: sharing is lossless to its floor of 16.5×, and the codec at the same block ratio (32× on the recurrent block) is near-lossless in fidelity (held-out KL 0.010, top-1 0.954) with 0.200 against the teacher's 0.245 on the first 200 problems under the oracle decoder, within the resolution of n = 200. To the floor the two are within noise of the full cache and of each other; beyond it only the codec continues (0.282 at 26.4×, 0.218 at 29.3×). We do not claim standalone superiority on Huginn. The Huginn result is that the loop axis is low-rank in a second recurrent architecture and that the codec composes with the model's own sharing; the matched-memory contest between the two mechanisms is Table 1's, where sharing applied to the prompt collapses.

We are grateful for the reviewer's engagement and respect the assessment. We hope the measurements above and the revised PDF let the area chair weigh the remaining concern, which is about the regime in which a smaller cache is worth a slower batch-1 step rather than about the measurements themselves.

Comments on the paper

2 comments
AuthorsOpenReview ↗

(2/2) - Response to all reviewers

3. End-to-end request latency (prefill + decode, wall time of the whole batch in seconds, one B200, two-pass prefill; new appendix table). Teacher / LLA 4× (W = 64) / LLA 14.2× (W = 128):

promptgeneratedbatchteacherLLA 4×LLA 14.2×
1k512132.279.179.4
1k5123239.683.291.7
1k204832212.6328.4364.1
4k1283227.129.531.1
4k5123299.689.598.0
4k204832443.6342.7362.4
16k12832OOM86.577.2
16k51232OOM203.7174.5
16k204832OOM689.5577.0

At batch 1 the teacher is faster at every prompt length under this Python-driven protocol, where both decode steps are launch-bound. A new appendix table replays one decode step of each decoder from a CUDA graph, with the host out of both: the batch-1 step at 4k is then 21.1 ms for the 4× codec and 19.8 ms at 14.2× against the teacher's 17.7 ms (eager: 190 / 162 / 78 ms), and a 4k-prompt request with 512 generated tokens takes 11.1 s against the teacher's 9.1 s (eager decode alone: 83.8 s against 33.8 s), so the batch-1 gap of Table 13 and Sec. 7 is the driver, not the store (item 4). At batch 32 the ordering follows the context: with 1k prompts the teacher's cache fits and its shorter decode step wins; at 4k the 4× codec is faster end to end once the output amortises the second prefill pass (−10% at 512 generated tokens, −23% at 2,048); at 16k the teacher does not fit batch 32 at all and the codecs complete the batch at 47–114 generated tokens/s. On Ouro-2.6B-Thinking the teacher does not fit batch 32 at 4k either (LLA 4×: 59.4 s / 181.5 s for 128 / 512 generated tokens). With the encode-only prefill the LLA prefill term halves (4k, batch 32: 8.9 → 4.4 s; 16k: 48 → 24 s) and the short-output rows move with it (4k / 128 / batch 32: 25.8 s vs the teacher's 27.1 s).

4. Concurrency and kernel attribution (reply to gujv): the full batch sweep behind Table 2 is now an appendix figure, and two controls locate the gain. Serving the teacher's own uncompressed K/V through our fused kernel shows that the kernel is not what separates LLA from the teacher, and replaying one decode step of each decoder from a CUDA graph, with the Python driver out of both, shows the same from the other side: the batch-1 gap falls from 2.4× to 1.2×, per step the teacher's read is the shorter one wherever its cache fits, and the codecs' throughput advantage is the batch the smaller store admits (at 4k: batch 128 at 525 / 668 tokens/s for the 4× / 14.2× store against the teacher's last batch 32 at 526 tokens/s; at 16k: batch 32 / 128 against the teacher's 8). Sec. 7 now states both protocols.

5. Huginn's own training-free cache sharing (reply to asQG): measured on the same checkpoint and recurrence in the model's own generation loop at 6.6×, 11× and 16.5× (its floor), and composed with a per-slot codec, which holds the full-cache accuracy to 26.4× end to end (128× on the recurrent block).

6. RULER at 4k–32k (reply to gujv) for the teacher and both served codecs, six tasks, 100 items per task and length. At 4× the served codec keeps single-needle retrieval to 32k and multikey retrieval to 16k, tracks variables at least as well as the teacher from 16k on, and loses multivalue retrieval from 16k; at 14.2× every task degrades with length, as that codec's GSM8K leads one to expect.

7. Smaller additions: a MELT side-by-side of published numbers with the protocol caveat (reply to AbkY); the negative result of post-training the looped model itself for the latent cache (reply to gujv); SnapKV / PyramidKV placed as orthogonal token-eviction methods next to the existing eviction-composition table.

We believe the prefill result removes the shared critical weakness: the method's prompt cost is the teacher's prefill, and the end-to-end comparison now shows where the smaller store pays (4k prompts with ≥ 512 generated tokens at batch 32, and every setting where the teacher's cache does not fit the batch). We would be grateful if the reviewers would consider the revised evidence in their assessment.

AuthorsOpenReview ↗

(1/2) - Response to all reviewers

We thank the three reviewers for careful reviews that agree on the paper's strengths (the cross-loop analysis, the matched-budget comparisons, the gap between reconstruction fidelity and streaming generation and its mitigation) and converge on one weakness: the prefill cost of Appendix Table 13 and the absence of an end-to-end latency comparison. We took that weakness as the first priority of the discussion period. Below is a summary of what changed; the per-reviewer replies give the details. All new measurements use the same single-B200 protocol as Tables 2 and 13 of the submission, and the revised PDF is uploaded (the main text stays within 10 pages; the new material is in the appendix sections "Prefill cost, store equivalence and end-to-end requests" and "Additional figures and tables", with pointers from Secs. 5 and 7).

1. Prefill now costs the same as the teacher's own prefill. The Table 13 numbers (109 s vs 6.1 s at 64k; 408 s vs 20 s at 128k) came from our research decoder, which encoded the prompt 1,024 tokens at a time and re-read the whole past through an unfused PyTorch latent read for every chunk, twice. That is an implementation, not a property of the method. Because the encoder is linear in the loop slices, c = Σ_t (x_t − μ_t) W_down[t], the latent of every token can be accumulated loop by loop inside the frozen teacher's ordinary forward, with its native attention, and the T×D trajectory is never materialised. Measured prefill time in seconds (batch 1, one B200):

4k16k64k128k
teacher, cache-writing prefill0.150.766.0120.2
LLA 4× (submitted path, re-measured)1.439.62112.4423.4
LLA 4×, new prefill, encode only0.160.816.2320.4
LLA 4×, new prefill, two-pass0.321.6112.340.8
LLA 14.2× / 21.3×, encode only0.150.765.9820.0

Encoding the prompt is therefore the teacher's prefill plus 1–7% at 4× and at or below it at 14.2× and 21.3×. The two-pass variant, which also reproduces the streaming decoder's first-token logits exactly and is what our accuracy evaluations use, costs twice the teacher's prefill rather than 10–20×. On Ouro-2.6B-Thinking the encoder prefills 64k in 12.3 s against the teacher's 12.0 s, and at 128k the teacher's own cache no longer fits one B200 while the 4× encoder does (40.9 s). The appendix paragraph that attributed the old cost to holding the complete trajectory before down-projection is withdrawn; the transient memory is the teacher's activations plus one fp32 accumulator per latent (new appendix table).

2. The new prefill builds the same store. On GSM8K prompts the two prefills differ by at most 3.8% (K) and 4.0% (V) in relative Frobenius norm, which is exactly the gap between two standard attention kernels of the same model (SDPA vs eager: 3.6% / 3.8%); first-token KL is ≤ 2.4×10⁻³ (between the two teacher kernels 2.5×10⁻³), the argmax agrees on 8/8 prompts and greedy continuations are identical on 7/8. Served GSM8K on the full test set (n = 1,319) with the new prefill: 0.695 strict / 0.698 flexible, against 0.672 / 0.677 for the submitted path re-run in the same job (the submission reported 0.674). Paired over the 1,319 problems the new store is ahead on 107 and behind on 76 (McNemar exact p = 0.026). Served MATH-500 (n = 100, math-verify): 0.50 vs 0.52, within the ±0.05 resolution of n = 100.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.