- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes LastKV, where all layers attend to the last-layer KV of completed context. Recurrence operates across chunks, preserving parallel processing within each chunk, while learned blending retains recent layer-local information. The architecture reduces persistent KV storage and is evaluated on language modeling, state tracking, novel-view synthesis, and video generation.
Critical strengths
- The design is very simple and achieve good performance. In addition, this work addresses an important memory bottleneck while retaining direct access to historical token positions.
- The experiment section also includes video generation.
- The implementation of Last Layer KV cache is easy to follow/
Critical weaknesses
- The title is too vague. It seems this work is comparable with Attention in the first glance, but actually far away from it. I do not think it is a proper way to hook the reader.
- In related work section, Feedback Transformer use deep-to-shallow feedback, CLA and YOCO use KV sharing. The combination of last-layer sharing, chunkwise recurrence, and blending is useful, but seems not a big deal. Table 1 compare with YOCO, but not include CLA and Feedback Transformer.
- Figure 5 demonstrates substantial cache savings, but reports inference storage rather than end-to-end inference latency. Reporting part of quality/peak memory/prefill latency/decoding latency for the same configurations would make the practical advantage more convincing.
Pls reply all weaknesses and questions point-by-point, I may reconsidering the voting based on the rebuttal.
Questions for authors
-
What is the benefit over Feedback Transformer and CLA under matched training and storage budgets? Provide more discussion.
-
Can the authors provide matched fixed-view Objaverse results and uncertainty estimates for the small long-context improvements?
-
What prefill/decode latency and peak memory are achieved by the same configurations used for the main quality results?
3 comments
W3 & Q3: Figure 5 demonstrates substantial cache savings, but reports inference storage rather than end-to-end inference latency. Reporting part of quality/peak memory/prefill latency/decoding latency for the same configurations would make the practical advantage more convincing. What prefill/decode latency and peak memory are achieved by the same configurations used for the main quality results?
A3: We present the prefill latency and decoding throughput as below. Below table reports single-stream performance (concurrency 1) of the Transformer (Llama-3.1-8B) and LastKV (the same 8B backbone). Both are served with vLLM 0.25 on one H100. Each request prefills an L-token prompt and then generates 4,096 tokens. We report the median of three runs. LastKV's prefill throughput is 1.6–2.9% lower than the Transformer's. Its decode throughput ranges from 2.0% lower to 0.4% higher. The prefill gap comes from processing the prompt chunk by chunk, since each chunk depends on the state left by the previous one. All tokens within a chunk are still processed in parallel, so the overhead stays small. Decode speed is essentially unchanged because, for every generated token, LastKV's 32 layers read the shared source and move the same amount of KV data as the Transformer. LastKV's advantage is therefore not per-request speed but how many long sessions fit in memory.
| Length | Prefill tok/s | TTFT (s) | Decode tok/s | E2E tok/s | ||||
|---|---|---|---|---|---|---|---|---|
| Transformer | LastKV | Transformer | LastKV | Transformer | LastKV | Transformer | LastKV | |
| 32K | 26,772 | 25,996 | 1.224 | 1.260 | 124.2 | 121.7 | 119.7 | 117.4 |
| 64K | 19,557 | 19,044 | 3.351 | 3.441 | 105.4 | 103.7 | 97.0 | 95.4 |
| 128K | 12,606 | 12,409 | 10.40 | 10.56 | 80.7 | 81.0 | 67.0 | 67.0 |
The advantage of LastKV appears when a server must keep many long sessions alive. LastKV stores 4 KiB of cache per token (one source shared by all layers), plus a fixed two-chunk local window per layer for each session, versus 128 KiB per token for the Transformer. Below table compares the two at fixed concurrency. We report steady-state output throughput: the tokens generated by all concurrent sessions over three follow-up turns (128 tokens per session per turn), divided by the wall-clock time of those turns. In every setting of the table, the concurrency exceeds the Transformer's resident capacity. Its evicted sessions re-prefill their whole context on every turn, so its throughput collapses to the prefill rate regardless of concurrency. Both systems use vLLM's default prefix cache. These throughput gains are specific to memory-oversubscribed serving scenarios
| Concurrency | Method | 32K | 64K | 128K | 256K |
|---|---|---|---|---|---|
| 16 | Transformer | 87 | 33 | 11.3 | 3.4 |
| 16 | LastKV | 448 | 259 | 141 | 74 |
| 32 | Transformer | 88 | 33 | — | — |
| 32 | LastKV | 496 | 275 | — | — |
| 64 | Transformer | 89 | 33 | — | — |
| 64 | LastKV | 523 | 283 | — | — |
Q2: Can the authors provide matched fixed-view Objaverse results?
A2: We applied the fixed-view evaluation to the LastKV and baseline checkpoints on the full GSO test set. We report the fixed-view performance with intervals based on full test samples. As the table shows, LastKV obtains 25.97 PSNR (95% CI [25.79, 26.14]), against 25.36 [25.14, 25.59] for Transformer, and the two intervals do not overlap.
| Method | Objaverse (GSO) PSNR ↑ |
|---|---|
| Transformer | 25.36 [25.14, 25.59] |
| DeltaNet | 23.94 [23.76, 24.12] |
| Gated DeltaNet | 24.32 [24.14, 24.50] |
| LaCT | 23.99 [23.77, 24.20] |
| LastKV | 25.97 [25.79, 26.14] |
Dear reviewer WJ5S,
We would like to provide a few additional clarification for the peak memory usage.
W3 & Q3: Figure 5 demonstrates substantial cache savings, but reports inference storage rather than end-to-end inference latency. Reporting part of quality/peak memory/prefill latency/decoding latency for the same configurations would make the practical advantage more convincing. What prefill/decode latency and peak memory are achieved by the same configurations used for the main quality results?
A3: We report the peak GPU memory usage for the same 8B backbone, with LastKV using a 4K chunk size, consistent with our main quality evaluation. We break down memory consumption into model weights, runtime workspace, and KV/state storage. As shown below, LastKV reduces total peak memory from 20.7/24.7/32.7/48.7 GiB to 17.8/17.9/18.2/18.7 GiB at 32K/64K/128K/256K context lengths, corresponding to reductions of 14%/28%/44%/62%. Since model weights and runtime workspace remain identical, these savings come entirely from the reduced KV/state footprint. Notably, as context length increases from 32K to 256K, LastKV's total memory grows by only 0.9 GiB, compared with 28.0 GiB for the Transformer. This confirms that LastKV's cache savings translate directly into substantially lower end-to-end inference memory usage, particularly at long context lengths.
| Seqlen | Method | Model Weight | Buffer | KV Weight | Total |
|---|---|---|---|---|---|
| 32K | Transformer | 15.06 | 1.10 | 4.50 | 20.7 |
| 32K | LastKV | 15.06 | 1.10 | 1.64 | 17.8 |
| 64K | Transformer | 15.06 | 1.10 | 8.50 | 24.7 |
| 64K | LastKV | 15.06 | 1.10 | 1.77 | 17.9 |
| 128K | Transformer | 15.06 | 1.10 | 16.50 | 32.7 |
| 128K | LastKV | 15.06 | 1.10 | 2.02 | 18.2 |
| 256K | Transformer | 15.06 | 1.10 | 32.50 | 48.7 |
| 256K | LastKV | 15.06 | 1.10 | 2.52 | 18.7 |
We thank the reviewer for considering our paper simple and effective We really appreciate that, and we would like to make a few clarifications.
W1: The title is too vague. It seems this work is comparable with Attention in the first glance, but actually far away from it. I do not think it is a proper way to hook the reader.
A1: Thank you for this valuable suggestion. We agree that the current title may be somewhat ambiguous and could lead readers to misunderstand the scope and nature of our work. We will carefully reconsider the title and explore alternative names that can better reflect the core idea and contributions of our method, while avoiding potential confusion with existing attention-based approaches.
W2 & Q1: In related work section, Feedback Transformer use deep-to-shallow feedback, CLA and YOCO use KV sharing. The combination of last-layer sharing, chunkwise recurrence, and blending is useful, but seems not a big deal. Table 1 compare with YOCO, but not include CLA and Feedback Transformer. What is the benefit over Feedback Transformer and CLA under matched training and storage budgets? Provide more discussion.
A2: We further compare LastKV with Feedback Transformer [1] and CLA [2]. As Feedback Transformer uses per-token-recurrece, training it is slow and impractical for sequence length up to 4k or 32k. In order to provide a fair comparison, we compare Feedback Transformer with LastKV using the same chunk recurrent method and the same chunk size, while maintaining Feedback Transformer’s original native feedback mechanisms.
1. 124M, short-context pretrain, same setting as the draft’s table 1
2. 760M, short-context pretrain, same setting as the draft’s table 1
[1] Angela Fan, Thibaut Lavril, Edouard Grave, Armand Joulin, Sainbayar Sukhbaatar, “Addressing Some Limitations of Transformers with Feedback Memory”. [2] William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, Jonathan Ragan Kelly, “Reducing Transformer Key-Value Cache Size with Cross-Layer Attention”.