acceptodds
Under review as a conference paper at ICLR 2027

Last Layer Key-Value Cache is All You Need

Abstract

The key-value (KV) cache of an auto-regressive transformer grows with both sequence length and network depth: every layer stores its own keys and values. Cross-layer KV cache sharing lets several layers read one layer's cache, but existing designs share only among a few layers, each reading a cache from a lower layer. We want the opposite: every layer reading the last layer's KV. Done at every token, this would break exact parallel training, so shares at chunk boundaries instead, as a chunk recurrence. Each chunk runs as a standard per-layer-KV Transformer and, once complete, writes only its last-layer KV cache, which every layer of every later chunk reads through ordinary attention. Within a chunk, training keeps a standard Transformer's parallelism. Only the most recent chunk keeps every layer's KV; every older chunk keeps only the last layer's. The remaining question is: can auto-regressive Transformers actually work from one shared last-layer cache? Across language modeling, novel view synthesis, and autoregressive video generation, experiments show that task quality holds or improves: lowers language-model validation loss and stays competitive on RULER retrieval, raises novel-view PSNR for 3D reconstruction, and maintains VBench scores for video generation. Continued pretraining converts standard Transformer weights into , which applies wherever an autoregressive model must carry long-context memory.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache, long-context language modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WJ5SOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LastKV, where all layers attend to the last-layer KV of completed context. Recurrence operates across chunks, preserving parallel processing within each chunk, while learned blending retains recent layer-local information. The architecture reduces persistent KV storage and is evaluated on language modeling, state tracking, novel-view synthesis, and video generation.

Critical strengths
  1. The design is very simple and achieve good performance. In addition, this work addresses an important memory bottleneck while retaining direct access to historical token positions.
  2. The experiment section also includes video generation.
  3. The implementation of Last Layer KV cache is easy to follow/
Critical weaknesses
  1. The title is too vague. It seems this work is comparable with Attention in the first glance, but actually far away from it. I do not think it is a proper way to hook the reader.
  2. In related work section, Feedback Transformer use deep-to-shallow feedback, CLA and YOCO use KV sharing. The combination of last-layer sharing, chunkwise recurrence, and blending is useful, but seems not a big deal. Table 1 compare with YOCO, but not include CLA and Feedback Transformer.
  3. Figure 5 demonstrates substantial cache savings, but reports inference storage rather than end-to-end inference latency. Reporting part of quality/peak memory/prefill latency/decoding latency for the same configurations would make the practical advantage more convincing.

Pls reply all weaknesses and questions point-by-point, I may reconsidering the voting based on the rebuttal.

Questions for authors
  1. What is the benefit over Feedback Transformer and CLA under matched training and storage budgets? Provide more discussion.

  2. Can the authors provide matched fixed-view Objaverse results and uncertainty estimates for the small long-context improvements?

  3. What prefill/decode latency and peak memory are achieved by the same configurations used for the main quality results?

3 comments

We thank the reviewer for considering our paper simple and effective We really appreciate that, and we would like to make a few clarifications.

W1: The title is too vague. It seems this work is comparable with Attention in the first glance, but actually far away from it. I do not think it is a proper way to hook the reader.

A1: Thank you for this valuable suggestion. We agree that the current title may be somewhat ambiguous and could lead readers to misunderstand the scope and nature of our work. We will carefully reconsider the title and explore alternative names that can better reflect the core idea and contributions of our method, while avoiding potential confusion with existing attention-based approaches.

W2 & Q1: In related work section, Feedback Transformer use deep-to-shallow feedback, CLA and YOCO use KV sharing. The combination of last-layer sharing, chunkwise recurrence, and blending is useful, but seems not a big deal. Table 1 compare with YOCO, but not include CLA and Feedback Transformer. What is the benefit over Feedback Transformer and CLA under matched training and storage budgets? Provide more discussion.

A2: We further compare LastKV with Feedback Transformer [1] and CLA [2]. As Feedback Transformer uses per-token-recurrece, training it is slow and impractical for sequence length up to 4k or 32k. In order to provide a fair comparison, we compare Feedback Transformer with LastKV using the same chunk recurrent method and the same chunk size, while maintaining Feedback Transformer’s original native feedback mechanisms.

1. 124M, short-context pretrain, same setting as the draft’s table 1

MethodWiki PPL↓LAMBADAHellaSwagWinoGr.ARC-EARC-CMMLUPIQASIQACOPABoolQAvg10↑
Transformer54.7322.231.450.843.625.924.963.336.955.060.941.50
LastKV52.6423.131.751.644.626.324.965.136.961.060.242.50
Chunked Feedback Transformer54.2422.131.752.845.725.224.564.036.661.058.642.21
CLA55.5422.231.352.143.725.024.863.536.661.055.841.61

2. 760M, short-context pretrain, same setting as the draft’s table 1

MethodWiki PPL↓LAMBADAHellaSwagWinoGr.ARC-EARC-CMMLUPIQASIQACOPABoolQAvg10↑
Transformer26.5735.746.452.156.928.925.371.838.669.060.648.50
LastKV25.9137.647.854.158.831.325.870.839.575.057.849.90
Chunked Feedback Transformer26.2237.746.952.259.331.324.872.239.069.056.448.89
CLA26.4936.946.352.057.431.525.271.137.668.053.147.91

[1] Angela Fan, Thibaut Lavril, Edouard Grave, Armand Joulin, Sainbayar Sukhbaatar, “Addressing Some Limitations of Transformers with Feedback Memory”. [2] William Brandon, Mayank Mishra, Aniruddha Nrusimha, Rameswar Panda, Jonathan Ragan Kelly, “Reducing Transformer Key-Value Cache Size with Cross-Layer Attention”.

W3 & Q3: Figure 5 demonstrates substantial cache savings, but reports inference storage rather than end-to-end inference latency. Reporting part of quality/peak memory/prefill latency/decoding latency for the same configurations would make the practical advantage more convincing. What prefill/decode latency and peak memory are achieved by the same configurations used for the main quality results?

A3: We present the prefill latency and decoding throughput as below. Below table reports single-stream performance (concurrency 1) of the Transformer (Llama-3.1-8B) and LastKV (the same 8B backbone). Both are served with vLLM 0.25 on one H100. Each request prefills an L-token prompt and then generates 4,096 tokens. We report the median of three runs. LastKV's prefill throughput is 1.6–2.9% lower than the Transformer's. Its decode throughput ranges from 2.0% lower to 0.4% higher. The prefill gap comes from processing the prompt chunk by chunk, since each chunk depends on the state left by the previous one. All tokens within a chunk are still processed in parallel, so the overhead stays small. Decode speed is essentially unchanged because, for every generated token, LastKV's 32 layers read the shared source and move the same amount of KV data as the Transformer. LastKV's advantage is therefore not per-request speed but how many long sessions fit in memory.

LengthPrefill tok/sTTFT (s)Decode tok/sE2E tok/s
TransformerLastKVTransformerLastKVTransformerLastKVTransformerLastKV
32K26,77225,9961.2241.260124.2121.7119.7117.4
64K19,55719,0443.3513.441105.4103.797.095.4
128K12,60612,40910.4010.5680.781.067.067.0

The advantage of LastKV appears when a server must keep many long sessions alive. LastKV stores 4 KiB of cache per token (one source shared by all layers), plus a fixed two-chunk local window per layer for each session, versus 128 KiB per token for the Transformer. Below table compares the two at fixed concurrency. We report steady-state output throughput: the tokens generated by all concurrent sessions over three follow-up turns (128 tokens per session per turn), divided by the wall-clock time of those turns. In every setting of the table, the concurrency exceeds the Transformer's resident capacity. Its evicted sessions re-prefill their whole context on every turn, so its throughput collapses to the prefill rate regardless of concurrency. Both systems use vLLM's default prefix cache. These throughput gains are specific to memory-oversubscribed serving scenarios

ConcurrencyMethod32K64K128K256K
16Transformer873311.33.4
16LastKV44825914174
32Transformer8833——
32LastKV496275——
64Transformer8933——
64LastKV523283——

Q2: Can the authors provide matched fixed-view Objaverse results?

A2: We applied the fixed-view evaluation to the LastKV and baseline checkpoints on the full GSO test set. We report the fixed-view performance with intervals based on full test samples. As the table shows, LastKV obtains 25.97 PSNR (95% CI [25.79, 26.14]), against 25.36 [25.14, 25.59] for Transformer, and the two intervals do not overlap.

MethodObjaverse (GSO) PSNR ↑
Transformer25.36 [25.14, 25.59]
DeltaNet23.94 [23.76, 24.12]
Gated DeltaNet24.32 [24.14, 24.50]
LaCT23.99 [23.77, 24.20]
LastKV25.97 [25.79, 26.14]

Dear reviewer WJ5S,

We would like to provide a few additional clarification for the peak memory usage.

W3 & Q3: Figure 5 demonstrates substantial cache savings, but reports inference storage rather than end-to-end inference latency. Reporting part of quality/peak memory/prefill latency/decoding latency for the same configurations would make the practical advantage more convincing. What prefill/decode latency and peak memory are achieved by the same configurations used for the main quality results?

A3: We report the peak GPU memory usage for the same 8B backbone, with LastKV using a 4K chunk size, consistent with our main quality evaluation. We break down memory consumption into model weights, runtime workspace, and KV/state storage. As shown below, LastKV reduces total peak memory from 20.7/24.7/32.7/48.7 GiB to 17.8/17.9/18.2/18.7 GiB at 32K/64K/128K/256K context lengths, corresponding to reductions of 14%/28%/44%/62%. Since model weights and runtime workspace remain identical, these savings come entirely from the reduced KV/state footprint. Notably, as context length increases from 32K to 256K, LastKV's total memory grows by only 0.9 GiB, compared with 28.0 GiB for the Transformer. This confirms that LastKV's cache savings translate directly into substantially lower end-to-end inference memory usage, particularly at long context lengths.

SeqlenMethodModel WeightBufferKV WeightTotal
32KTransformer15.061.104.5020.7
32KLastKV15.061.101.6417.8
64KTransformer15.061.108.5024.7
64KLastKV15.061.101.7717.9
128KTransformer15.061.1016.5032.7
128KLastKV15.061.102.0218.2
256KTransformer15.061.1032.5048.7
256KLastKV15.061.102.5218.7
Reviewer_NPGZOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes LastKV, where for the Transformer every layer attahced to the last layer's K and V instead of using the original K V. Then the paper claims by doing this, the storage is cut down by L, also deep feature can feed into shallow layer by doing the recurrent operation with chunked tokens. The experiments show a comprehensive results, from the LLM to video generation and more, LaskKV matches or even better than Transformers whiling using less KV memory.

Critical strengths

Strengths:

  1. The idea itself is interesting, simple but do have the real impact. Just a simply modification of the Transformer architecure can bring such a change of the results.

  2. This paper carries out a really comprehensive experiment results, LLM, video generation, synthetic state tracking, novel-view synthesis etc, it is a broad cover for the experiments. And LastKV do show a great potential on all these directions.

Critical weaknesses

Weakness:

  1. The first weakness DO concern me a lot, I feel like incremental novely here, the core mechanism 'last layer KV sharing' came our early back in 2024 from LCKV (Wu & Tu, ACL 2024), and then Wu et al. (NAACL 2025) further studied this top-layer sharing family systematically, neither paper is cited. Please correct me if i am wrong,the main difference to me here is LCKV use per token sharing, LastKV use per chunk sharing with the recurrence mechanism, so it will be better to see how previous method like LCKV, NAACL, Feedback Transformer behave on those bencmark.

  2. Since the paper introduce the recurrence here, I am not sure if this recurrence does help, for fig8, recurrence only help on LLM long acc only, the rest abalation results feel like nearly the same within fig.8.

  3. The setting of the experiments also confuses me. I am not sure what is the motivation for the 'reusekv', 'perlayerkv' and 'sharekv'. Inside table 1, 124M uses the perlayerkv, 760M uses the sharekv, But from Table7 we can tell, if use reusekv, 124M and 760M will score as 17.5 (less than Hysparse 18.74) and 30.7(less than Transformer 34.2), so i am not sure which standard to use in real case.

I am happy to raise my score if my concern is well-addressed.

Questions for authors

Besides the weakness, i also have some questions

  1. what is the chunkation size for each task, and i am happy to see a chunk size sweep for some easy run results to see how the chunkation size help.

  2. I am also curious about the inference time latency.

9 comments

We thank the reviewer for considering our paper interesting, simple and having real impact! We really appreciate that, and we would like to make a few clarifications.

W1: Two related works (i.e., LCKV Wu & Tu, ACL 2024, and Wu et al. (NAACL 2025) ) uncited. LCKV uses per token sharing, LastKV uses per chunk sharing with recurrence mechanism. It would be better to compare LCKV and Feedback Transformer with LastKV.

A1: We agree that comparing LastKV with both Feedback Transformer [1] and LCKV [2] would provide a more comprehensive evaluation. We have therefore added experiments comparing LastKV with Feedback Transformer and LCKV at both 124M and 760M model scales, under the same short-pretraining setting as below tables. For Feedback Transformer, its original per-token recurrence introduces substantial training overhead, making it computationally expensive for long sequences (e.g., 4K and 32K). To enable a more practical and controlled comparison, we adapt its recurrence to the chunk-level setting, using the same chunk size and chunk-wise recurrent training procedure as LastKV, while preserving its original feedback mechanism. We also include LCKV, which employs per-token KV sharing and requires two forward passes in our experimental setup, in contrast to LastKV's per-chunk KV sharing with recurrent updates. The results show that LastKV consistently achieves the best average performance (Avg10) and the lowest WikiText perplexity at both model scales. These results further demonstrate the effectiveness of LastKV's chunk-level KV sharing and recurrent mechanism.

1. 124M, short-context pretrain, same setting as the draft’s table 1

MethodWiki PPL↓LAMBADAHellaSwagWinoGr.ARC-EARC-CMMLUPIQASIQACOPABoolQAvg10↑
Transformer54.7322.231.450.843.625.924.963.336.955.060.941.50
LastKV52.6423.131.751.644.626.324.965.136.961.060.242.50
Chunked Feedback Transformer54.2422.131.752.845.725.224.564.036.661.058.642.21
LCKV(2 forwards)59.8220.530.550.343.625.125.563.637.256.062.041.42

2. 760M, short-context pretrain, same setting as the draft’s table 1

MethodWiki PPL↓LAMBADAHellaSwagWinoGr.ARC-EARC-CMMLUPIQASIQACOPABoolQAvg10↑
Transformer26.5735.746.452.156.928.925.371.838.669.060.648.50
LastKV25.9137.647.854.158.831.325.870.839.575.057.849.90
Chunked Feedback Transformer26.2237.746.952.259.331.324.872.239.069.056.448.89
LCKV(2 forwards)29.9532.443.749.856.630.424.771.338.366.061.347.44

[1] Angela Fan, Thibaut Lavril, Edouard Grave, Armand Joulin, Sainbayar Sukhbaatar, “Addressing Some Limitations of Transformers with Feedback Memory”.

[2] Haoyi Wu, Kewei Tu , “Layer-Condensed KV Cache for Efficient Inference of Large Language Models”.

W2: The paper introduces the recurrence, but I am not sure if the recurrence helps.

A2: I think recurrence is an inevitable consequence of the LastKV design. Since all layers reuse the last-layer KV states of previously processed chunks, the deep representations of past tokens are inevitably fed back into the computation of future chunks. This naturally introduces recurrence across both time and depth. Therefore, recurrence is not an additional mechanism arbitrarily introduced by the paper, but an inherent property of sharing last-layer KV across all layers.

W3: The setting of the experiments also confuses me. I am not sure what is the motivation for the 'reusekv', 'perlayerkv' and 'sharekv'. I am not sure which standard to use in real case.

A3: The three variants explore the trade-off between efficiency and representation flexibility: reusekv directly reuses the last-layer KV, introducing no additional projections and offering the simplest implementation. sharekv uses a shared projection to construct historical KV, providing additional flexibility with limited overhead. perlayerkv uses layer-specific projections, allowing each layer to access historical information in its own representation space. In practice, we recommend reusekv for efficiency-oriented scenarios, while sharekv and perlayerkv can be considered when prioritizing performance. No single variant consistently dominates across tasks. We will clarify these motivations and practical guidelines in the revised paper.

Q1: what is the chunkation size for each task, and i am happy to see a chunk size sweep for some easy run results to see how the chunkation size help.

A1: We conducted chunk size sweeps on both short-context and long-context tasks to investigate the effect of chunk size on model performance. Specifically, we evaluated chunk sizes (C \in {256, 512, 1024}) on standard language modeling and downstream benchmark tasks, and (C \in {1024, 2048, 4096}) on long-context evaluation using RULER.

1. 124M, short-context pretrain, same setting as the draft’s table 1

Chunk SizeWikiText PPLLAMBADAHellaSwagWinoGrandeARC-EARC-CMMLUPIQASIQACOPABoolQAverage
25652.9123.5831.5950.1244.2326.1125.4064.4736.5961.0059.9442.30
51252.6823.4432.2952.3345.2025.0024.1066.5436.8057.0058.9042.16
102452.6423.0731.7551.6244.5726.2824.9265.1336.9061.0060.2142.54

The results show that performance is relatively stable across different chunk sizes. Among the three configurations, (Chunk=1024) achieves the best average downstream benchmark score.

2. 760M, long-context pretrain, same setting as the draft’s table 2

We further conducted a chunk size sweep on RULER to investigate the impact of chunk size on long-context modeling.

Chunk Size (C)Tail PTLFull-sequence PTLRULER Average4K8K16K32K
10242.46942.493736.6745.1740.5234.6026.40
20482.47032.494842.9655.8147.2038.6330.21
40962.47282.497743.2453.7450.0241.4427.75

The results indicate that chunk size has a more noticeable impact on long-context performance. Increasing the chunk size from 1024 to 2048 substantially improves the RULER average score. Increasing it further to 4096 yields a smaller additional improvement.

Q2: I am also curious about the inference time latency.

A2: We present the prefill latency and decoding throughput as below. Below table reports single-stream performance (concurrency 1) of the Transformer (Llama-3.1-8B) and LastKV (the same 8B backbone). Both are served with vLLM 0.25 on one H100. Each request prefills an L-token prompt and then generates 4,096 tokens. We report the median of three runs. LastKV's prefill throughput is 1.6–2.9% lower than the Transformer's. Its decode throughput ranges from 2.0% lower to 0.4% higher. The prefill gap comes from processing the prompt chunk by chunk, since each chunk depends on the state left by the previous one. All tokens within a chunk are still processed in parallel, so the overhead stays small. Decode speed is essentially unchanged because, for every generated token, LastKV's 32 layers read the shared source and move the same amount of KV data as the Transformer. LastKV's advantage is therefore not per-request speed but how many long sessions fit in memory.

LengthPrefill tok/sTTFT (s)Decode tok/sE2E tok/s
TransformerLastKVTransformerLastKVTransformerLastKVTransformerLastKV
32K26,77225,9961.2241.260124.2121.7119.7117.4
64K19,55719,0443.3513.441105.4103.797.095.4
128K12,60612,40910.4010.5680.781.067.067.0

The advantage of LastKV appears when a server must keep many long sessions alive. LastKV stores 4 KiB of cache per token (one source shared by all layers), plus a fixed two-chunk local window per layer for each session, versus 128 KiB per token for the Transformer. Below table compares the two at fixed concurrency. We report steady-state output throughput: the tokens generated by all concurrent sessions over three follow-up turns (128 tokens per session per turn), divided by the wall-clock time of those turns. In every setting of the table, the concurrency exceeds the Transformer's resident capacity. Its evicted sessions re-prefill their whole context on every turn, so its throughput collapses to the prefill rate regardless of concurrency. Both systems use vLLM's default prefix cache. These throughput gains are specific to memory-oversubscribed serving scenarios

ConcurrencyMethod32K64K128K256K
16Transformer873311.33.4
16LastKV44825914174
32Transformer8833——
32LastKV496275——
64Transformer8933——
64LastKV523283——
Reviewer_HrejOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LastKV, a Transformer architecture that shares the last-layer KV representations of completed context across all layers. This reduces persistent KV storage and introduces feedback from deep representations into shallow-layer computation. Experiments cover various tasks including language modeling, synthetic state tracking, novel-view synthesis, and video generation, verify the effectiveness of the method.

Critical strengths
  1. Sharing deep historical KV representations across layers is simple and interesting. The motivation is clear.
  2. Experiments cover diverse tasks and models. Extensive results verify the efficacy of the proposed method.
  3. Physical KV allocations and training times are presented. The analysis of chunk-boundary artifacts and previous-chunk blending also helps explain an important implementation challenge. They are important for practical application.
Critical weaknesses
  1. The practical inference trade-off is not fully evaluated. Figure 5 reports training time and inference KV allocation, but not prefill latency or decoding throughput, which is more practical.
  2. The distinction from closely related feedback architectures needs stronger experimental support. The paper discusses Feedback Transformer and other recurrent architectures, but the adapted T²MLR-LastKV and FBT-LastKV baselines restrict those methods to last-layer historical KV. Their performance therefore does not establish an advantage over the original architectures. Comparisons with native feedback mechanisms would clarify the benefit of the proposed method.
Questions for authors

Please refer to the Weaknesses.

4 comments

We thank the reviewers for considering our methods simple and interesting, and we would like to provide a few clarification.

W1: The practical inference trade-off is not fully evaluated. Lack of prefill latency or decoding throughput.

A1: We present the prefill latency and decoding throughput as below. Below table reports single-stream performance (concurrency 1) of the Transformer (Llama-3.1-8B) and LastKV (the same 8B backbone). Both are served with vLLM 0.25 on one H100. Each request prefills an L-token prompt and then generates 4,096 tokens. We report the median of three runs. LastKV's prefill throughput is 1.6–2.9% lower than the Transformer's. Its decode throughput ranges from 2.0% lower to 0.4% higher. The prefill gap comes from processing the prompt chunk by chunk, since each chunk depends on the state left by the previous one. All tokens within a chunk are still processed in parallel, so the overhead stays small. Decode speed is essentially unchanged because, for every generated token, LastKV's 32 layers read the shared source and move the same amount of KV data as the Transformer. LastKV's advantage is therefore not per-request speed but how many long sessions fit in memory.

LengthPrefill tok/sTTFT (s)Decode tok/sE2E tok/s
TransformerLastKVTransformerLastKVTransformerLastKVTransformerLastKV
32K26,77225,9961.2241.260124.2121.7119.7117.4
64K19,55719,0443.3513.441105.4103.797.095.4
128K12,60612,40910.4010.5680.781.067.067.0

The advantage of LastKV appears when a server must keep many long sessions alive. LastKV stores 4 KiB of cache per token (one source shared by all layers), plus a fixed two-chunk local window per layer for each session, versus 128 KiB per token for the Transformer. Below table compares the two at fixed concurrency. We report steady-state output throughput: the tokens generated by all concurrent sessions over three follow-up turns (128 tokens per session per turn), divided by the wall-clock time of those turns. In every setting of the table, the concurrency exceeds the Transformer's resident capacity. Its evicted sessions re-prefill their whole context on every turn, so its throughput collapses to the prefill rate regardless of concurrency. Both systems use vLLM's default prefix cache. These throughput gains are specific to memory-oversubscribed serving scenarios

concurrencymethod32K64K128K256K
16Transformer873311.33.4
16LastKV44825914174
32Transformer8833——
32LastKV496275——
64Transformer8933——
64LastKV523283——

W2: The distinction from closely related feedback architectures needs stronger experimental support. Comparisons with native feedback mechanisms would clarify the benefit of the proposed method.

A2: We further compare LastKV with Feedback Transformer [1] and LCKV [2]. We have added experiments comparing LastKV with Feedback Transformer and LCKV at both 124M and 760M model scales, under the same short-pretraining setting as below tables. For Feedback Transformer, its original per-token recurrence introduces substantial training overhead, making it computationally expensive for long sequences (e.g., 4K and 32K). To enable a more practical and controlled comparison, we adapt its recurrence to the chunk-level setting, using the same chunk size and chunk-wise recurrent training procedure as LastKV, while preserving its original feedback mechanism. We also include LCKV, which employs per-token KV sharing and requires two forward passes in our experimental setup, in contrast to LastKV's per-chunk KV sharing with recurrent updates. The results show that LastKV consistently achieves the best average performance (Avg10) and the lowest WikiText perplexity at both model scales. These results further demonstrate the effectiveness of LastKV's chunk-level KV sharing and recurrent mechanism.

  1. 124M, Short Pretrain, Same as Table 1
MethodWiki PPL↓LAMBADAHellaSwagWinoGr.ARC-EARC-CMMLUPIQASIQACOPABoolQAvg10↑
Transformer54.7322.231.450.843.625.924.963.336.955.060.941.50
LastKV52.6423.131.751.644.626.324.965.136.961.060.242.50
Chunked Feedback Transformer54.2422.131.752.845.725.224.564.036.661.058.642.21
LCKV59.8220.530.550.343.625.125.563.637.256.062.041.42
  1. 760M, Short Pretrain, Same as Table 1
MethodWiki PPL↓LAMBADAHellaSwagWinoGr.ARC-EARC-CMMLUPIQASIQACOPABoolQAvg10↑
Transformer26.5735.746.452.156.928.925.371.838.669.060.648.50
LastKV25.9137.647.854.158.831.325.870.839.575.057.849.90
Chunked Feedback Transformer26.2237.746.952.259.331.324.872.239.069.056.448.89
LCKV29.9532.443.749.856.630.424.771.338.366.061.347.44

[1] Angela Fan, Thibaut Lavril, Edouard Grave, Armand Joulin, Sainbayar Sukhbaatar, “Addressing Some Limitations of Transformers with Feedback Memory”.

[2] Haoyi Wu, Kewei Tu , “Layer-Condensed KV Cache for Efficient Inference of Large Language Models”.

Dear Reviewer Hrej,

Thank you again for your valuable feedback. In response to W2, we would like to provide additional experimental evidence to further clarify the distinction between LastKV and existing feedback architectures.

W2: The distinction from closely related feedback architectures needs stronger experimental support. The paper discusses Feedback Transformer and other recurrent architectures, but the adapted T²MLR-LastKV and FBT-LastKV baselines restrict those methods to last-layer historical KV. Their performance therefore does not establish an advantage over the original architectures. Comparisons with native feedback mechanisms would clarify the benefit of the proposed method.

A2: We agree that the previously reported T²MLR-LastKV and FBT-LastKV variants do not fully reflect the capabilities of their original architectures, as they restrict historical KV representations to the last layer.

To address this concern, we conduct additional experiments with FBT, LRT, and T²MLR using their own feedback mechanisms, rather than the LastKV-adapted variants. We also include the Chunked Feedback Transformer for comparison. All methods are evaluated under the same 124M short-pretraining setting as Table 2.

The implementation details are summarized below:

  • LastKV processes each chunk through the backbone once.
  • Chunked Feedback Transformer [1]: Since the original Feedback Transformer's per-token recurrence is computationally expensive for long sequences (e.g., 4K or 32K), we adopt the same chunk size and chunk-wise recurrence as LastKV while preserving its native feedback mechanism.
  • FBT [2] performs 3 full forward passes over the entire sequence, totaling 36 layer calls.
  • LRT [3] performs one full forward pass followed by alternating updates of even and odd token positions, equivalent to approximately 24 full-sequence layer calls.
  • T²MLR [4] performs one initial full forward pass, 15 recurrent iterations over layers 6–7, and one final full forward pass, totaling 54 layer calls. Gradients are truncated for earlier recurrent iterations.

These configurations retain the core feedback mechanisms of the respective methods, with the approximations noted above. In contrast to the repeated computation required by FBT, LRT, and T²MLR, LastKV enables cross-layer feedback through shared historical KV representations with only a single forward pass per chunk, resulting in substantially lower training overhead.

1. Performance comparison

MethodWiki PPL↓LAMBADAHellaSwagWinoGr.ARC-EARC-CMMLUPIQASIQACOPABoolQAvg10↑
Transformer54.7322.231.450.843.625.924.963.336.955.060.941.50
LastKV52.4623.231.651.946.024.723.765.837.359.061.542.47
Chunked Feedback Transformer54.3222.331.351.744.725.325.364.337.459.062.042.32
FBT52.5823.031.652.744.926.524.064.437.660.059.742.43
LRT52.6220.631.551.145.325.324.164.936.258.061.641.85
T²MLR51.2323.532.054.244.424.825.163.438.257.060.242.29

The results show that LastKV achieves the highest average accuracy (42.47) across the ten downstream tasks, outperforming the Transformer baseline and remaining competitive with the native feedback architectures. While T²MLR achieves a lower WikiText perplexity, LastKV provides the best overall downstream average.

2. Parameter and training efficiency

Beyond predictive performance, we further compare model parameters and training costs.

MethodParametersTraining time
Transformer134.11M1×
LastKV134.11M≈1.1×
Chunked Feedback Transformer134.11M≈2.4×
FBT135.29M (+0.9%)≈2.0×
LRT148.48M (+10.7%)≈4.0×
T²MLR137.06M (+2.2%)≈1.9×

Importantly, LastKV achieves these results without introducing additional model parameters and incurs only approximately 10% training overhead relative to the standard Transformer. In comparison, FBT and T²MLR require approximately 1.9–2.0× the baseline training time, while LRT requires approximately 4×.

These results highlight the main distinction of LastKV: rather than introducing more expensive feedback computations or iterative refinement, LastKV reuses the last-layer historical KV representations across layers, enabling feedback from deep representations with minimal additional computational cost.

Overall, the additional comparisons demonstrate that LastKV offers a favorable performance–efficiency trade-off even against architectures using their own feedback mechanisms. Although individual alternatives may perform better on specific metrics, LastKV achieves competitive predictive performance with substantially lower training overhead and no additional parameters. We believe these experiments directly address the reviewer's concern and further clarify the practical benefits of our proposed design.

[1] Angela Fan, Thibaut Lavril, Edouard Grave, Armand Joulin, Sainbayar Sukhbaatar, “Addressing Some Limitations of Transformers with Feedback Memory”.

[2] Xi Wang, Ziyang Cai, Zheng Zhan, Harry Dong, Ying Fan, Gustavo de Rosa, Tim Pearce, John Langford, “Full-bandwidth Transformer”.

[3] Zeyi Huang, Xuehai He, Liliang Ren, Yiping Wang, Baolin Peng, Hao Cheng, Shuohang Wang, Pengcheng He, Jianfeng Gao, Yong Jae Lee, Yelong Shen, “Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior”.

[4] Ziyang Cai, Xingyu Zhu, Yihe Dong, Yinghui He, Sanjeev Arora, “T²MLR: Transformer with Temporal Middle-Layer Recurrence”.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to Reviewers and Area Chair

We thank all three reviewers for their careful reading and constructive feedback. We appreciate their recognition of LastKV's architectural simplicity, substantial KV-cache savings, and broad evaluation across language modeling, novel view synthesis, and video generation. Reviewer Hrej raised concerns about practical inference efficiency and the need for comparisons with feedback architectures using their native mechanisms. Reviewer NPGZ questioned the novelty relative to LCKV, the contribution of recurrence, and the consistency of KV configurations and experimental settings. Reviewer WJ5S emphasized clearer positioning relative to prior work, comparisons with CLA and Feedback Transformer, and more rigorous measurements of inference efficiency and generation quality. These suggestions motivated additional architecture comparisons, recurrence and chunk-size ablations, inference profiling, and controlled evaluations.

We summarize the additional results below and address each reviewer's concerns in the individual responses.

  • Architectural contribution and distinction from prior work (NPGZ, WJ5S). We clarify that LastKV's contribution is not the original idea of last-layer KV sharing or deep-to-shallow feedback, which have been explored in LCKV and Feedback Transformer. Rather, LastKV realizes last-layer historical KV sharing across all Transformer layers through chunk-level recurrence, without requiring iterative forward computation. Each chunk is processed through the Transformer layers only once, while its last-layer KV representations are shared across subsequent chunks. This preserves within-chunk parallelism, enables deep-to-shallow feedback, and reduces historical KV-cache storage to approximately 1/L of standard per-layer caching, apart from a small layer-local window.

  • Comparison with LCKV under its recommended configuration (NPGZ). Following the reviewer's suggestion, we reproduced LCKV with two warm-up layers, seven forward iterations, and two backward iterations, rather than relying only on the earlier two-forward-iteration baseline. At 124M, LastKV achieves slightly higher ten-task mean accuracy (42.50 vs. 42.40), while LCKV achieves slightly lower WikiText perplexity (51.95 vs. 52.64). Importantly, LCKV requires 2.6× the training FLOPs of LastKV under its recommended configuration. This comparison highlights LastKV's primary distinction: achieving competitive predictive performance with substantially less iterative computation.

  • Comparisons with native feedback and KV-sharing architectures (Hrej, WJ5S). We extend our comparisons to FBT, LRT, T²MLR, Chunked Feedback Transformer, and CLA, evaluating feedback architectures using their own mechanisms rather than only LastKV-adapted variants. At 124M, LastKV achieves the highest ten-task mean accuracy (42.47), compared with 42.43 for FBT, 42.29 for T²MLR, and 41.85 for LRT. LastKV requires only 1.1× Standard Transformer's training time, versus approximately 2.0×, 1.9×, and 4.0× for FBT, T²MLR, and LRT, respectively, without introducing additional parameters. At 760M, LastKV also achieves higher mean accuracy (49.90) than Standard Transformer (48.50), Chunked Feedback Transformer (48.89), and CLA (47.91). These comparisons support a favorable quality–compute trade-off, while acknowledging that individual baselines perform better on certain metrics.

  • Contribution of recurrence (NPGZ). We clarify that recurrence arises naturally from reusing the last-layer KV states of completed chunks under LastKV's single-pass computation scheme. Our non-recurrent ablation modifies the construction of historical representations to isolate the contribution of recurrent feedback, rather than simply disabling an independent module. The ablation in Figure 8 shows improvements across multiple tasks: recurrence increases novel-view synthesis PSNR by approximately 0.66 dB and average video-generation VBench score by 0.54 points, while also improving language-modeling perplexity and long-context accuracy. These results provide evidence that recurrent feedback contributes beyond historical KV sharing alone.

  • KV configuration consistency and chunk-size sensitivity (NPGZ). We clarify the trade-offs among reusekv, sharekv, and perlayerkv, and will adopt reusekv as the default configuration for primary results in the revision to align quality and efficiency comparisons. Results for alternative variants will be reported separately. We also evaluate chunk sizes from 256 to 1024 in short-context settings, where performance remains relatively stable, and from 1024 to 4096 in long-context settings, where retrieval accuracy is more sensitive to chunk size.

  • Practical inference efficiency and peak memory (Hrej, NPGZ, WJ5S). We provide additional prefill, decoding, and GPU-memory measurements using the same Llama-3.1-8B backbone on an H100. Across 32K–128K contexts, LastKV incurs only 1.6–2.9% lower prefill throughput, while decoding throughput remains essentially unchanged (−2.0% to +0.4%). More importantly, total peak GPU memory decreases from 32.7 to 18.2 GiB at 128K and from 48.7 to 18.7 GiB at 256K, corresponding to 44% and 62% reductions, respectively. From 32K to 256K, LastKV's total GPU memory increases by only 0.9 GiB, compared with 28.0 GiB for Standard Transformer. These results demonstrate that the historical KV-cache reduction translates into substantial end-to-end memory savings with modest inference overhead.

  • Multi-session serving efficiency (Hrej, WJ5S). To evaluate whether KV-cache savings translate into practical serving benefits, we additionally measure aggregate throughput under memory-oversubscribed long-context serving scenarios. With 16 concurrent 64K sessions, LastKV achieves 259 tokens/s versus 33 tokens/s for Standard Transformer, as its smaller KV footprint reduces repeated context prefilling caused by cache eviction. These gains are specific to memory-constrained multi-session serving rather than faster single-request decoding, clarifying the practical setting in which LastKV provides its largest benefits.

  • Controlled novel-view synthesis and evaluation reliability (WJ5S). We conduct fixed-view evaluation on the full GSO test set and report uncertainty estimates. LastKV achieves 25.97 PSNR (95% CI [25.79, 26.14]), compared with 25.36 (95% CI [25.14, 25.59]) for Standard Transformer. The non-overlapping confidence intervals provide additional evidence for LastKV's reconstruction-quality improvement under matched evaluation conditions. Together with our language-modeling and video-generation results, these findings support the applicability of chunk-level last-layer KV sharing across different autoregressive tasks.

Together, these additional experiments strengthen the evidence for LastKV's quality–compute–memory trade-off and clarify its relationship to existing KV-sharing and feedback architectures. In particular, LastKV's contribution lies in enabling efficient deep-to-shallow historical information sharing through chunk-level recurrence, rather than introducing last-layer KV sharing itself. We will also clarify the relevant prior work, baseline configurations, and title in the revision. We thank the reviewers again for their constructive suggestions, which have helped us present a more rigorous and transparent evaluation of LastKV.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.