acceptodds
Under review as a conference paper at ICLR 2027

COMB: An Encoder Plug-in for Native Position-Independent Caching

Abstract

The Key-Value (KV) cache of Large Language Models (LLMs) is prefix-based, making it highly inefficient for processing contexts retrieved in arbitrary order. Position-Independent Caching (PIC) has been proposed to enable KV reuse without positional constraints; however, existing approaches often incur substantial accuracy degradation, limiting their practical adoption. To address this issue, we propose native PIC by reintroducing the encoder to prevalent decoder-only LLMs and explicitly training it to support PIC. We further develop COMB, a PIC-aware caching system that integrates seamlessly with existing inference frameworks. Experimental results show that COMB reduces Time-to-First-Token (TTFT) by 51-94% and increases throughput by 3 with comparable accuracy. Furthermore, the evaluation on DeepSeek-V2-Lite-Chat demonstrates the applicability of COMB to other types of decoder-only LLMs. Our code is available at https://anonymous.4open.science/r/Comb.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Model, Position-Independent Caching, Encoder-Decoder Transformer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qV3xOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces COMB, a position-independent caching method that attaches a separate trainable encoder to a frozen decoder-only LLM. Document chunks are encoded independently by the encoder and then cached, and the decoder reads them through interleaved cross-attention layers. Only the encoder and cross-attention layers are trained, and the original model can be recovered by disabling them. The authors also integrated a PIC managements system on vLLM. On LongBench subtasks, COMB achieves 51-94% lower TTFT on cache hits, up to 3$\times$ higher throughput, while reducing 75% of KV memory.

Critical strengths

S1. Non-intrusive design.

Training only an added encoder and cross-attention layers on top of a frozen decoder lets COMB reduce exactly to the original model when disabled, which addresses the trade-off between post-training and training-aware PIC. Because documents are processed by a shallow encoder and stored in only a few layers, the architecture also reduces KV memory and encoder compute, supported by a complexity analysis.

S2. A complete, openly released implementation with transparent costs.

The paper provides a working PIC management system integrated with vLLM, anonymized code and trained models, and reports training cost and limitations, which makes the approach practical to reproduce and build on.

Critical weaknesses

W1. Related work and baseline coverage.

CEPE [1], which uses a small encoder for processing chunks, seem to be close prior work but not discussed. On the PIC side, BlockAttention is the only training-aware baseline, whereas KVLink [2] and TurboRAG [3], placed in the same category by the paper, are omitted.

The BlockAttention numbers (HotpotQA F1 of about 13 vs. 44 for prefix caching, as read from Figs. 4 and 6) are far below the parity with full attention reported in the original paper. The checkpoint and prompt format used are not described, and BlockAttention is excluded from the TTFT comparison, so “lowest TTFT among all approaches” is unverified against it.

W2. The evaluation does not match the agentic motivation.

The paper motivates PIC with RAG agentic and workloads and static content such as system prompts and few-shot examples, yet evaluates only single-turn LongBench tasks under a 100% cache-hit protocol with short outputs. Adding results on multi-turn agentic tasks with realistic hit rates, and instruction-following behavior when system prompts or instructions are routed through the encoder path would be helpful.

W3. Incomplete efficiency reporting and ablations.

Only TTFT and throughput are reported. TPOT and end-to-end latency are missing, although cross-attention layers add per-token compute. The hit rate used in the online experiment (Fig. 10) is not stated.

Also, encoder depth and cross-attention placement, which determine this accuracy–efficiency trade-off, are not ablated, although Section 3.3 suggests accuracy improves with more encoder layers.

W4. The system component is described only at a high level.

The paper does not describe cache eviction or tiering, how variable-length cross-attention KV is integrated with paged KV management and scheduling in vLLM, or an overhead breakdown. The memory and TTFT gains follow largely from the architecture rather than the system.

[1] Yen, Howard, Tianyu Gao, and Danqi Chen. "Long-context language modeling with parallel context encoding." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

[2] Yang, Jingbo, et al. "Kvlink: Accelerating large language models via efficient kv cache reuse." Advances in Neural Information Processing Systems 38 (2026): 133797-133824.

[3] Lu, Songshuo, et al. "Turborag: Accelerating retrieval-augmented generation with precomputed kv caches for chunked text." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Questions for authors

Q1. Can KVLink and, if feasible, TurboRAG, which the paper places in the same training-aware category, be compared under controlled settings? Can the training-free baselines (CacheBlend, EPIC) be evaluated at several recomputation budgets, with accuracy–TTFT curves, in the same serving engine and version?

Q2. Which BlockAttention checkpoint, base model, and prompt format were used? Please report results with the official model and its recommended input format, and its TTFT in the same serving environment.

Q3. How does COMB perform on multi-turn agentic tasks with realistic hit rates, including task success, end-to-end latency, and instruction-following behavior when system prompts are routed through the encoder?

Q4. What are TPOT and end-to-end latency across batch sizes and document lengths, and what is the net memory saving once the extra weights are counted? What hit rate was used in Fig. 10? How do accuracy-efficiency tradeoff changes with the number of encoder layers and the cross-attention placement?

Q5. Does the system implement eviction or tiering of PICaches, and how is variable-length cross-attention KV handled in vLLM’s paged KV and scheduler? Can an overhead breakdown be provided?

6 comments
AuthorsOpenReview ↗

Response (Part 1)

Thank you for the review. We believe some of the concerns arise primarily from presentation clarity, and we will revise the paper to make the mechanism and scope more explicit.

[W1] Related work on parallel encoding

We agree that CEPE shares similarities with our paper. We will incorporate a discussion of CEPE in the Related Work section of our revised paper. However, we believe our work is fundamentally different for three reasons:

  1. Motivation and core concept: CEPE proposes parallel encoding, meaning that when processing very long inputs for prefill, it can parallelize the prefill phase. In contrast, our approach focuses on trading storage for computation, enabling existing KV caches to be reused at arbitrary positions (which we call PIC). Our starting point is entirely different.
  2. Methodological differences: CEPE employs a smaller hidden size and a shallower encoder depth. Furthermore, it only uses the final hidden states from the encoder's output for cross-attention, whereas we leverage the hidden states from every layer of the encoder for cross-attention. The ablation study on encoder depth demonstrates that their accuracy would be lower than ours due to their smaller model size.
  3. System implementation: We have built COMB, a PIC-aware caching system that integrates with existing inference frameworks. CEPE focuses solely on parallel encoding and lacks a storage system to manage its KV cache. As Reviewer ouYC noted in the review,

Authors didn't stop at training the COMB-model. A further serving system was designed and implemented based on Transformers and vLLM. With this serving system, the paper can present more comprehensive inference performance evaluation.

[Q1][W1] Comparison with the baselines

The KVLink paper didn't release its 8B model, only the 1B and 3B versions. Retraining a 8B model from scratch would require significant computational resources.

On the other hand, the method used by TurboRAG is almost identical to BlockAttention, just as the BlockAttention paper describes:

We were surprised to find that another ICLR 2025 submission, TurboRAG (Lu & Tang, 2025), independently proposed similar methods and reached similar conclusions as ours. They also proposed independent attention (which is the block attention in this paper) and reordered positions (the position re-encoding in this paper).

Therefore, we selected BlockAttention as a representative of training-aware PIC methods to illustrate that existing training-aware PIC methods are intrusive and permanently alter model parameters. Existing training-aware methods, regardless of their training scale or accuracy, fundamentally modify the model’s behavior in a persistent manner. In contrast, our method operates without altering the underlying model, preserving its original behavior while enabling the desired functionality.

[Q2][W1] Details on evaluating BlockAttention

We used the open-source checkpoint ldsjmdy/Tulu3-Block-FT from the BlockAttention paper. The code we used was copied directly from their open-source repository; further details can be found in the benchmarks/block_attention folder of the anonymous repository linked in our abstract.

The experimental results on the HotpotQA dataset differ from the original paper because the original study used the OpenCompass dataset, whereas we used the LongBench dataset. HotpotQA is simply the task name, and the LongBench dataset is more challenging than OpenCompass, which is why the BlockAttention scores are lower in our paper.

[W2][Q3] Motivation on RAG

We want to clarify that in the Introduction section, we did not state that our motivation is handling agentic workloads. The scenario we target is strictly RAG. We would like to clarify the scope of our work and the rationale behind our evaluation choices.

Scope. COMB targets RAG, where independently retrieved documents are reused across requests in varying combinations and orders. This is exactly the setting that calls for position-independent KV caching. Agentic workloads are also an important and emerging setting worth studying. However, their reuse pattern (multi-turn tool calls, iterative context updates) is quite different, and the representative works in this line, such as CacheBlend, EPIC, TurboRAG, and Block-Attention, have not yet addressed it either. We will explore it in future work.

Benchmarks. LongBench-style multi-document QA and summarization are the standard benchmarks for this line of work. Representative prior works, such as CacheBlend (EuroSys'25, Best Paper), EPIC (ICML'25), TurboRAG (EMNLP'25), and Block-Attention (ICLR'25), use the same protocol.

Our evaluation follows this established practice and can be compared directly with these baselines. Asking for a different benchmark suite would hold COMB to a standard that none of the prior work in this area was held to. We will state the scope more explicitly in the revision.

AuthorsOpenReview ↗

Response (Part 2)

[W3][Q4] Analysis on cache-hit rates

Our method primarily reduces TTFT without making corresponding optimizations to TPOT, so TPOT remains similar to the baseline. We can report the cache hit rate in the online experiment as shown below.

MethodsCache hit rate (%)
Prefix caching5
CacheBlend94
EPIC94
COMB94

All PIC methods have the same cache hit rate, and the results demonstrate that the primary reason for the improvement in TTFT for PIC methods is that their cache hit rate is higher than that of prefix caching.

[W3][Q4] Ablation study on encoder depth

We ran ablation experiments on different encoder depths, keeping the training dataset and hyperparameters the same.

# of encoder layersHotpotQA score (%)MuSiQue score (%)2WikiMQA score (%)MultiNews score (%)
234.9717.2323.8719.89
443.3023.9834.5621.00
847.9830.7439.8621.79

We found that as the encoder depth increases, the model's learning capacity grows, and test scores steadily improve. When the number of encoder layers reaches 8, the score of the COMB model gets close to the original backbone model. Therefore, we decided to set the number of encoder layers to 8.

[W4][Q5] Implementation details

COMB uses an LRU eviction policy, and manages PICaches independent of vLLM. The PICaches are managed with a block size of 1, meaning that there is only one token per block. This allows us to handle variable-length KV token counts.

For more details, please refer to the anonymous repository linked in the abstract, especially the file comb/storage/pic_manager.py.

Overhead breakdown:

Encoder attentionEncoder MLPLinear projectionsCross-attentionCache management/transfer overhead through CUDA IPCDecoder
Latency (ms)321651620.04772

Thank you for the detailed response.

I accept the explanation on the reason of not including TurboRAG, but I think it would be worthwhile to include KVLink as baseline. I also appreciate for other experimental results and implementation details that I have required. However, my main concern regarding similarity to CEPE remains unresolved even with the authors' explanation. Different motivation does not ensures the novelty from a method with largely similar approach.

Therefore, I will keep my rating of 2.

Reviewer_ouYCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper's main contribution is to propose COMB, a plug-in encoder architecture that makes position-independent caching (PIC) native to decoder-only LLMs without changing the decoder part. It freezes the decoder, adds a lightweight trainable encoder whose layers are interleaved with decoder layers in a comb-like way. Then it only trains the encode. It divides user's input as documents and queris. Users' documents are independently encoded into PICaches, queries and generated tokens stay in the decoder. The authors also implement a PIC management system on Transformers and vLLM , with hash lookup, chunk processing, asynchronous cache storage. Evaluation result shows COMB can lower KV memory andand rerduces prefill cost with the given benchmark.

Critical strengths
  1. The core design of COMB is clear and directly targets a valuable trade-off, when compared post-training PIC and training-aware PIC. Post-training PIC is flexible but loses accuracy; training-aware PIC is accurate but intrusive. Freezing the decoder and training only the encoder plus query-side cross-attention is a reasonable way to combine the two, and because disabling the encoder and cross-attention restores the original decoder-only model, PIC becomes something that can be turned on or off rather than a permanent change to model behavior. And this method is a pluggable way for different trained models, though it needs further training.

  2. Author didn't stop at training the COMB-model. A further serving system was designed and implemented based on Transformers and vLLM. With this serving system, the paper can present more comprehensive inference performance evaluation

Critical weaknesses

Insufficient validation of position independence. The paper does not provide direct experimental evidence for the claimed “independence.” For example, how does performance change when multiple documents are provided in different orders or interleaved with one another? The authors should provide additional experimental results to demonstrate the effectiveness of the proposed method.

Questions for authors
  1. How will be the accuracy impact when multiple documents sequency change?
  2. How will the accuracy impact when documents interleaves?
  3. How to determine the number of encoder layers? Why it is 8 in this paper?
1 comment
AuthorsOpenReview ↗

Thank you for the thoughtful review

Thank you for the thoughtful review and for recognizing the importance of the serving problem, the empirical gains, and the architectural design of COMB. We would like to clarify the main concerns.

[Q1] Performance when documents are reordered

We randomized the document order in the test set and found that COMB’s scores remain very similar across different random seeds. The results are shown in the table below.

Random seedHotpotQA score (%)MuSiQue score (%)2WikiMQA score (%)MultiNews score (%)
2148.5431.1341.8621.75
4249.3430.6940.2621.94
6350.5031.1141.4021.97

[Q2] Scenarios where documents interleave

We first clarify the distinction between the two settings, as we understand them. Reordering means each document remains intact and contiguous, and only the order among documents changes (addressed in Q1). Interleaving means segments of different documents are mixed with one another (e.g., A1, B1, A2, B2), so a single document is no longer contiguous in the context.

COMB targets RAG, where retrieved documents (or chunks) are inserted as intact units, so interleaving within a document is not the typical pattern. Moreover, if interleaving does occur, each contiguous segment is simply treated as an independent chunk and encoded separately by COMB. This reduces interleaving to reordering at chunk granularity, which COMB natively supports since every chunk is encoded position-independently.

[Q3] Ablation study on encoder depth

We ran ablation experiments on different encoder depths, keeping the training dataset and hyperparameters the same.

# of encoder layersHotpotQA score (%)MuSiQue score (%)2WikiMQA score (%)MultiNews score (%)
234.9717.2323.8719.89
443.3023.9834.5621.00
847.9830.7439.8621.79

We found that as the encoder depth increases, the model's learning capacity grows, and test scores steadily improve. When the number of encoder layers reaches 8, the score of the COMB model gets close to the original backbone model. Therefore, we decided to set the number of encoder layers to 8.

Reviewer_W7uyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes COMB, an encoder plug-in that enables position-independent KV caching for decoder-only LLMs. COMB independently encodes documents and allows a frozen decoder to access their cached representations through cross-attention. The encoder is trained to preserve the original model's generation quality while supporting arbitrary document ordering. The authors implement COMB on top of existing inference frameworks and evaluate it using Llama-3.1-8B-Instruct and DeepSeek-V2-Lite-Chat. Experiments show comparable accuracy, 51-94% lower TTFT, up to 3x higher throughput, and 75-78% lower KV-cache memory usage on 5 LongBench tasks.

Critical strengths

Practical efficiency gains: COMB achieves improvements in throughput, TTFT, and KV-cache memory consumption.

Critical weaknesses

1.Limited novelty: The encoder-decoder architecture is close to prior work such as CEPE [1]; the paper does not sufficiently clarify its distinct technical novelty. 2.Unfair baseline comparisons: COMB uses substantially more trainable parameters than the baselines, making it difficult to attribute accuracy improvements to the architecture itself. 3.Missing ablations: The paper lacks systematic ablations on encoder depth, cross-attention placement, and parameter size to justify its architectural choices. 4.Limited evaluation: Experiments focus on LongBench QA and summarization tasks without evaluating realistic RAG or agent workloads involving dynamically changing documents and complex reasoning. 5.Incomplete serving analysis: The paper does not sufficiently analyze cache-hit rates, generation lengths, decode latency, and total GPU memory consumption under varying workloads. [1] Howard Yen, Tianyu Gao, Danqi Chen: Long-Context Language Modeling with Parallel Context Encoding. ACL 2024.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

We appreciate your valuable feedback. We have read the review carefully and supplemented the necessary clarifications following your recommendation. Here is our response to your questions.

[W1] Novelty compared to prior work

We agree that CEPE shares similarities with our paper. We will incorporate a discussion of CEPE in the Related Work section of our revised paper. However, we believe our work is fundamentally different for three reasons:

  1. Motivation and core concept: CEPE proposes parallel encoding, meaning that when processing very long inputs for prefill, it can parallelize the prefill phase. In contrast, our approach focuses on trading storage for computation, enabling existing KV caches to be reused at arbitrary positions (which we call PIC). Our starting point is entirely different.
  2. Methodological differences: CEPE employs a smaller hidden size and a shallower encoder depth. Furthermore, it only uses the final hidden states from the encoder's output for cross-attention, whereas we leverage the hidden states from every layer of the encoder for cross-attention. The ablation study on encoder depth demonstrates that their accuracy would be lower than ours due to their smaller model size.
  3. System implementation: We have built COMB, a PIC-aware caching system that integrates with existing inference frameworks. CEPE focuses solely on parallel encoding and lacks a storage system to manage its KV cache. As Reviewer ouYC noted in the review,

Authors didn't stop at training the COMB-model. A further serving system was designed and implemented based on Transformers and vLLM. With this serving system, the paper can present more comprehensive inference performance evaluation.

[W2] Controlled comparison with the baselines

We agree that increasing the baseline’s training budget to match our method would likely increase their accuracy. However, this misses the key distinction of our approach: it is non-intrusive.

Existing methods, regardless of their training scale or accuracy, fundamentally modify the model’s behavior in a persistent manner. In contrast, our method operates without altering the underlying model, preserving its original behavior while enabling the desired functionality.

This difference is orthogonal to training cost and cannot be captured by simply matching parameter count or GPU usage.

[W3] Ablation study on encoder depth

We ran ablation experiments on different encoder depths, keeping the training dataset and hyperparameters the same.

# of encoder layersHotpotQA score (%)MuSiQue score (%)2WikiMQA score (%)MultiNews score (%)
234.9717.2323.8719.89
443.3023.9834.5621.00
847.9830.7439.8621.79

We found that as the encoder depth increases, the model's learning capacity grows, and test scores steadily improve. When the number of encoder layers reaches 8, the score of the COMB model gets close to the original backbone model. Therefore, we decided to set the number of encoder layers to 8.

[W4] Limited evaluation (no realistic RAG / agent workloads)

We would like to clarify the scope of our work and the rationale behind our evaluation choices.

Scope. COMB targets RAG, where independently retrieved documents are reused across requests in varying combinations and orders. This is exactly the setting that calls for position-independent KV caching. Agentic workloads are also an important and emerging setting worth studying. However, their reuse pattern (multi-turn tool calls, iterative context updates) is quite different, and the representative works in this line, such as CacheBlend, EPIC, TurboRAG, and Block-Attention, have not yet addressed it either. We will explore it in future work.

Benchmarks. LongBench-style multi-document QA and summarization are the standard benchmarks for this line of work. Each query comes with its own set of independently sourced passages, most of them distractors. Multi-hop tasks such as 2WikiMQA, MuSiQue, and HotpotQA require reasoning across documents. Because every query in our experiment uses a different set of documents, the evaluation already covers dynamically changing document combinations. Representative prior works, such as CacheBlend (EuroSys'25, Best Paper), EPIC (ICML'25), TurboRAG (EMNLP'25), and Block-Attention (ICLR'25), use the same protocol.

Our evaluation follows this established practice and can be compared directly with these baselines. Asking for a different benchmark suite would hold COMB to a standard that none of the prior work in this area was held to. We will state the scope more explicitly in the revision.

[W5] Analysis on cache-hit rates

Our method primarily reduces TTFT without making corresponding optimizations to decode latency, so the decode latency remains similar to the baseline. We can report the cache hit rate in the online experiment as shown below.

MethodsCache hit rate (%)
Prefix caching5
CacheBlend94
EPIC94
COMB94

All PIC methods have the same cache hit rate, and the results demonstrate that the primary reason for the improvement in TTFT for PIC methods is that their cache hit rate is higher than that of prefix caching.

Thank you again for your careful review and constructive feedback. As the discussion period is coming to an end, we would be very grateful if you had an opportunity to review our responses and additional results. If any concern remains unresolved or any further clarification would be helpful, please let us know, and we will respond promptly before the discussion closes. We fully understand the time constraints and sincerely appreciate your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.