- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces COMB, a position-independent caching method that attaches a separate trainable encoder to a frozen decoder-only LLM. Document chunks are encoded independently by the encoder and then cached, and the decoder reads them through interleaved cross-attention layers. Only the encoder and cross-attention layers are trained, and the original model can be recovered by disabling them. The authors also integrated a PIC managements system on vLLM. On LongBench subtasks, COMB achieves 51-94% lower TTFT on cache hits, up to 3$\times$ higher throughput, while reducing 75% of KV memory.
Critical strengths
S1. Non-intrusive design.
Training only an added encoder and cross-attention layers on top of a frozen decoder lets COMB reduce exactly to the original model when disabled, which addresses the trade-off between post-training and training-aware PIC. Because documents are processed by a shallow encoder and stored in only a few layers, the architecture also reduces KV memory and encoder compute, supported by a complexity analysis.
S2. A complete, openly released implementation with transparent costs.
The paper provides a working PIC management system integrated with vLLM, anonymized code and trained models, and reports training cost and limitations, which makes the approach practical to reproduce and build on.
Critical weaknesses
W1. Related work and baseline coverage.
CEPE [1], which uses a small encoder for processing chunks, seem to be close prior work but not discussed. On the PIC side, BlockAttention is the only training-aware baseline, whereas KVLink [2] and TurboRAG [3], placed in the same category by the paper, are omitted.
The BlockAttention numbers (HotpotQA F1 of about 13 vs. 44 for prefix caching, as read from Figs. 4 and 6) are far below the parity with full attention reported in the original paper. The checkpoint and prompt format used are not described, and BlockAttention is excluded from the TTFT comparison, so “lowest TTFT among all approaches” is unverified against it.
W2. The evaluation does not match the agentic motivation.
The paper motivates PIC with RAG agentic and workloads and static content such as system prompts and few-shot examples, yet evaluates only single-turn LongBench tasks under a 100% cache-hit protocol with short outputs. Adding results on multi-turn agentic tasks with realistic hit rates, and instruction-following behavior when system prompts or instructions are routed through the encoder path would be helpful.
W3. Incomplete efficiency reporting and ablations.
Only TTFT and throughput are reported. TPOT and end-to-end latency are missing, although cross-attention layers add per-token compute. The hit rate used in the online experiment (Fig. 10) is not stated.
Also, encoder depth and cross-attention placement, which determine this accuracy–efficiency trade-off, are not ablated, although Section 3.3 suggests accuracy improves with more encoder layers.
W4. The system component is described only at a high level.
The paper does not describe cache eviction or tiering, how variable-length cross-attention KV is integrated with paged KV management and scheduling in vLLM, or an overhead breakdown. The memory and TTFT gains follow largely from the architecture rather than the system.
[1] Yen, Howard, Tianyu Gao, and Danqi Chen. "Long-context language modeling with parallel context encoding." Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
[2] Yang, Jingbo, et al. "Kvlink: Accelerating large language models via efficient kv cache reuse." Advances in Neural Information Processing Systems 38 (2026): 133797-133824.
[3] Lu, Songshuo, et al. "Turborag: Accelerating retrieval-augmented generation with precomputed kv caches for chunked text." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
Questions for authors
Q1. Can KVLink and, if feasible, TurboRAG, which the paper places in the same training-aware category, be compared under controlled settings? Can the training-free baselines (CacheBlend, EPIC) be evaluated at several recomputation budgets, with accuracy–TTFT curves, in the same serving engine and version?
Q2. Which BlockAttention checkpoint, base model, and prompt format were used? Please report results with the official model and its recommended input format, and its TTFT in the same serving environment.
Q3. How does COMB perform on multi-turn agentic tasks with realistic hit rates, including task success, end-to-end latency, and instruction-following behavior when system prompts are routed through the encoder?
Q4. What are TPOT and end-to-end latency across batch sizes and document lengths, and what is the net memory saving once the extra weights are counted? What hit rate was used in Fig. 10? How do accuracy-efficiency tradeoff changes with the number of encoder layers and the cross-attention placement?
Q5. Does the system implement eviction or tiering of PICaches, and how is variable-length cross-attention KV handled in vLLM’s paged KV and scheduler? Can an overhead breakdown be provided?
6 comments
Response (Part 2)
[W3][Q4] Analysis on cache-hit rates
Our method primarily reduces TTFT without making corresponding optimizations to TPOT, so TPOT remains similar to the baseline. We can report the cache hit rate in the online experiment as shown below.
| Methods | Cache hit rate (%) |
|---|---|
| Prefix caching | 5 |
| CacheBlend | 94 |
| EPIC | 94 |
| COMB | 94 |
All PIC methods have the same cache hit rate, and the results demonstrate that the primary reason for the improvement in TTFT for PIC methods is that their cache hit rate is higher than that of prefix caching.
[W3][Q4] Ablation study on encoder depth
We ran ablation experiments on different encoder depths, keeping the training dataset and hyperparameters the same.
| # of encoder layers | HotpotQA score (%) | MuSiQue score (%) | 2WikiMQA score (%) | MultiNews score (%) |
|---|---|---|---|---|
| 2 | 34.97 | 17.23 | 23.87 | 19.89 |
| 4 | 43.30 | 23.98 | 34.56 | 21.00 |
| 8 | 47.98 | 30.74 | 39.86 | 21.79 |
We found that as the encoder depth increases, the model's learning capacity grows, and test scores steadily improve. When the number of encoder layers reaches 8, the score of the COMB model gets close to the original backbone model. Therefore, we decided to set the number of encoder layers to 8.
[W4][Q5] Implementation details
COMB uses an LRU eviction policy, and manages PICaches independent of vLLM. The PICaches are managed with a block size of 1, meaning that there is only one token per block. This allows us to handle variable-length KV token counts.
For more details, please refer to the anonymous repository linked in the abstract, especially the file comb/storage/pic_manager.py.
Overhead breakdown:
| Encoder attention | Encoder MLP | Linear projections | Cross-attention | Cache management/transfer overhead through CUDA IPC | Decoder | |
|---|---|---|---|---|---|---|
| Latency (ms) | 32 | 165 | 16 | 2 | 0.047 | 72 |
Thank you for the detailed response.
I accept the explanation on the reason of not including TurboRAG, but I think it would be worthwhile to include KVLink as baseline. I also appreciate for other experimental results and implementation details that I have required. However, my main concern regarding similarity to CEPE remains unresolved even with the authors' explanation. Different motivation does not ensures the novelty from a method with largely similar approach.
Therefore, I will keep my rating of 2.
Response (Part 1)
Thank you for the review. We believe some of the concerns arise primarily from presentation clarity, and we will revise the paper to make the mechanism and scope more explicit.
[W1] Related work on parallel encoding
We agree that CEPE shares similarities with our paper. We will incorporate a discussion of CEPE in the Related Work section of our revised paper. However, we believe our work is fundamentally different for three reasons:
[Q1][W1] Comparison with the baselines
The KVLink paper didn't release its 8B model, only the 1B and 3B versions. Retraining a 8B model from scratch would require significant computational resources.
On the other hand, the method used by TurboRAG is almost identical to BlockAttention, just as the BlockAttention paper describes:
Therefore, we selected BlockAttention as a representative of training-aware PIC methods to illustrate that existing training-aware PIC methods are intrusive and permanently alter model parameters. Existing training-aware methods, regardless of their training scale or accuracy, fundamentally modify the model’s behavior in a persistent manner. In contrast, our method operates without altering the underlying model, preserving its original behavior while enabling the desired functionality.
[Q2][W1] Details on evaluating BlockAttention
We used the open-source checkpoint
ldsjmdy/Tulu3-Block-FTfrom the BlockAttention paper. The code we used was copied directly from their open-source repository; further details can be found in thebenchmarks/block_attentionfolder of the anonymous repository linked in our abstract.The experimental results on the HotpotQA dataset differ from the original paper because the original study used the OpenCompass dataset, whereas we used the LongBench dataset. HotpotQA is simply the task name, and the LongBench dataset is more challenging than OpenCompass, which is why the BlockAttention scores are lower in our paper.
[W2][Q3] Motivation on RAG
We want to clarify that in the Introduction section, we did not state that our motivation is handling agentic workloads. The scenario we target is strictly RAG. We would like to clarify the scope of our work and the rationale behind our evaluation choices.
Scope. COMB targets RAG, where independently retrieved documents are reused across requests in varying combinations and orders. This is exactly the setting that calls for position-independent KV caching. Agentic workloads are also an important and emerging setting worth studying. However, their reuse pattern (multi-turn tool calls, iterative context updates) is quite different, and the representative works in this line, such as CacheBlend, EPIC, TurboRAG, and Block-Attention, have not yet addressed it either. We will explore it in future work.
Benchmarks. LongBench-style multi-document QA and summarization are the standard benchmarks for this line of work. Representative prior works, such as CacheBlend (EuroSys'25, Best Paper), EPIC (ICML'25), TurboRAG (EMNLP'25), and Block-Attention (ICLR'25), use the same protocol.
Our evaluation follows this established practice and can be compared directly with these baselines. Asking for a different benchmark suite would hold COMB to a standard that none of the prior work in this area was held to. We will state the scope more explicitly in the revision.