acceptodds
Under review as a conference paper at ICLR 2027

Deferred Contextualization for Efficient Long-Context KV Reuse

Abstract

Shared-corpus long-context QA repeatedly queries the same document blocks. Per-query re-encoding is expensive and several cache reuse mechanisms have been proposed recently offering varying levels of quality-speed tradeoffs. We analyze this tradeoff through C4: four factors that govern whether reusable caches preserve multi-block reasoning quality: Compactness of relevant blocks relative to the instruction, Contiguity among them, Cross-attention across them, and Cleanliness, requiring each relevant block to be encoded without attending to unrelated predecessors. Through controlled experiments we show that each factor matters: accuracy drops when relevant blocks are far from the instruction or from one another; independently cached blocks lose cross-block interaction; and encodings contaminated by irrelevant blocks cannot be fully recovered by decode-time masking alone. Motivated by the C4 lens, we propose PRISM, a training-free KV reuse pipeline. PRISM encodes each block independently alongside the instruction, yielding clean reusable KV states, and reassigns relevant blocks to compact, contiguous positions at query time. PRISM establishes cross-block dependencies during query time through two operations: iterative span extraction, which conditions each block's extraction on the query and on spans extracted from preceding blocks; and in-place KV recomputation, which updates only the extracted span states with full cross-attention. Across long-context QA, claim verification, and synthetic graph traversal tasks, PRISM recovers much of the quality of per-query re-encoding while reducing time-to-first-token by nearly 5×, without fine-tuning or full re-encoding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache reuse, Multi-hop reasoning, Retrieval-augmented generation, Efficient LLM inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_AxXPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies efficient KV-cache reuse for long-context question answering over shared corpora. The challenge is that independently cached document blocks can be reused efficiently across queries, but their KV states lack the cross-block contextual information available when the retrieved blocks are jointly encoded. The paper proposes PRISM, a training-free framework for efficient KV-cache reuse. Experiments show that PRISM improves accuracy over existing KV-cache reuse methods while achieving substantially lower time-to-first-token (TTFT) than full re-encoding.

Critical strengths
  1. Efficient KV-cache reuse is an important problem for long-context LLM applications. The paper addresses a meaningful tradeoff between computational efficiency and reasoning accuracy.

  2. The C4 framework provides a useful perspective for analyzing cache-reuse methods, although the individual factors are largely known.

  3. Extensive empirical evaluation across models and tasks was conducted. The reported efficiency improvements are substantial.

Critical weaknesses
  1. The C4 framework is presented as a general characterization of the factors governing cache-reuse quality. However, the 4 factors are primarily motivated and validated through empirical observations. The 4 factors may not represent fundamentally independent mechanisms. The framework is essentially qualitative. It does not provide quantitative measures of 4 four factors or a predictive model connecting them to reasoning accuracy.

  2. The technical novelty looks limited. First, the main technical components build heavily on existing techniques. The contribution of the paper appears to lie in the integration of existing techniques, rather than in a fundamentally new caching or contextualization mechanism.

  3. The main technical content (Section 4) looks somewhat incomplete. The entire technical framework and many important technical details are missing. I think PRISM deserves a substantially more rigorous presentation. In particular, the conditional attention-based extraction procedure is not sufficiently specified. It is unclear how attention scores are computed between the conditioning context and independently cached blocks, how previously extracted spans are represented during subsequent extraction steps, and what additional computation is required.

Questions for authors
  1. What is the essential technical innovation distinguishing PRISM from a straightforward combination of evidence extraction and selective KV recomputation?

  2. How to determine whether the selected spans contain all information necessary for cross-block reasoning?

  3. How does the method behave when the reasoning process requires multiple rounds of information exchange between blocks?

6 comments

We thank the reviewer for finding the C4 framework a useful perspective. To clarify PRISM's contribution, we compared it with other methods that use query attention to select tokens for recomputation, and performed a more focused analysis to isolate the impact of iterative extraction. TL;DR from the additional experiments: PRISM's accuracy is higher by 14 to 56 points compared to all existing methods on the synthetic chain traversal task, where iterative extraction is important. Under matched budgets, PRISM is the best in 5 out of 6 settings on the real LongMemEval task. More details follow.

W2, Q1, Q2:

While attention-based selection and selective KV recomputation exist separately, a simple combination falls short across blocks. The closest "extraction + selective recomputation" methods are InfoFlow KV (2603.05353, ICML 2026) and ProphetKV (2602.02579, ICML 2026): like PRISM, they use query attention to select tokens, but they score tokens in a single pass over the assembled cache. PRISM instead scores each block's tokens conditioned on both the question and spans already selected from earlier blocks, replacing global top-k with per-block thresholds.

To see why conditioning across blocks matters, we first test on the chain traversal task, where chain links are spread across blocks and the model must follow them to the final destination (averaged over 500-, 1000-, and 2000-token blocks, n=100 per setting):

ModelHopsRe-encodeBlockSinkInfoFlowProphetKVPRISM, indep.PRISM
Llama-3.1-8B20.960.560.420.480.630.97
Llama-3.1-8B30.860.200.100.140.240.70
Qwen3-8B20.980.770.770.790.830.93
Qwen3-8B30.960.510.480.490.550.78
Qwen2.5-7B20.710.390.370.400.410.66
Qwen2.5-7B30.620.170.200.200.170.34

PRISM outperforms InfoFlow and ProphetKV in every setting at matched budget and comes closest to Re-encode. Scoring each block's tokens independently (PRISM, indep.) drops accuracy sharply, so spans from earlier blocks help select the right tokens in later ones.

On the full LongMemEval set, with new runs at PRISM's per-question budget:

ModelRetrieverAPECacheBlendEPICInfoFlowProphetKVPRISM
Llama-3.1-8Boracle0.5370.5650.5840.6510.6580.693
Llama-3.1-8BQRR0.4880.4770.4860.5810.5880.602
Qwen3-8Boracle0.3860.5190.5230.5740.5670.584
Qwen3-8BQRR0.3560.5070.4860.5420.5160.516
Qwen2.5-7Boracle0.4210.5490.5490.5560.5650.595
Qwen2.5-7BQRR0.3930.4810.4880.4930.4810.533

PRISM leads in 5 of 6 settings across models and retrievers. On multi-session questions needing facts from several sessions, iterative extraction is also ahead of independent scoring (0.476 vs 0.453 over five models, p=0.010).

To check that these gains come from better KV rewriting, we take LongMemEval samples that Re-encode answers correctly and measure the negative log-likelihood (NLL) of Re-encode's answer under each method. At matched budgets, we report the share of the NLL gap between BlockSink (0% recomputed) and Re-encode (100%) each method closes:

MethodLlama-3.1-8B (n = 182)Qwen3-8B (n = 159)
Random1%1%
CacheBlend25%7%
ProphetKV54%28%
InfoFlow67%47%
PRISM83%50%

Random tokens barely help, so what matters is which tokens are rewritten, and PRISM's selection closes the most.

To see the impact of span completeness (Q2), we compare PRISM with answering from the extracted spans alone (OIE) on LongMemEval (oracle), using Gemini 3.1 Pro to label whether the spans are complete:

ModelQuestions with complete spansComplete: OIE / PRISMIncomplete: OIE / PRISM
Llama-3.1-8B72%0.76 / 0.710.29 / 0.64
Qwen2.5-7B40%0.80 / 0.640.25 / 0.57
Qwen2.5-14B72%0.85 / 0.730.44 / 0.61
Qwen3-8B22%0.91 / 0.720.22 / 0.54
Qwen3-14B68%0.85 / 0.720.26 / 0.59

OIE performs well when spans are complete, but drops sharply when a fact is missing. PRISM is much more robust to incomplete spans (0.54–0.64 vs. 0.22–0.44), even on Qwen3-8B where only 22% of questions have complete spans. Since complete spans cannot be guaranteed, this robustness matters: over all questions, PRISM is ahead of OIE in 4 of 5 models.

> Together, these results show that neither half is enough alone: scoring the cache in one pass (InfoFlow KV, ProphetKV) picks less useful tokens, and answering from the spans alone (OIE) fails when a fact is missed. PRISM instead selects tokens iteratively across blocks and rewrites their KV in-place.

W1:

The factors in C4 come from controlled experiments that change one cache property at a time (Section 3). Each factor is a measurable property of the cache (Figure 2 now reports its accuracy cost on six models), which tells us what a reusable cache must preserve and led to PRISM's design. We will make this clear in Section 3, and explore predicting accuracy from these measurements in future work.

W3:

We will make Section 4 clearer in the revision (see also Algorithm 1 in Appendix A).

For each retrieved block BiB_i in order, the extraction context is ci=[I,qt,S1,…,Si−1]c_i = [I, q_t, S_1, \dots, S_{i-1}] (Algorithm 1): the instruction II is already cached, and the question and earlier spans are added as plain text (Figure 7). We run one forward pass of these tokens over the cached KV of II and BiB_i only, and score each 10-token chunk of BiB_i by the QRHead attention it receives from them, minus the attention it receives from the "Question:" prefix (following Chen et al., 2025; Zhang et al., 2025,). Chunks scoring above μi+λσi\mu_i + \lambda\sigma_i (λ=1\lambda = 1) are expanded to sentence/paragraph boundaries to form SiS_i.

Because BiB_i is already cached, extraction only adds one forward pass over cic_i per block: no tokens are generated and no new KV states are saved. This cost is included in the TTFT we measure, which is 2.2–6.8× lower than Re-encode's on LongMemEval and CLIPPER.

Q3:

PRISM extracts spans in a single pass over the retrieved blocks, so spans from earlier blocks guide selection in later ones. This handles multi-step reasoning that follows the block order (as in the 3-hop chains and multi-session questions above). To see what happens when links go backward or jump back and forth across blocks, we test chain traversal under reversed and shuffled block orders (500-token blocks, n=100):

ModelOrderHopsRe-encodeBlockSinkPRISM
Llama-3.1-8Bforward2/30.98/0.930.58/0.240.98/0.65
Llama-3.1-8Breversed2/30.87/0.710.59/0.350.83/0.53
Llama-3.1-8Bshuffled30.840.360.49
Qwen3-8Bforward2/30.99/0.970.81/0.650.98/0.92
Qwen3-8Breversed2/31.00/0.960.70/0.490.95/0.57
Qwen3-8Bshuffled30.970.530.77

PRISM still beats BlockSink in every setting, but accuracy drops on reversed and shuffled chains, as expected from a single pass. Extending extraction to multiple passes, so that spans found in later blocks can also guide earlier ones, is a natural next step.

Dear Reviewer,

With less than 24 hours remaining in the discussion period, we would appreciate your feedback on our responses, particularly on what PRISM adds beyond combining evidence extraction with selective recomputation, the span-completeness analysis, and the clarified extraction procedure. If these address your concerns, we kindly ask you to consider raising your rating. We would also be happy to discuss any remaining questions.

Thank you for your time and consideration.

Reviewer_8ATCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose PRISM, a training-free KV cache reuse pipeline that aims to approach the quality of per-query re-encoding for long-context QA over a shared corpus. The design is motivated by the C4 lens, four factors that the authors argue govern reuse quality: Compactness, Contiguity, Cross-attention, and Cleanliness. The pipeline consists of three components: (1) independent caching of each block with pre-RoPE keys, (2) query-time reassignment of retrieved blocks to contiguous positions near the instruction, and (3) deferred contextualization, which extracts query-relevant spans with retrieval heads, conditions each block's extraction on spans from earlier blocks, and recomputes KV states only for those span tokens. The authors evaluate on LongMemEval, CLIPPER, LaRA, and a synthetic chain traversal task with five 7B to 14B models. On LongMemEval with Llama-3.1-8B under oracle retrieval, PRISM is reported to reach 0.70 accuracy versus 0.74 for RE-ENCODE while reducing time-to-first-token (TTFT) from 1.60 s to 0.36 s.

Critical strengths

S1. The authors tackle an important problem in long-context inference, namely reusing precomputed KV caches across queries without losing cross-block reasoning, and analyze it through four factors that affect reuse quality.

S2. The authors propose a simple pipeline that outperforms the evaluated cache-reuse baselines on LongMemEval while substantially reducing TTFT compared to full re-encoding.

Critical weaknesses

W1. Insufficient novelty and justification of key contributions

The novelty of PRISM over existing KV cache reuse methods is unclear, and its key design choices are insufficiently justified. This is consequential because query-aware token selection and selective KV recomputation are presented as central contributions, yet closely related methods already address these problems.

W1.1. The proposed method substantially overlaps with existing methods. InfoFlow KV [1] and ProphetKV [2] already use query-conditioned attention to select tokens for selective KV recomputation and recover cross-chunk information flow. InfoFlow KV also reconstructs global positions under RoPE. Both methods are training-free. Appendix C names iterative, query-conditioned span extraction as PRISM's distinction, but since [1] and [2] are already query-conditioned, only the iterative conditioning across blocks remains, and its benefit is not established (W1.2). The authors should clarify what is fundamentally new beyond [1] and [2].

W1.2. The benefit of iterative span extraction is insufficiently established. PRISM uses retrieval heads from QRHead [3] and conditions each block's extraction on previously selected spans (Section 4). However, the only direct comparison with independent extraction is OIE versus OE (0.64 vs. 0.62, Table 2), using one model under oracle retrieval in an extraction-only setting. The authors should explain why iterative conditioning is necessary and evaluate it against independent query-aware selection within the full PRISM pipeline under matched recomputation budgets.

W1.3. The C4 factors are not adequately isolated. The paper claims controlled experiments isolating each factor, while noting only that some cannot be perfectly disentangled (Section 3). However, Figure 2b increases both the gap between relevant blocks and the second block's distance from the instruction, and its sharp drop beyond 30k tokens coincides with the Compactness drop in Figure 2a. This confounds Contiguity with Compactness. Similarly, Figure 2d changes both the irrelevant prefix and the absolute positions of relevant blocks, confounding Cleanliness with positional effects. The authors should clarify the framework's contribution beyond known long-context degradation phenomena and provide controls that isolate these factors.

W2. Insufficient experimental validation

The experiments do not sufficiently establish PRISM's advantages over existing methods or validate its key components. This is particularly important given the claimed improvements in reasoning quality and inference efficiency.

W2.1. Important baselines are missing. The main experiments exclude InfoFlow KV [1] and ProphetKV [2], which directly address query-aware selective KV recomputation. InfoFlow KV is cited but not evaluated, while ProphetKV and Cache-Craft [4] are not discussed. Moreover, recomputation budgets are matched only for EPIC in Table 2, rather than consistently across the main comparisons. The authors should compare against at least [1] and [2] under matched models, retrieved contexts, and recomputation or latency budgets.

W2.2. The ablations do not consistently support in-place KV recomputation. While Table 2 shows PRISM outperforming OIE and BSIE with Llama-3.1-8B, Appendix Table 5 shows that OIE, which answers from the extracted spans alone, outperforms PRISM across all three retrievers on Qwen2.5-14B (e.g., 0.69 vs. 0.60 with QRRetriever). BSIE, which appends the spans without recomputation, also matches or exceeds PRISM across all six Qwen2.5 settings, and Table 5 omits both Qwen3 models. Furthermore, conditional extraction is evaluated against independent extraction only in the extraction-only setting (W1.2). The authors should independently ablate conditional extraction and in-place recomputation within the full pipeline across all five models and retrieval settings.

W2.3. The synthetic evaluation favors PRISM's forward extraction order. In the synthetic chain traversal task, successive blocks contain successive edges of the reasoning chain (Appendix B.1), matching PRISM's forward iterative extraction (Algorithm 1). However, real cross-block dependencies need not follow document order. The authors should evaluate permuted and reversed chain orders to establish whether PRISM can recover dependencies under different evidence orderings.

W2.4. The efficiency advantage is not consistent across settings. Under CLIPPER with QRRetriever, PRISM has higher end-to-end latency than Re-encode for all five models (Appendix Table 4). For example, Qwen3-8B takes 13.41 seconds with PRISM versus 9.56 seconds with Re-encode, despite a modest TTFT reduction (1.04 to 0.64 seconds). Offline cache construction and storage costs are also excluded. The authors should report amortized costs and clarify when PRISM provides a meaningful end-to-end efficiency advantage. References

[1] Teng, Xin, et al. "InfoFlow KV: Information-Flow-Aware KV Recomputation for Long Context." ICML, 2026.

[2] Wang, Shihao, et al. "ProphetKV: User-Query-Driven Selective Recomputation for Efficient KV Cache Reuse in Retrieval-Augmented Generation." ICML, 2026.

[3] Zhang, Wuwei, et al. "Query-focused retrieval heads improve long-context reasoning and re-ranking." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

[4] Agarwal, Shubham, et al. "Cache-craft: Managing chunk-caches for efficient retrieval-augmented generation." Proceedings of the ACM on Management of Data 3.3 (2025): 1-28.

Questions for authors

Please refer to W1 and W2.

4 comments

We thank the reviewer for the detailed review.

W1.1, W1.2 and W2.1:

While InfoFlow KV, ProphetKV, and PRISM all use query attention to select tokens for recomputation, prior methods score tokens in a single pass over the assembled cache, often missing cross-block dependencies. PRISM instead scores each block's tokens conditioned on both the question and spans already selected from earlier blocks, replacing global top-k with per-block thresholds. We compare against both at matched budget.

First, on the chain traversal task, where each block has one or more links of the chain and the model must follow it to the final destination (averaged over 500-, 1000-, and 2000-token blocks, n=100 per setting):

ModelHopsRe-encodeBlockSinkInfoFlowProphetKVPRISM, indep.PRISM
Llama-3.1-8B20.960.560.420.480.630.97
Llama-3.1-8B30.860.200.100.140.240.70
Qwen3-8B20.980.770.770.790.830.93
Qwen3-8B30.960.510.480.490.550.78
Qwen2.5-7B20.710.390.370.400.410.66
Qwen2.5-7B30.620.170.200.200.170.34

PRISM outperforms InfoFlow and ProphetKV in every setting and comes closest to Re-encode. Scoring each block's tokens independently (PRISM, indep.) drops accuracy sharply, showing that spans from earlier blocks help select relevant tokens in later ones. On LongMemEval, iterative extraction is ahead on multi-session questions needing facts from several sessions (0.476 vs 0.453 over five models, p=0.010). Inside PRISM, scoring with all heads instead of the QRHeads, the per-block threshold keeps 2.5–7× fewer tokens, and accuracy drops by 3–12 points (significant in 5 of 6 settings).

Second, on the full LongMemEval set, with new runs at PRISM's per-question budget:

ModelRetrieverAPECacheBlendEPICInfoFlowProphetKVPRISM
Llama-3.1-8Boracle0.5370.5650.5840.6510.6580.693
Llama-3.1-8BQRR0.4880.4770.4860.5810.5880.602
Qwen3-8Boracle0.3860.5190.5230.5740.5670.584
Qwen3-8BQRR0.3560.5070.4860.5420.5160.516
Qwen2.5-7Boracle0.4210.5490.5490.5560.5650.595
Qwen2.5-7BQRR0.3930.4810.4880.4930.4810.533

PRISM leads in 5 of 6 settings across models and retrievers.

Pooled over these six settings (2,580 paired predictions, paired McNemar test), PRISM is significantly ahead of InfoFlow KV (p = 0.008) and ProphetKV (p = 0.002), as well as CacheBlend, EPIC and APE (p < 0.001). We will discuss Cache-Craft in the revised Related Work.

W1.3:

In Contiguity, positional gaps are created by skipping position ids with no tokens in between, whereas in Cleanliness, gaps are filled with irrelevant blocks. Following the reviewer's suggestion, we reran both with the distance between the relevant blocks and the instruction fixed (Llama-3.1-8B, as in Section 3; LongMemEval, n=100).

Contiguity: we fix the second block at position 50K and move only the first block to vary the gap. Accuracy drops from 75% (at gap 0) to 68%, 68% and 70% (at gaps 10k, 30k and 50k).

Cleanliness: we keep the relevant blocks at position ~55K and vary only whether the irrelevant blocks appear before them. Accuracy drops from 73% without irrelevant blocks to 59% with them.

Even with the distance fixed, the inter-block gap costs up to 7 points and the irrelevant blocks 14 points.

While these effects are known individually, C4 frames them as requirements for a reusable block cache, which PRISM is built to satisfy.

W2.2:

To isolate KV rewriting, we take LongMemEval samples that Re-encode answers correctly and measure the negative log-likelihood (NLL) of Re-encode's answer under each method. At matched budgets, we report the share of the NLL gap between BlockSink (0% recomputed) and Re-encode (100%) each method closes:

MethodLlama-3.1-8B (n = 182)Qwen3-8B (n = 159)
Random1%1%
CacheBlend25%7%
ProphetKV54%28%
InfoFlow67%47%
PRISM83%50%

Random tokens barely help, so gains come from token selection, not recomputation alone.

BSIE appends the same spans as text without recomputation. Across all five models and three retrievers on LongMemEval, PRISM matches or beats it in 12 of 15 settings (by up to 10 points), while BSIE leads only on Qwen2.5 (by at most 4).

We also compare PRISM with answering from extracted spans alone (OIE). Using Gemini 3.1 Pro, we label whether extracted spans are complete or miss a required fact (LongMemEval, oracle):

ModelQuestions with complete spansComplete: OIE / PRISMIncomplete: OIE / PRISM
Llama-3.1-8B72%0.76 / 0.710.29 / 0.64
Qwen2.5-7B40%0.80 / 0.640.25 / 0.57
Qwen2.5-14B72%0.85 / 0.730.44 / 0.61
Qwen3-8B22%0.91 / 0.720.22 / 0.54
Qwen3-14B68%0.85 / 0.720.26 / 0.59

OIE performs well when spans are complete, but drops sharply when a fact is missing. PRISM does much better, rewriting the spans while keeping every block’s full KV. OIE's overall accuracy thus tracks span completeness: ahead of PRISM on Qwen2.5-14B, and below BlockSink on Qwen3-8B (22% complete).

W2.3:

We test how reversing or shuffling the chain affects each method (500-token blocks, 100 chains each):

ModelOrderHopsRe-encodeBlockSinkPRISM
Llama-3.1-8Bforward2/30.98/0.930.58/0.240.98/0.65
Llama-3.1-8Breversed2/30.87/0.710.59/0.350.83/0.53
Llama-3.1-8Bshuffled30.840.360.49
Qwen3-8Bforward2/30.99/0.970.81/0.650.98/0.92
Qwen3-8Breversed2/31.00/0.960.70/0.490.95/0.57
Qwen3-8Bshuffled30.970.530.77

PRISM outperforms BlockSink across all orders, though accuracy drops when chain order differs from extraction order. Re-encode also drops in several settings, so part of the drop comes from the order itself.

W2.4:

We thank the reviewer for raising the efficiency point. To explain it properly, we revisited our pipeline and found three avoidable overheads:

  • KV recomputation built indices layer by layer in Python and copied the cache -> now built once on GPU and updated in place.
  • Cache positions were reassigned at every decoding step -> now set once before decoding.
  • PRISM built each block's token-to-text map at query time -> now done at caching time.

These changes do not affect outputs. We re-measured all methods in the same setup (HF Transformers, bf16, batch size 1, greedy decoding, one 96 GB RTX PRO 6000) on the same 49 random samples per dataset (averaged over the five Table 1 models):

SettingContextRecomputedTTFT (s), Re-encode/PRISMSpeed-up
LongMemEval, oracle6,446163 (2.5%)1.030/0.1805.7×
LongMemEval, QRR9,141196 (2.1%)1.681/0.2476.8×
CLIPPER, oracle7,1691,419 (19.8%)1.253/0.5202.4×
CLIPPER, QRR8,8921,777 (20.0%)1.639/0.7302.2×

TTFT includes PRISM's span extraction and KV recomputation, excluding block retrieval (same for all methods). PRISM's TTFT is 5.7–6.8× lower than Re-encode's on LongMemEval and 2.2–2.4× lower on CLIPPER. Both datasets have similar context lengths and Re-encode times.

TTFT/TT:

PRISM's TTFT gains depend on how much it recomputes: LongMemEval answers rely on a few chat turns (~2% recomputed), while CLIPPER claims span several chapters (~20%). Also, because CLIPPER answers are longer (~240 vs. ~95 tokens), decoding takes a larger share of total time: Re-encode takes 1.2–1.3× as long as PRISM in total time on LongMemEval, and 1.03–1.05× on CLIPPER.

PRISM's TTFT speed-up also grows with QRR-retrieved context length on both datasets (Llama-3.1-8B, Qwen2.5-7B, Qwen3-8B; 149 LongMemEval, 171 CLIPPER queries):

< 5k tokens5k–10k≥ 10k
LongMemEval2.9×5.9×8.5×
CLIPPER2.0×2.2×2.5×

Pre-encoding a block offline requires one prefill (about 0.2–0.6 s). Since PRISM then skips re-encoding the block at query time, this one-time cost pays for itself after the block is retrieved 1.1–1.9 times. The block caches take 6.7–22.9 GB per LongMemEval history, the same as a full-corpus cache. Every cache-reuse method pays this storage cost. We will report these costs in the paper.

Thanks for the additional experiments. The matched-budget comparisons, iterative extraction ablations, and controlled C4 experiments address several of my earlier concerns. I will keep my score at 2, however. While the iterative extraction results are encouraging, I still find the methodological novelty over InfoFlow KV and ProphetKV limited. For W2.2, I am still not fully convinced of the consistent benefit of in-place KV recomputation, particularly given the OIE and BSIE results on Qwen2.5. The sensitivity to evidence ordering is also concerning. On Qwen3-8B, reversing the 3-hop chain reduces PRISM's accuracy from 92% to 57%, while Re-encode remains nearly unchanged (97% to 96%). Finally, the reported total-time improvement on CLIPPER is only about 3–5% after optimization, and the amortized cost analysis remains limited. The additional results are helpful, but these concerns still prevent me from raising my score.

Thank you for the careful follow-up. Briefly, our new experiments show:

  1. PRISM outperforms InfoFlow KV and ProphetKV across all block orders, and running extraction in both directions adds +21–28 points on reversed chains while staying consistent on LongMemEval.
  2. In-place recomputation leads BSIE and OIE in 11 of 12 settings outside Qwen2.5 (p<0.001).
  3. The difference in KV states between independent caching and Re-encode explains why PRISM selects far fewer tokens on LongMemEval (~2.5%) than on CLIPPER (~20%), giving higher TTFT speedups on LongMemEval (5.7–6.8× vs. 2.2–2.4×).

Evidence order. Because each block is conditioned only on earlier blocks, a single pass can miss hops when a chain runs backward. To handle this, we add a backward extraction pass (from the last block to the first, conditioned on spans from later blocks) and recompute the combined spans from both passes. On 3-hop chains (500-token blocks, n=100):

ModelOrderRe-encodeBlockSinkInfoFlow KVProphetKVPRISMPRISM, bidir.
Llama-3.1-8Bforward0.930.240.130.190.650.63
Llama-3.1-8Breversed0.710.350.180.150.520.73
Llama-3.1-8Bshuffled0.840.360.180.210.490.59
Qwen3-8Bforward0.970.650.640.650.920.94
Qwen3-8Breversed0.960.490.530.480.570.85
Qwen3-8Bshuffled0.970.530.440.490.770.83

The backward pass recovers most of the gap on reversed chains (Qwen3-8B 0.57 → 0.85, Llama-3.1-8B 0.52 → 0.73; p<0.001 for both), leaves forward chains unchanged (0.92 → 0.94, 0.65 → 0.63), and costs one extra extraction pass (~1.4× recomputed tokens). On LongMemEval (oracle, n=430), accuracy is also unchanged (Llama-3.1-8B 0.69 vs. 0.69; Qwen3-8B 0.60 vs. 0.58, n.s.). Note that even without the backward pass, PRISM stays ahead of InfoFlow KV and ProphetKV in every order.

In-place recomputation (W2.2). The BSIE/OIE cases are specific to the Qwen2.5 series. On the other models (Llama-3.1-8B, Qwen3-8B, Qwen3-14B; oracle and QRR), PRISM is ahead of BSIE in all 6 settings and of OIE in 5 of 6 (pooled p<0.001 for both). On Qwen2.5, appending the spans (BSIE) works as well as rewriting them, while OIE is ahead only on Qwen2.5-14B when the extracted spans are complete (and behind PRISM on Qwen2.5-7B; ref. span-completeness table, W2.2).

Novelty. While PRISM shares query-attention selection and selective recomputation with InfoFlow KV and ProphetKV, the key difference is how tokens are selected across blocks: PRISM scores tokens per block, conditioned on spans from other blocks, with per-block thresholds instead of a single global top-k. This captures cross-block links that single-pass global selection misses: at the same budget, PRISM leads both by 14–56 points on forward chains and stays ahead on reversed and shuffled chains. Extending this conditioning to both directions adds another 6–28 points on reversed and shuffled chains, indicating that cross-block conditioning is what drives the gains.

Efficiency. To see why the speedup differs across datasets, we checked how much the KV states of independently cached blocks deviate from those under Re-encode, across tokens:

DatasetKV deviation in top 10% of tokens (Llama-3.1-8B, Qwen3-8B)Tokens recomputed by PRISMTTFT speedup vs. Re-encode
LongMemEval47–56%~2.5%5.7–6.8×
CLIPPER30–32%~20%2.2–2.4×

On LongMemEval, about half of the KV deviation is concentrated in the top 10% of tokens, and PRISM selects only ~2.5% of tokens. On CLIPPER, claims span across book chapters, so the deviation is more spread out and PRISM selects more tokens (~20%, giving a 2.2–2.4× TTFT speedup). Further, CLIPPER requires generating long explanations (~240 tokens), which dominates total time: even BlockSink, which recomputes nothing, is only 1.01–1.23× faster end-to-end than Re-encode.

Reviewer_SWECOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses KV cache reuse for long-context QA over a shared corpus, where the same document blocks are queried repeatedly. The paper analyzes the quality-speed tradeoff of cache reuse through C4, four factors that a reusable cache should satisfy (Compactness, Contiguity, Cross-attention, and Cleanliness). The paper proposes a training-free KV reuse pipeline, PRISM, which encodes each block independently in advance, places the retrieved blocks compactly and contiguously at query time, and recomputes the KV states of only the extracted query-relevant spans. Across long-context QA, claim verification, and synthetic graph traversal tasks, PRISM recovers much of the quality of per-query re-encoding while reducing time-to-first-token by nearly 5 times.

Critical strengths

This paper addresses an interesting and timely topic.

Critical weaknesses

(1) A comparison with EpiCache (ICML'26) [a] is needed (important). In addition, the related work should explain the relationship with KV eviction or compression methods.

(2) How is this work different from the line of cache-augmented generation (e.g., CAG [b]) methods? An explanation of the relationship with Cartridges (ICLR'26) [c] also seems to be needed.

(3) I think experiments under different KV cache memory budgets are needed (as in Figure 5 of EpiCache).

(4) Figure 1 is an important illustration for the paper. It could be elaborated so that readers can understand it and be convinced of the paper's claim.

Questions for authors

Please answer the following important questions, considering to the weaknesses above.

  • How does PRISM compare with EpiCache, and how does it relate to KV eviction and compression methods?
  • How does this work differ from cache-augmented generation (e.g., CAG) and Cartridges?
  • How do the results change under different KV cache memory budgets?
3 comments

We thank the reviewer for pointing us to EpiCache and related KV eviction work, which helped us position PRISM alongside KV eviction and compression methods. We compared PRISM with EpiCache at the same KV memory and across memory budgets. TL;DR from the additional experiments: at the same KV memory, PRISM is more accurate than EpiCache in every setting on LongMemEval, and with ~10K tokens it stays above EpiCache at every budget from 2K to 64K tokens, even without eviction (new Figure 5). More details follow.

Q1/W1:

EpiCache, like SnapKV and other eviction methods, compresses the KV cache of one long context to fit a memory budget: tokens are dropped, and nothing is recomputed. PRISM addresses a different problem: restoring the cross-block attention lost when blocks are cached independently. The two are complementary, since eviction could also compress PRISM's block caches. We will add this discussion to the Related Work.

We ran EpiCache with the authors' released code and settings (4 episodes, 8 medoids, Qwen3-Embedding-0.6B, head-wise selection, layer-wise allocation). We first reproduced their LongMemEval setting (~21K-token contexts, F1): with a KV budget of M = 8K tokens (the number of tokens whose KV is kept), EpiCache stays within 2 points of full KV (Llama-3.1-8B: 39.1 vs. 38.3; Qwen2.5-7B: 41.2 vs. 43.0), as in their Table A6.

We then ran EpiCache on the full LongMemEval set (~115K-token contexts, n=430) under PRISM's evaluation setup and judge. Without eviction, EpiCache retains the entire context (equivalent to our Full baseline). To compare with PRISM, we set EpiCache's KV budget M on each question to the total length of the retrieved blocks (~6.7K tokens under oracle, ~10.1K under QRR), i.e., matching PRISM's retrieved context length, not its recompute budget:

ModelEpiCache, no eviction (= Full)EpiCache, M = retrieved length (oracle / QRR)PRISM (oracle / QRR)
Llama-3.1-8B0.410.14 / 0.140.69 / 0.60
Qwen3-8B0.420.23 / 0.250.58 / 0.52
Qwen2.5-7B0.450.27 / 0.310.60 / 0.53

PRISM is ahead in every setting (paired McNemar, p<0.001). At this matched KV budget, EpiCache must discard over 90% of the 115K context, whereas PRISM keeps the retrieved blocks whole. Increasing EpiCache's budget M improves its accuracy, as shown below.

Q3/W3:

As the reviewer suggested (cf. Figure 5 of the EpiCache paper), we also vary the KV budget of both methods, shown in the new Figure 5 of our revised paper: M from 2K to 64K tokens for EpiCache, and the number of retrieved blocks K for PRISM. EpiCache improves with M, from 0.13–0.30 at 8K to 0.34–0.44 at 64K tokens, approaching its no-eviction accuracy (0.41–0.45). PRISM reaches 0.52–0.60 with five blocks (~10K tokens), above EpiCache at every budget, including no eviction.

Q2/W2:

CAG preloads the entire knowledge base into the context, caches its KV once, and answers every query from that cache without retrieval. This corresponds to the full-context reuse setting in Section 2, i.e., our Full baseline in Table 1. PRISM shares CAG's premise of computing KV states once offline and reusing them, but caches each block separately and loads only the retrieved blocks at query time. On LongMemEval, this holds 10–17× less KV in GPU memory (0.4–2.0 GB vs. 6.7–22.9 GB for the full ~115K-token context) and achieves higher accuracy (0.52–0.70 vs. 0.41–0.53 in Table 1), as the evidence is not diluted by unrelated blocks. A block cache is also easy to update: adding or removing a session only affects that block, whereas a CAG cache must be re-encoded.

Cartridges also build a reusable KV state offline, but learn it: a compact cache is trained for each corpus via self-study (synthetic conversations and context distillation). This requires gradient optimization per corpus, and retraining whenever the corpus changes. PRISM is training-free: each block is encoded in a single forward pass, blocks can be added or removed independently, and both the retrieved blocks and recomputed tokens are selected dynamically per query. The two methods thus suit different settings: Cartridges fit a static corpus queried many times, whereas PRISM fits an evolving collection such as a growing chat history. Because Cartridges require training a cache for each corpus, which is outside our training-free setting, we do not compare accuracy directly; we will discuss both methods in the Related Work.

W4:

We thank the reviewer for this suggestion. We will update Figure 1 so that it can be read on its own, with a caption that walks through each pipeline and states which C4 factor it violates and why.

Dear Reviewer,

With less than 24 hours remaining in the discussion period, we would appreciate your feedback on our responses, particularly the new comparison with EpiCache (new Figure 5) and the discussion of CAG and Cartridges. If these address your concerns, we kindly ask you to consider raising your rating. We would also be happy to discuss any remaining questions.

Thank you for your time and consideration.

Dear Reviewer,

As the discussion period is nearing its end, we wanted to follow up and ask whether you have had a chance to read our response and revised manuscript. Please let us know if any questions remain that we can clarify before the deadline, and if our updates address your concerns, we would be grateful if you could reconsider your evaluation. We appreciate your time and feedback.

Reviewer_7tZnOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

The paper studies KV-cache reuse for shared-corpus long-context QA, where there is a trade-off between the accuracy of re-encoding the retrieved blocks for every query and the speed of reusing precomputed caches. To better understand this trade-off, the manuscript introduces the C4 framework, which uses four factors (Compactness, Contiguity, Cross-attention and Cleanliness) to characterize when reused caches preserve multi-block reasoning quality. Based on this analysis, it proposes PRISM, a training-free pipeline that uses query-guided span extraction and in-place KV recomputation to restore cross-block context in independently cached blocks. The C4 factors are examined with controlled experiments on LongMemEval using four Llama models, and PRISM is evaluated on LongMemEval, CLIPPER and LaRA with five 7B-14B models, where, according to the manuscript, it recovers much of the accuracy of per-query re-encoding, with time-to-first-token (TTFT) speedups approaching 5x on LongMemEval.

Critical strengths
  1. The paper provides an informative diagnostic of the cross-block interaction gap caused by independent caching. Figure 2c holds the evidence and positional layout fixed while varying cross-block attention: with Llama-3.1-8B, Vanilla achieves 70% accuracy, compared with 20% for Block and 25% for BlockSink. The synthetic chain-traversal probe provides complementary evidence, with BlockSink accuracy falling from 54-58% at two hops to 11-24% at three hops. The paper also includes a useful comparison at what it describes as a matched recomputation-token budget: EPIC with 215 tokens achieves 0.61, compared with 0.70 for PRISM (Table 2). Together with the extraction-only, Snippet, Steer and BSIE variants, these experiments provide useful comparisons of evidence selection and cache-repair strategies, particularly in the Llama-3.1-8B setting.

  2. PRISM has a potentially practical training-free design, requiring neither model-weight updates nor learned bridge tokens. The main accuracy evaluation covers five 7B-14B models across LongMemEval, CLIPPER and LaRA, with latency results on LongMemEval and CLIPPER and additional retrieval, parameter-sensitivity and head-transfer analyses. On LongMemEval under oracle retrieval, the five main models show a reported 4.4-5.0x TTFT speedup over Re-encode (Table 3). For Llama-3.1-8B, Qwen2.5-14B and Qwen3-14B, PRISM closes 72-85% of the BlockSink-to-Re-encode accuracy gap in the same setting (Table 1). These results suggest a potentially useful accuracy-latency trade-off on LongMemEval.

Critical weaknesses
  1. The contribution beyond existing work is not clearly established. First, the C4 framework mostly relabels known effects. Three of its four factors, including Compactness, Contiguity and Cleanliness, describe well-known long-context problems, such as sensitivity to position and to irrelevant context, and the paper's Related Work says that they are "motivated" by prior studies. The fourth, Cross-attention, is the problem that CacheBlend, Block-Attention and KVLink were built to solve. The paper says these factors "govern" reuse quality but provides insufficient evidence that they predict which method performs better. Second, most components of PRISM already exist. CacheBlend (EuroSys 2025) already caches blocks independently, moves them to new positions and recomputes a subset of tokens. APE (ICLR 2025, not cited) already encodes blocks independently behind a shared prefix, which is the paper's "offline block encoding." Figure 1 suggests that no existing method satisfies all four factors, but only because it leaves out CacheBlend and EPIC. Choosing which tokens to recompute by query attention is not new either: InfoFlow KV does this without training and on two of the same models. The paper cites InfoFlow KV but does not compare against it. What is left is mainly the iterative step, in which each block's extraction depends on spans already extracted from earlier blocks. Its only support is an extraction-only comparison of 0.64 vs. 0.62 (Table 2), a gap that the authors' own McNemar test finds not significant in the same setting (Table 8). Apart from EPIC, no method is compared at a matched recomputation budget.

  2. Several headline claims are not supported by the paper's evaluation results. First, the speed claims are overstated. The "nearly 5x" TTFT reduction holds only on LongMemEval; on CLIPPER it is 1.3-1.8x (Table 4). The text says PRISM "has lower TTFT than CacheBlend for every model", but on CLIPPER CacheBlend has lower TTFT in all 10 model-retriever settings, and PRISM's total time is higher than Re-encode's in 9 of 10. The paper also does not report token counts, the inference framework or the batch size, so the timings are hard to interpret. Second, the accuracy claims are overstated. PRISM is up to 11 points less accurate than Re-encode, and in 16 of the 20 LongMemEval/CLIPPER settings it beats the best other cache-reuse method by 3 points or less, with no significance tests (Table 1). On LaRA it is the best cache-reuse method for two of five models, not "three of five" as stated, because the text does not count Full, one of the paper's own baselines. The Table 7 caption says the drop at larger k is due to distractors "rather than PRISM specifically," but PRISM loses 17 points while Re-encode loses 9. Third, the claim that cached states "must be rewritten" rests on one Llama-3.1-8B result (Table 2). In the full ablation (Table 5), PRISM is best in only 5 of 12 settings, all on Llama models, and for Qwen2.5-14B the extracted spans alone, with no recomputation, beat PRISM under all three retrievers. Finally, the C4 experiments in Figure 2 report no sample sizes and use only Llama models, although four of the five evaluation models are Qwen. In the Contiguity and Cleanliness experiments, the relevant blocks also move farther from the instruction, so these two effects cannot be separated from Compactness.

Questions for authors
  1. InfoFlow KV (ICML 2026) also chooses tokens to recompute by query attention without training. How does PRISM compare with InfoFlow KV, CacheBlend and APE at the same recomputation budget? The only evidence for the iterative step is an extraction-only gap of 0.64 vs. 0.62 (Table 2), so how does full PRISM perform with non-iterative extraction, or with plain query-attention scoring in place of QRHead? (For the authors' awareness only, and not part of my rating: the arXiv preprints A3 (2511.17560), ProphetKV (2602.02579), QCFuse (2606.05875) and InfiniRetri (2502.12962) use similar ideas and are worth discussing.)

  2. According to Table 8, 1-2-point changes are not significant, yet PRISM's margin over the best other cache-reuse method is 3 points or less in 16 of 20 settings. How many questions are behind each table cell and each point in Figure 2, and are the main gains significant under a paired test? Could you also add the Llama-3.3-70B results that the Figure 2 caption mentions?

  3. In the Contiguity and Cleanliness experiments, the relevant blocks also move farther from the instruction. What fills the positional gaps? Could both experiments be rerun with that distance held fixed, for example by moving cleanly encoded blocks with pre-RoPE reassignment?

  4. The reported latencies are hard to reconcile. Re-encode's TTFT for Llama-3.1-8B is 1.60 s on LongMemEval but 0.24 s on CLIPPER despite similar block sizes, and PRISM's total time on CLIPPER exceeds Re-encode's in 9 of 10 settings. Which inference framework, batch size and prefilled token counts were used? Does TTFT include retrieval and the extraction passes? How many tokens do PRISM, CacheBlend and EPIC each recompute per query?

  5. Table 5 shows that extracted spans alone (OIE) beat PRISM for Qwen2.5-14B under all three retrievers, and that appending spans without recomputation (BSIE) ties or beats it for Qwen2.5-7B. What evidence shows that rewriting KV states is what drives the gains? On the synthetic task, how does PRISM perform when the chain order is reversed or shuffled rather than following its left-to-right extraction order?

  6. Figure 6 reads like a generation prompt, whereas Section 4 describes non-generative attention scoring. How exactly are the earlier spans encoded and recomputed during extraction? Since the QRHead heads were selected on LongMemEval itself, do the head-selection data overlap the evaluation questions?

5 comments

We thanks the reviewer for identifying that PRISM presents a potentially practical training-free design with iterative retrieval as its key innovation over prior work, and reporting some additional missing related work. We compared with them all and performed a more focused analysis to isolate the impact of iterative retrieval. TL;DR from the additional experiments: PRISM's accuracy is higher by 14 to 56 points compared to all existing methods on synthetic chain traversal task where iterative retrieval is important. Under budget constraint setting, PRISM is the best in 5 out of 6 settings on the real LME task. More details follow.

Q1 (W1)

While InfoFlow KV, ProphetKV, and PRISM all use query attention to select tokens, prior methods score tokens in a single pass, often missing cross-block dependencies. PRISM instead scores each block's tokens conditioned on previous spans, and replaces global top-k with per-block thresholds. We compare against both at matched budget.

First, on the chain traversal task, where each block has one or more links of the chain and the model must follow it to the final destination (averaged over 500-, 1000-, and 2000-token blocks, n=100 per setting):

ModelHopsRe-encodeBlockSinkInfoFlowProphetKVPRISM, indep.PRISM
Llama-3.1-8B20.960.560.420.480.630.97
Llama-3.1-8B30.860.200.100.140.240.70
Qwen3-8B20.980.770.770.790.830.93
Qwen3-8B30.960.510.480.490.550.78
Qwen2.5-7B20.710.390.370.400.410.66
Qwen2.5-7B30.620.170.200.200.170.34

PRISM outperforms InfoFlow and ProphetKV in every setting and comes closest to our skyline Re-encode. Scoring each block's tokens independently (PRISM, indep.) drops accuracy sharply, showing that spans from earlier blocks help select relevant tokens in later ones. On LME, iterative extraction is ahead on multi-session questions needing facts from several sessions (0.476 vs 0.453 over five models, p=0.010). Inside PRISM, scoring with all heads instead of the QRHeads, the per-block threshold keeps 2.5–7× fewer tokens, and accuracy drops by 3–12 points (significant in 5 of 6 settings).

Second, on the full LME set, with new runs at PRISM's per-question budget:

ModelRetrieverAPECacheBlendEPICInfoFlowProphetKVPRISM
Llama-3.1-8Boracle0.5370.5650.5840.6510.6580.693
Llama-3.1-8BQRR0.4880.4770.4860.5810.5880.602
Qwen3-8Boracle0.3860.5190.5230.5740.5670.584
Qwen3-8BQRR0.3560.5070.4860.5420.5160.516
Qwen2.5-7Boracle0.4210.5490.5490.5560.5650.595
Qwen2.5-7BQRR0.3930.4810.4880.4930.4810.533

Q2 (W2.2, W2.4)

We report results on full LME (n=430), 300 paired questions from CLIPPER (600 total), and the long-context (128K) subset of LaRA's Reasoning task (n=53), covering financial reports and academic papers of 90–110K tokens; we will state these counts in Section 5.1.

We run paired McNemar test on LME to compare PRISM against other methods (Table 8). CacheBlend recomputes ~10× more tokens than PRISM; we also evaluate at matched budgets (as in Q1). We pool 2,580 pairs across all six settings (oracle and QRR across Llama-3.1-8B, Qwen3-8B, and Qwen2.5-7B) and report mean accuracy:

PRISM vsTokens recomputedAccuracy, PRISM / methodp
CacheBlend, 15% (Table 1)~1,260 (PRISM ~130)0.587 / 0.560<0.001
CacheBlend= PRISM0.587 / 0.516<0.001
EPIC= PRISM0.587 / 0.519<0.001
InfoFlow KV= PRISM0.587 / 0.5660.008
ProphetKV= PRISM0.587 / 0.5630.002
APEnone0.587 / 0.430<0.001

PRISM is significantly ahead of every method, including CacheBlend at 10× its budget. Over all five models in Table 1, PRISM's lead over CacheBlend is also significant for both retrievers (p<0.001 each).

For Figure 2, we extend the analysis to all five evaluation models and Llama-3.3-70B (LME, n=50); Cross-attention and Cleanliness hold across all models, while Compactness and Contiguity effects are strongest for Llama.

Q3 (W2.4)

In Contiguity, positional gaps are created by skipping position ids with no tokens in between, whereas in Cleanliness, gaps are filled with irrelevant blocks. Following your suggestion, we reran both with the distance between the relevant blocks and the instruction fixed.

Contiguity: we fix the second block at position 50K and move only the first block to vary the gap. Accuracy drops from 75% (at gap 0) to 68%, 68% and 70% (at gaps 10k, 30k and 50k).

Cleanliness: we keep the relevant blocks at position ~55K and vary only whether the irrelevant blocks appear before them. Accuracy drops from 73% without irrelevant blocks to 59% with them.

Even with the distance fixed, the inter-block gap costs up to 7 points and the irrelevant blocks 14 points.

While these effects are known individually, C4 frames them as requirements for a reusable block cache, which PRISM is built to satisfy.

Q4 (W2.1)

We thank the reviewer for raising the latency point. To explain it properly, we revisited our pipeline and found three avoidable overheads:

  • KV recomputation built indices layer by layer in Python and copied the cache -> now built once on GPU and updated in place.
  • Cache positions were reassigned at every decoding step -> now set once before decoding.
  • PRISM built each block's token-to-text map at query time -> now done at caching time.

These changes do not affect outputs. We re-measured all methods in the same setup (HF Transformers, bf16, batch size 1, greedy decoding, one 96 GB RTX PRO 6000) on the same 49 random samples per dataset (averaged over the five Table 1 models):

SettingContextRecomputedTTFT (s), Re-encode/PRISMSpeed-up
LongMemEval, oracle6,446163 (2.5%)1.030/0.1805.7×
LongMemEval, QRR9,141196 (2.1%)1.681/0.2476.8×
CLIPPER, oracle7,1691,419 (19.8%)1.253/0.5202.4×
CLIPPER, QRR8,8921,777 (20.0%)1.639/0.7302.2×

TTFT includes PRISM's span extraction and KV recomputation, excluding block retrieval (same for all methods). PRISM's TTFT is 5.7–6.8× lower than Re-encode's on LongMemEval and 2.2–2.4× lower on CLIPPER. Both datasets have similar context lengths and Re-encode times. The 1.60 s vs. 0.24 s gap in the submission came from the small probe sets, where CLIPPER's oracle prompts were much shorter (~3.4K vs. ~11.6K tokens).

TTFT/TT:

PRISM's TTFT gains depend on how much it recomputes: LongMemEval answers rely on a few chat turns (~2% recomputed), while CLIPPER claims span several chapters (~20%). Also, because CLIPPER answers are longer (~240 vs. ~95 tokens), decoding takes a larger share of total time: Re-encode takes 1.2–1.3× as long as PRISM in total time on LongMemEval, and 1.03–1.05× on CLIPPER.

PRISM's TTFT speed-up also grows with QRR-retrieved context length on both datasets (Llama-3.1-8B, Qwen2.5-7B, Qwen3-8B; 149 LongMemEval, 171 CLIPPER queries):

< 5k tokens5k–10k≥ 10k
LongMemEval2.9×5.9×8.5×
CLIPPER2.0×2.2×2.5×

Finally, CacheBlend recomputes a fixed 15% of context (~950–1,350 tokens here), and EPIC the first 32 tokens per block. On LongMemEval, PRISM recomputes 6–7× fewer tokens than CacheBlend and is faster and more accurate. On CLIPPER it recomputes more, so CacheBlend reaches the first token sooner at equal or lower accuracy; we will correct this in the paper.

Q5 (W2.3)

>To isolate KV rewriting, we take LongMemEval samples that Re-encode answers correctly and measure the negative log-likelihood (NLL) of Re-encode's answer under each method. At matched budgets, we report the share of the NLL gap between BlockSink (0% recomputed) and Re-encode (100%) each method closes:

MethodLlama-3.1-8B (n = 182)Qwen3-8B (n = 159)
Random1%1%
CacheBlend25%7%
ProphetKV54%28%
InfoFlow67%47%
PRISM83%50%

Random tokens barely help, so gains come from token selection, not recomputation alone.

>We also compare PRISM with answering from extracted spans alone (OIE). Using Gemini 3.1 Pro, we label whether extracted spans are complete or miss a required fact (LongMemEval, oracle):

ModelQuestions with complete spansComplete: OIE / PRISMIncomplete: OIE / PRISM
Llama-3.1-8B72%0.76 / 0.710.29 / 0.64
Qwen2.5-7B40%0.80 / 0.640.25 / 0.57
Qwen2.5-14B72%0.85 / 0.730.44 / 0.61
Qwen3-8B22%0.91 / 0.720.22 / 0.54
Qwen3-14B68%0.85 / 0.720.26 / 0.59

OIE performs well when spans are complete, but drops sharply when a fact is missing. PRISM does much better, rewriting the spans while keeping every block’s full KV. OIE's overall accuracy thus tracks span completeness: ahead of PRISM on Qwen2.5-14B, and below BlockSink on Qwen3-8B (22% complete).

>Chain order: We test how reversing or shuffling the chain affects each method (500-token blocks, 100 chains each):

ModelOrderHopsRe-encodeBlockSinkPRISM
Llama-3.1-8Bforward2/30.98/0.930.58/0.240.98/0.65
Llama-3.1-8Breversed2/30.87/0.710.59/0.350.83/0.53
Llama-3.1-8Bshuffled30.840.360.49
Qwen3-8Bforward2/30.99/0.970.81/0.650.98/0.92
Qwen3-8Breversed2/31.00/0.960.70/0.490.95/0.57
Qwen3-8Bshuffled30.970.530.77

PRISM outperforms BlockSink across all orders, though accuracy drops when chain order differs from extraction order. Re-encode also drops in several settings, so part of the drop comes from the order itself.

Q6

The Figure 6 template is used only for extraction (no text is generated). For each block, we append the template (with earlier blocks' spans as plain text) to the block's KV and run a forward pass. We then use QRHead to compute attention from template tokens to block tokens and select spans above the threshold. QRHead heads are selected on 70 held-out LongMemEval questions with no overlap with the 430 evaluation questions.

Dear Reviewer,

With less than 24 hours remaining in the discussion period, we would appreciate your feedback on our responses, particularly the new matched-budget comparisons with InfoFlow KV, ProphetKV, CacheBlend and APE, the paired significance tests, and the latency measurements. If these address your concerns, we kindly ask you to consider raising your rating. We would also be happy to discuss any remaining questions.

Thank you for your time and consideration.

Comments on the paper

3 comments

Dear Authors,

Please note that the author-reviewer discussion period is ending soon, but no rebuttal has been submitted yet We encourage you to respond to the reviewers' comments and address their key concerns before the discussion period closes.

AC

AuthorsOpenReview ↗

Additional comparison: training-free vs. training-based adaptation

To complement our training-free baselines, we compare PRISM with Attuner [1], a concurrent ICLR 2027 submission that addresses the same problem by training a LoRA adapter on each model's query projections. PRISM requires no training but significantly outperforms Attuner in all six model and retriever settings on LongMemEval (our reimplementation; details and significance tests in Appendix B.3):

ModelRetrieverAttunerPRISMRe-encode
Llama-3.1-8Boracle0.530.690.74
Llama-3.1-8BQRR0.490.600.64
Qwen3-8Boracle0.520.580.67
Qwen3-8BQRR0.440.520.63
Qwen2.5-7Boracle0.470.600.65
Qwen2.5-7BQRR0.430.530.60

On CLIPPER, the results are mixed: PRISM is clearly ahead on Llama, while the trained adapter gives Attuner the edge on Qwen (more details in Appendix B.3).

[1] ATTUNER: Recomputation-Free KV Cache Reuse via Query-Side Adaptation. ICLR 2027 submission. https://openreview.net/forum?id=l5NfZoPKFW

AuthorsOpenReview ↗

Rebuttal summary

Dear Area Chair,

Below is a summary of the main review points and our rebuttal:

  1. Comparison with InfoFlow KV and ProphetKV. Both methods score the assembled cache in a single pass, whereas PRISM selects tokens block by block with per-block thresholds, conditioned on spans extracted from other blocks. On LongMemEval at matched budget, PRISM is significantly ahead of both (pooled over six settings, p = 0.008 and p = 0.002), and ahead of CacheBlend even when CacheBlend recomputes ~10× more tokens (Tables 3 and 4). On multi-hop chain traversal at the same budget, PRISM leads InfoFlow KV and ProphetKV by 14–56 points on forward chains, while scoring blocks independently (PRISM, indep.) loses most of this gain. Running the conditioning in both directions (PRISM, bidir.) adds 6–28 points on reversed and shuffled chains with consistent accuracy on LongMemEval.

  2. In-place recomputation. Rewriting the selected spans in place closes 50–83% of the NLL gap to Re-encode, versus 1% for random tokens. Against appending spans as text (BSIE) or answering from spans alone (OIE), PRISM is ahead in 11 of 12 settings outside Qwen2.5 (p < 0.001). Even where OIE is competitive, it relies on complete spans and drops sharply when a fact is missed (0.22–0.44), whereas PRISM keeps every block's full KV (0.54–0.64).

  3. C4 factors. While these effects are known individually, C4 frames them as requirements for a reusable block cache: each factor is a measurable property of the cache that tells us what independent caching must preserve, which led to PRISM's design. Following the reviewers' suggestions, we isolated Contiguity and Cleanliness with the distance to the instruction fixed (costs of up to 7 and 14 points), and extended the analysis in Figure 2 to six models, including Llama-3.3-70B.

  4. Efficiency. After removing implementation overheads, PRISM cuts TTFT by 5.7–6.8× on LongMemEval and 2.2–2.4× on CLIPPER. The speedup differs because KV differences from Re-encode are concentrated in a few tokens on LongMemEval (~2.5% recomputed), but spread across chapters on CLIPPER (~20% recomputed). On CLIPPER, long explanations (~240 tokens) dominate total time, so even zero-recomputation BlockSink is only 1.01–1.23× faster end-to-end; TTFT isolates the prefill cost PRISM targets.

  5. KV eviction and compression. PRISM beats EpiCache across all KV memory budgets, from matched memory (p < 0.001) all the way to keeping the full context with no eviction (LongMemEval, Figure 5).

  6. Training-free vs. training-based adaptation. We also compared PRISM with Attuner, a concurrent ICLR 2027 submission that addresses the same problem by training a LoRA adapter for each model. We reimplemented Attuner and reproduced its gains on its own setting. Although PRISM requires no training, it significantly outperforms Attuner in all six model and retriever settings on LongMemEval (p ≤ 0.012), and in 8 of 12 settings overall (Appendix B.3).

Thanks

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.