acceptodds
Under review as a conference paper at ICLR 2027

Fast and Quality-Preserving Non-prefix KV Cache Reuse for Large Language Model Serving

Abstract

Key-Value (KV) Cache reuse reduces prefill computation for long-context large language models (LLMs), but reusing independently encoded text chunks in non-prefix contexts degrades generation quality because cross-chunk attention is missing. Selective recomputation can mitigate this problem, but it requires both selecting effective tokens for recomputation and preventing the costs of recomputation and cache loading from offsetting the benefits of reuse. We observe that the frequency-domain structure of KV provides an effective signal for identifying tokens that help restore cross-chunk dependencies. Based on this observation, we propose CacheTune, a frequency-guided method for non-prefix KV Cache reuse. CacheTune analyzes independently encoded KV Chunks offline to generate recomputation indices shared across layers, recomputes the selected KVs in the new context, and reuses the remaining cache, without additional training or online importance analysis. The same indices also determine which KVs must be loaded, so the volume of cache transfers decreases as the recomputation ratio increases, while allowing transfer to overlap with computation. CacheTune further calibrates the recomputation ratio based on hardware computation costs and cache access costs to accommodate different storage tiers. Across mainstream models and long-context tasks, CacheTune achieves a – speedup in time-to-first-token (TTFT) and – the throughput of full recomputation while maintaining comparable generation quality. With SSD/HDD caches, the TTFT speedup still reaches –. The code is available at https://anonymous.4open.science/r/CacheTune-2123/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, KV Cache, frequency-domain analysis, I/O-aware Recomputation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mjWGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper finds that the low-frequency part of the KV cache is very helpful for cross-chunk dependency recovery, and thus proposes CacheTune to enable non-prefix KV cache reuse. CacheTune uses a dynamic recomputation ratio to further reduce the latency of KV cache loading.

Critical strengths
  1. Figure 1 shows that recomputing the KV cache for low-frequency indices can achieve higher KV cache similarity, attention distribution, and accuracy. Based on this observation, the paper designed CacheTune.
  2. CacheTune determines the recomputation ratio based on the medium where the KV cache is stored.
Critical weaknesses
  1. The paper does not explain why low-frequency indices are more worthy of being recomputed, and the final experiments show that CacheTune does not have a significant accuracy advantage over other methods.
  2. The experimental setup of EPIC recomputes only the first 16 tokens, but the EPIC paper reports that recomputing the first 32 tokens yields better results. The accuracy and TTFT of EPIC-32 may be superior to those of CacheTune. Additionally, subsequent TTFT experiments did not compare against EPIC.
  3. The token indices recomputed by CacheTune are independent of the query, but different queries may require the recomputation of different tokens. For example, query-aware methods such as Quest and SnapKV outperform query-agnostic sparse attention methods.
Questions for authors
  1. Why are low-frequency tokens worth recomputing, rather than high-frequency tokens or tokens with higher attention scores?
  2. How does CacheFuse compare to EPIC-32 and EPIC-64?
4 comments
AuthorsOpenReview ↗

Response to Reviewer mjWG - Part 1/3: Point-by-point responses

Thank you for your questions and suggestions. In the revised manuscript, we have added an explanation of low-frequency selection and comparisons with EPIC-32/64. We first address your comments point by point, then present the detailed reasoning behind low-frequency selection in Parts 2/3 and 3/3. We hope this addresses all your questions.

W1 & Q1: Why select low-frequency positions, and how much quality does CacheTune gain over existing methods?

Thank you for your question. CacheTune is inspired by image and signal reconstruction: when image energy is concentrated at low frequencies, retaining these components yields a small relative squared reconstruction error. Revised §3, Figure 1 (original Appendix E.1, Figure 9) shows that independently encoded K/V also concentrate their energy at low frequencies along the token axis. We therefore use this low-frequency structure to guide recomputation of a small set of positions and restore cross-chunk context dependencies missing from independent encoding.

To let you read our responses to the other concerns first, we present the detailed reasoning behind recomputing low-frequency positions in Parts 2/3 and 3/3:

  • Part 2/3: Starting from image reconstruction, we derive the relationship between low-frequency energy concentration and KV reconstruction error.
  • Part 3/3: We first test semantic preservation through frequency-band reconstruction, then explain how low-frequency structure yields recomputation positions and examine their role through attention to preceding context, KV drift, and actual recomputation results.

We first address your concerns about CacheTune's limited gains over existing methods and why we do not use attention scores to select recomputation indices.

(1) Generation quality gains over existing methods

Figure 4 of the revised manuscript (Figure 3 in the original) compares generation quality across three models and five datasets. The table below reports CacheTune's gains over each method.

The gains over CacheSlide are smaller, while CacheTune achieves higher generation quality than the other methods listed in the table in all evaluated configurations.

Compared methodquality score differenceConfigurations with a higher CacheTune score
CacheBlend+0.60 to +5.22 points15/15
ProphetKV+0.39 to +3.91 points15/15
CacheSlide−0.01 to +1.46 points14/15
EPIC+1.39 to +6.35 points15/15
KVLink+0.78 to +3.05 points15/15

In non-prefix reuse, cross-chunk attention inevitably arises, so we cannot fully recover the model's inference accuracy. Recomputed KVs are also affected by errors that propagate across layers from the remaining, directly reused KVs, further reducing accuracy.

In our experiments, CacheTune retains approximately 95% of Full Recompute's generation quality on average by selecting only 15% of token positions offline for recomputation, bringing its quality close to full recomputation.

The compared methods already achieve good accuracy, leaving limited theoretical room for improvement, yet CacheTune still achieves higher generation quality. We consider it valuable to address the complex problem of non-prefix reuse with a lightweight but effective method such as CacheTune.

(2) Why not use attention scores directly?

We distinguish between two sources of attention scores:

  • Attention during independent chunk encoding. Each chunk's KVs are generated independently in advance, without the cross-chunk dependencies introduced by new preceding context. A token with high offline attention is therefore not necessarily a position whose KV changes most, or most needs recomputation, in the new context.
  • Full-context attention obtained through full recomputation. These scores can be used to analyze cross-chunk dependencies, but obtaining them requires the corresponding context computation online. In other words, deciding which KVs to reuse and which to recompute would first require the computation that reuse was intended to avoid, introducing substantial additional online computation.

Thus, selecting recomputation positions directly from attention scores makes it difficult to achieve both effective selection and low online overhead.

Manuscript revisions: Moved the KV energy experiment to §3, Figure 1, and added the derivation from image reconstruction to KV reconstruction in Appendix E.1.

AuthorsOpenReview ↗

Part 1/3: Point-by-point responses (continued)

W2 & Q2: Comparison with EPIC-32 and EPIC-64

Thank you for your suggestion.

Our comparison with EPIC-16 follows the default configuration in the system performance evaluation of the original EPIC paper. It also provides a moderate recomputation scope that balances generation quality and recomputation cost. We have added experiments with EPIC-32 and EPIC-64, as shown below.

Each cell reports “generation quality / TTFT speedup”; the Full Recompute column reports generation quality only.

ModelDatasetFull RecomputeCacheTuneEPIC-16EPIC-32EPIC-64
MistralSAMSum42.9342.55 / 4.02×37.99 / 4.81×38.92 / 3.13×38.98 / 1.87×
WikiMQA32.7130.02 / 4.22×27.12 / 5.94×27.53 / 4.53×27.51 / 2.63×
MuSiQue37.5035.50 / 4.21×29.15 / 5.53×29.17 / 4.06×29.20 / 2.34×
LlamaSAMSum40.2137.92 / 3.94×33.74 / 4.79×34.64 / 3.23×34.57 / 1.93×
WikiMQA25.6123.74 / 4.07×21.05 / 6.04×22.47 / 4.85×22.41 / 2.94×
MuSiQue23.6121.57 / 4.12×17.95 / 4.71×17.13 / 3.79×18.09 / 2.29×

Increasing EPIC's budget to 32/64 tokens yields limited and inconsistent quality gains, while its TTFT speedup continues to decrease. CacheTune still achieves higher generation quality than EPIC-32/64. EPIC-32 has lower TTFT on WikiMQA for both models; CacheTune has lower TTFT in the other four settings and in all comparisons with EPIC-64, providing a more favorable quality–latency trade-off.

Analysis: EPIC mainly recomputes tokens at the beginning of each cached chunk to mitigate the attention-sink effect introduced by independent encoding. This fixed block-start selection is simple and efficient, but may miss other positions within the chunk that are strongly affected by context. Expanding the block-start recomputation region may therefore fail to cover these positions effectively.

Subsequent storage-tier experiments: The subsequent TTFT experiments with CPU/SSD/HDD caches primarily evaluate cache offloading and loading optimizations, so they focus on methods with corresponding implementations. The original EPIC implementation does not provide a multi-tier offloading pipeline and was therefore not included in these experiments.

Manuscript revisions: Added comparisons with EPIC-32/64 in Appendix E.9.

W3: Can query-independent selection work across different queries?

Thank you for your question. Our original experiments already validated CacheTune's effectiveness across queries and compared it with a query-aware non-prefix reuse method:

  • Effectiveness across queries: Figure 1(c) in the original manuscript (Figure 2(c) in the revision) and Appendix E.2 specifically evaluated the effectiveness of the same offline-selected recomputation positions across different queries.
  • Comparison with a query-aware method: Figure 3 of the original manuscript (Figure 4 in the revision) compares CacheTune with ProphetKV. The experimental results show that CacheTune achieves an average relative improvement of 4.84% in generation quality over ProphetKV. Because query-aware selection introduces more online overhead, CacheTune also achieves lower TTFT.

CacheTune is effective across queries because it repairs cross-chunk dependencies missing from independently encoded KV Chunks:

  • Select recomputation positions offline; update KVs online. CacheTune uses the low-frequency structure of KVs to prioritize positions that are more likely to exhibit large drift when the context changes. The positions are fixed offline, while the repaired KVs are generated online using the current context.
  • Different queries can benefit from recovering the same dependencies. Although different queries seek different information, they may all rely on cross-chunk attention omitted during independent encoding. The same selected positions can therefore serve multiple queries.
  • Preserve query-specific information access. CacheTune retains KVs at all positions, so the model can still use query-dependent attention to access the information each query needs. This combines cache repair across queries with information access tailored to a specific query.
AuthorsOpenReview ↗

Part 2/3: Justification of the relationship between low-frequency energy and KV reconstruction error (W1 & Q1 continued)

1. Image energy and reconstruction error

Image transform coding represents an image as a weighted sum of a complete orthonormal frequency basis: I=∑kakΦkI=\sum_k a_k\Phi_k, where Φk\Phi_k is a frequency basis element and ak=⟨I,Φk⟩Fa_k=\langle I,\Phi_k\rangle_F is the corresponding projection coefficient.

Orthonormality means that distinct basis elements have zero inner product and each satisfies ∣Φk∣F2=1\\|\Phi_k\\|_F^2=1. Image energy is the sum of squared pixel magnitudes. The energy of the kk-th frequency component akΦka_k\Phi_k is therefore

\begin{aligned} \\|a_k\Phi_k\\|_F^2 &=\sum_{p,q}|a_k\Phi_k(p,q)|^2\\\\ &=|a_k|^2\sum_{p,q}|\Phi_k(p,q)|^2\\\\ &=|a_k|^2\\|\Phi_k\\|_F^2 =|a_k|^2, \end{aligned} \tag{1}

Here, (p,q)(p,q) ranges over image pixels. Equation (1) identifies each coefficient's squared magnitude as its component's energy. When components are summed, orthogonality cancels cross terms between frequencies, giving Parseval's energy identity ∣I∣F2=∑k∣ak∣2\\|I\\|_F^2=\sum_k|a_k|^2.

This relationship lets us directly calculate the error of low-frequency reconstruction. Retaining the original coefficients in the low-frequency set ΩL\Omega_L gives IL=∑k∈ΩLakΦkI_L=\sum_{k\in\Omega_L}a_k\Phi_k. Let ρI\rho_I denote the fraction of total image energy contained in these components. The retained components cancel in I−ILI-I_L, so the residual consists exactly of the discarded frequencies. Applying the same Parseval identity to this residual gives

\begin{aligned} \rho_I &:=\frac{\sum_{k\in\Omega_L}|a_k|^2} {\sum_k|a_k|^2},\\\\[3pt] \frac{\\|I-I_L\\|_F^2}{\\|I\\|_F^2} &=\frac{\sum_{k\notin\Omega_L}|a_k|^2} {\sum_k|a_k|^2}\\\\ &=1-\frac{\sum_{k\in\Omega_L}|a_k|^2} {\sum_k|a_k|^2} =1-\rho_I. \end{aligned} \tag{2}

Thus, the retained low-frequency energy fraction directly determines relative squared reconstruction error. When ρI\rho_I is close to 1, the error relative to the original image is small; under the same frequency budget, retaining a band with more energy gives a smaller error.

2. Extending image reconstruction to KV reconstruction

This derivation extends to KVs because the energy relationship depends on orthonormal transforms preserving squared norms, regardless of whether the signal is a two-dimensional image or a one-dimensional sequence. The unitary Fourier transform along the token axis has the same property. Let the Key or Value tensor at one layer of an independently encoded chunk be X∈RN×dX\in\mathbb R^{N\times d}, where NN is the number of tokens and dd is the number of channels after combining all heads. Keys use their pre-RoPE representations. Transforming along the token axis gives X^=FX\widehat X=FX, and retaining low frequencies followed by the inverse transform gives XL=F∗MLFXX_L=F^*M_LFX. Here, F∗F^* is the conjugate transpose, and MLM_L is a diagonal 0/10/1 mask retaining the corresponding conjugate-symmetric low-frequency set. The reconstruction still has the original NN positions.

For each channel cc, X:,cX_{:,c} is a one-dimensional sequence of length NN. Applying Parseval's identity separately to this sequence and its low-pass reconstruction residual gives

\begin{aligned} \\|X_{:,c}\\|_2^2 &=\sum_k|\widehat X_{k,c}|^2,\\\\ \\|X_{:,c}-(X_L)_{:,c}\\|_2^2 &=\sum_{k\notin\Omega_L}|\widehat X_{k,c}|^2. \end{aligned} \tag{3}

The two lines give the channel's total energy and reconstruction error from discarded frequencies. The squared Frobenius norm sums the channels' squared L2 norms, so summing total and error energy over channels gives the overall KV reconstruction error. With ρX\rho_X denoting the retained low-frequency energy fraction, we obtain

\begin{aligned} \rho_X &:=\frac{\sum_{c=1}^{d}\sum_{k\in\Omega_L}|\widehat X_{k,c}|^2} {\sum_{c=1}^{d}\sum_k|\widehat X_{k,c}|^2},\\\\[3pt] \frac{\\|X-X_L\\|_F^2}{\\|X\\|_F^2} &=\frac{\sum_{c=1}^{d}\\|X_{:,c}-(X_L)_{:,c}\\|_2^2} {\sum_{c=1}^{d}\\|X_{:,c}\\|_2^2}\\\\ &=\frac{\sum_{c=1}^{d}\sum_{k\notin\Omega_L}|\widehat X_{k,c}|^2} {\sum_{c=1}^{d}\sum_k|\widehat X_{k,c}|^2} =1-\rho_X. \end{aligned} \tag{4}

Equations (2) and (4) have the same form: images are decomposed along spatial coordinates and KVs along token coordinates, and in both cases the discarded spectral energy equals the squared reconstruction error.

Revised §3, Figure 1 (original Appendix E.1, Figure 9) shows low-frequency energy concentration in KVs. Equation (4) explains why low-pass filtering can remove rapid variations while retaining the independent cache's main energy structure.

AuthorsOpenReview ↗

Part 3/3: Justification for selecting low-frequency positions for recomputation (W1 & Q1 continued)

1. Frequency-band reconstruction experiments

To test whether this reconstruction advantage preserves document semantics, we conduct frequency-band reconstruction experiments with Mistral v0.3-7B on SAMSum and REALSumm as additional evidence in this response.

We retain approximately 50% of either the low-frequency or high-frequency components of the target document or dialogue KVs, reconstruct the KVs through the inverse transform, and use them for summarization, with the complete KVs as the reference. Both groups retain the DC (mean) component and use the same number of non-DC frequency degrees of freedom, so the comparison focuses on the effect of frequency-band selection.

We measure KV reconstruction error using centered SNR and evaluate the resulting summaries using ROUGE-L. SNR is calculated after subtracting each channel's mean along the token axis; higher values indicate smaller reconstruction errors.

DatasetKV reconstructionK SNR ↑\uparrow (dB)V SNR ↑\uparrow (dB)ROUGE-L ↑\uparrow (0–100)
SAMSumComplete KVs----43.56
Low-frequency12.3211.5939.47
High-frequency1.481.7815.14
REALSummComplete KVs----24.06
Low-frequency13.4912.3922.57
High-frequency1.291.6612.20

With matched frequency degrees of freedom, low-frequency reconstruction yields K/V SNR values approximately 10∼12 dB10\sim 12\text{ dB} higher than high-frequency reconstruction, and summary quality is closer to that of complete KVs. SNR measures representation error; summary quality tests whether reconstructed representations still support summarizing document content. Both support the role of low-frequency structure in preserving content and summary quality.

2. From low-frequency scores to recomputation positions

The preceding derivation and experiments motivate extracting low-frequency structure. We next test whether it identifies positions worth recomputing.

In non-prefix reuse, the discrepancy to repair is between the independently encoded cache XX and full-context encoding XfullX^{\mathrm{full}}. Comparing the same token in the same representation coordinates, we denote its context-induced drift by Δi=Xifull−Xi\Delta_i=X_i^{\mathrm{full}}-X_i. Offline selection must rank positions before XfullX^{\mathrm{full}} has been computed, using only XX. CacheTune scores each original position by the magnitude of its low-pass reconstruction. Using the low-pass projection operator P=F∗MLFP=F^*M_LF, we have

\ell_i=(X_L)_i=\sum_{j=1}^{N}P_{ij}X_j, \qquad s_i=\\|\ell_i\\|_2. \tag{5}

Here, ℓi\ell_i is the representation at original position ii after retaining low frequencies along the token axis and applying the inverse transform; sis_i measures its low-frequency magnitude. As in §4.1, K/V scores are combined and normalized within each layer, then aggregated across layers into shared recomputation indices.

“Low-frequency positions” means original token positions with large low-pass reconstruction magnitudes. Reconstruction supplies offline scores; online execution recomputes selected KVs and reuses the remaining original cache.

How selected positions relate to context-induced changes

To examine how the selected positions read added preceding context, we consider how this context updates target-token representations through self-attention and affects K/V in subsequent layers. On SAMSum and MuSiQue, we fix the target chunk, preceding context, and 15% selection budget, select positions with the highest low-pass or high-pass scores from the independent cache, then measure attention to distant preceding context and normalized K/V drift under full-context encoding.

DatasetSelectionAttention to distant preceding contextK driftV drift
SAMSumLow-pass70.51%0.83470.8760
High-pass15.72%0.33400.3791
MuSiQueLow-pass75.89%0.85000.9152
High-pass19.68%0.41890.5368

On both datasets, low-pass-selected positions allocate more attention to distant preceding context and exhibit greater K/V drift, reflecting larger context-related changes in their cached representations. This attention is used for post-hoc analysis of context dependencies; the indices are determined before full-context encoding.

Low-frequency reconstruction magnitudes can be computed from the independent cache and, in the comparisons above, prioritize positions with stronger interactions with preceding context and larger context-induced drift, supporting their use for offline selection. Figure 2 of the paper (original Figure 1) further shows that, under the same recomputation budget, updating these positions restores the full-context attention distribution and answer quality more effectively than high-frequency selection.

Reviewer_ry8YOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CacheTune, a frequency-guided method for non-prefix KV Cache reuse. It uses the low-frequency structure of independently encoded KV Chunks to identify (in an offline manner) the token positions most likely to experience context-induced KV drift. When a chunk is reused in a new context, only the selected positions are recomputed, while the remaining KV entries are reused from the cache. This approach effectively mitigates the context mismatch caused by missing cross-chunk attention dependencies in independently encoded KV Chunks. The method is evaluated across multiple models, datasets, and hardware platforms, achieving a 3.72–4.86× speedup in time-to-first-token (TTFT) while maintaining generation quality close to Full Recompute (with a 2.34–2.36× speedup for SSD/HDD cache configurations).

Critical strengths

The paper cleverly draws on signal-processing techniques for identifying important feature patterns, using a transformation between the token-sequence domain and the frequency domain to identify important components in the token sequence. From a signal-processing perspective, this provides insight into the structural patterns of KV sequences: most of their energy is concentrated in low-frequency components, which correspond to relatively slow variations along the sequence dimension and a smooth underlying structure, rather than high-frequency local details or differences between adjacent tokens. This observation helps identify token positions that are particularly susceptible to context-induced changes and KV drift, providing a new way to assess token importance. This approach may also inspire new research directions in other domains. In addition, the paper analyzes the competition between KV recomputation and KV transfer along the latency-critical path and develops a model for selecting parameters under different hardware resource constraints.

Critical weaknesses

Given the interest of the proposed mechanism, a more thorough mechanistic explanation would help readers better understand the phenomenon underlying it. The paper demonstrates that low-frequency-guided selection achieves up to 90.23% Recall@15% for high-drift KV positions (Table 3), but it does not rigorously explain why tokens selected based on low-frequency structure are more susceptible to context-induced KV drift and provide rigorous verification of the possible mechanisms. It also doesn't explain why low-frequency filtering predicts KV drift more accurately than the raw KV norm (Figure 5). Based on the evidence presented, the paper establishes a correlation between KV drift and low-frequency reconstruction magnitudes, but does not sufficiently elucidate the underlying mechanism connecting the two.

Questions for authors
  1. Since the KV entries of directly reused tokens remain in the cache, the selected token positions, when recomputed layer by layer, still attend to cached KV entries that have not been recomputed and may therefore remain context-mismatched. Consequently, the recomputed KV Cache may still contain residual errors, which can propagate across layers. The paper states that sharing recomputation indices across layers establishes a continuous recomputation path. However, the residual KV error after layer-wise recomputation, as well as its propagation across layers, should be further quantified.
  2. Equation (3) assigns equal weights of 50% to K and V, while Equation (4) first normalizes the scores within each layer and then averages the normalized scores equally across all layers. However, the importance of K (which participates in Query–Key matching and influences attention weights) may differ from that of V, which provides the content aggregated according to those attention weights. Similarly, some layers, such as middle layers, may be more important for restoring cross-chunk dependencies than the first or last layers. Further weighted ablation studies should be conducted to establish whether equal weighting is a reasonable design choice.
  3. The complementary portion to be transferred consists of non-contiguous positions, whereas contiguous memory accesses are generally more efficient. The paper does not separately report the additional overhead introduced by sparse, non-contiguous access itself. The overhead of index processing and sparse access may offset some of the performance gains. A more detailed evaluation of the impact of sparse access is therefore needed to establish the practical performance boundaries of the proposed method.
5 comments
AuthorsOpenReview ↗

Response to Reviewer ry8Y - Part 1/3: Point-by-point responses

We sincerely thank you for recognizing the strengths of our paper and for your suggestions. Your comments gave us useful insights, several of which warranted closer investigation and discussion. We spent considerable time addressing them, and this helped us improve the paper.

In the revised paper, we have added explanations of low-frequency selection, residual-error analysis, and weighting ablations, and clarified sparse cache storage and loading.

W1: Why does low-frequency structure help identify positions for recomputation, and why is it more effective than the raw KV norm?

As you noted, CacheTune is primarily inspired by image and signal reconstruction. CacheTune extracts low-frequency structure along the token axis from the independently encoded cache and ranks positions by their reconstructed magnitudes. Low-frequency energy concentration provides a reconstruction-based rationale for this selection.

To let you read our responses to the other questions first, we present the detailed reasoning in Parts 2/3 and 3/3 below:

  • Part 2/3: Derive the relationship between low-frequency energy and KV reconstruction error.
  • Part 3/3: Use KV reconstruction experiments and KV drift analysis to explain why we use low-frequency selection and its advantage over raw L2 ranking.

Manuscript revisions: Section 3, Figure 1 moves the energy experiment into the main text; Appendix E.1 derives the energy–error relationship for image and KV reconstruction and explains position scoring.

Q1: How do residual errors propagate across layers?

We agree that shared indices sustain repair across layers but do not automatically eliminate errors introduced by reused KVs. Mismatched Keys and Values that are directly reused change attention weights and aggregated content. These changes pass through residual connections and feed-forward computations, affecting subsequent hidden states and introducing errors into the recomputed KVs.

We believe this issue warrants careful analysis. Appendix E.3 therefore describes the error measures, cache assembly, KV projection, and propagation paths in detail. Below, we first report our quantitative results on the propagation of KV errors across layers, then analyze how CacheTune helps mitigate this issue.

Evaluation and results. We quantified this error propagation using the 32-layer Mistral model, MuSiQue, and a 15% recomputation ratio, as shown in Figure 11 in Appendix E.3 of the revision. With full-context encoding as the reference, we compare relative Frobenius errors before and after repair at the same selected positions SS, fixed across layers; the complement UU remains directly reused. Keys are measured before RoPE to exclude positional rotation.

Figure 11 shows the following:

  • When the first-layer KVs and global positions are correctly aligned, recomputation in the first layer can recover the selected positions' hidden states. Their second-layer KVs can therefore remain close to those from full recomputation.
  • The KVUKV_U read in later layers already contain contextual errors. These errors enter the hidden states at SS and affect the regenerated KVs.

Compared with direct reuse at SS, recomputation lowers the mean K and V errors across layers by approximately 61% and 67%, respectively.

Why the recomputed KV errors decrease: We interpret this result through the interaction between low-frequency selection and continuous recomputation. Low-frequency selection prioritizes tokens more involved in cross-chunk interactions, while shared indices keep their updated representations involved in subsequent layers. Thus, even though later layers remain affected by unrecomputed KVs, continuous repair at these key positions helps recover missing cross-chunk context and reduces the errors introduced by direct KV reuse.

Manuscript revisions: Section 4.1 explains error sources; the new Appendix E.3 describes error measures, propagation paths, and the layer-wise example in Figure 11.

AuthorsOpenReview ↗

Part 1/3: Point-by-point responses (continued)

Q2: K/V and layer-weighting ablations

Thank you for your suggestion. We have added ablations of K/V weights and layer weights. Both ablations use a 15% recomputation ratio.

We use equal weighting as the default in the paper because it requires no additional model-specific tuning while achieving near-best results.

K/V weighting ablation

K determines query–cache matching, whereas V supplies the aggregated content. Their low-frequency scores can favor different positions. Since recomputation updates K and V at the same position, we combine their scores into one ranking. We compare K weights of 0,0.25,0.5,0.75,10,0.25,0.5,0.75,1, with complementary V weights.

K / V weightsMistral v0.3-7B SAMSumMistral v0.3-7B MuSiQueLlama 3-8B SAMSumLlama 3-8B MuSiQue
0.00 / 1.0042.3832.7237.6421.29
0.25 / 0.7542.0635.0636.5921.55
0.50 / 0.5042.5535.5037.9221.57
0.75 / 0.2542.3335.5737.4620.38
1.00 / 0.0041.7634.4536.8819.64

Equal weighting achieves the highest quality in three of the four settings and is close to the best in the remaining setting. Compared with using only K or V, combining both signals selects KV positions for joint updates more consistently.

Layer-weighting ablation

CacheTune first normalizes scores within each layer to remove differences in their total scale. Each layer then provides the relative importance of tokens, and aggregation across layers produces a shared ranking under a fixed recomputation budget.

We compare equal weighting with schemes that emphasize early, middle, or late layers.

  • For both 32-layer models, the three groups comprise layers 1–10, 11–22, and 23–32, respectively.
  • Each layer in the emphasized group receives weight 2, and the remaining layers receive weight 1. We then normalize the layer weights.
Layer-weighting schemeMistral v0.3-7B SAMSumMistral v0.3-7B MuSiQueLlama 3-8B SAMSumLlama 3-8B MuSiQue
Equal weighting42.5535.5037.9221.57
Early layers emphasized42.5934.7836.0621.10
Middle layers emphasized41.1133.8837.1420.46
Late layers emphasized42.3234.1437.9921.21

Emphasizing a fixed layer group does not consistently improve quality across models and tasks, whereas equal weighting gives near-best results in all four settings. We therefore retain equal-weight aggregation as the default.

Manuscript revisions: Appendix E.5 adds both ablations; Section 4.1 adds the design rationale and a reference to this appendix.

Q3: Could index processing and non-contiguous access offset the loading gains?

Thank you for your suggestion. CacheTune analyzes the frequency structure of cached KVs to determine recomputation indices and the corresponding reused positions offline. This allows the cache storage and transfer layouts to be organized in advance.

Therefore, once the indices are determined, we pack the KVs to be reused into a contiguous buffer for storage. This is one benefit of determining recomputation positions in advance.

In the original paper, “sparse” refers to loading only the positions to be reused relative to the original token sequence. Because these KVs are stored contiguously before transfer, online loading can use contiguous memory reads and batched transfers. We have clarified this implementation in Section 4.2 of the revision.

Table 4 in Appendix E.5 of the original paper (Table 5 in Appendix E.7 of the revision) reports the incremental ablation of sparse loading. On top of pipeline overlap and cross-layer prefetching, adding sparse loading further reduces TTFT by 8.5%-11.9%. In these configurations, reducing KV transfer therefore provides a net benefit while retaining online index handling and writeback operations.

Manuscript revisions: Section 4.2 adds the explanation of offline contiguous packing.

AuthorsOpenReview ↗

Part 2/3: Justification of the relationship between low-frequency energy and KV reconstruction error (W1 continued)

1. Image energy and reconstruction error

In image transform coding, an image can be represented as a weighted sum of a complete orthonormal frequency basis, I=∑kakΦkI=\sum_k a_k\Phi_k, where Φk\Phi_k is a frequency basis element and ak=⟨I,Φk⟩Fa_k=\langle I,\Phi_k\rangle_F is the image's projection coefficient in that frequency direction.

By orthonormality, distinct basis elements have zero inner product, and each satisfies ∣Φk∣F2=1\\|\Phi_k\\|_F^2=1. Image energy is defined as the sum of squared pixel magnitudes. The energy of the kk-th frequency component akΦka_k\Phi_k is therefore

\begin{aligned} \\|a_k\Phi_k\\|_F^2 &=\sum_{p,q}|a_k\Phi_k(p,q)|^2\\\\ &=|a_k|^2\sum_{p,q}|\Phi_k(p,q)|^2\\\\ &=|a_k|^2\\|\Phi_k\\|_F^2 =|a_k|^2, \end{aligned} \tag{1}

where (p,q)(p,q) ranges over image pixels. Equation (1)(1) identifies the squared coefficient magnitude as the component's energy. Summing the components eliminates cross terms by orthogonality, giving Parseval's identity ∣I∣F2=∑k∣ak∣2\\|I\\|_F^2=\sum_k|a_k|^2.

Retaining the original coefficients in the low-frequency set ΩL\Omega_L gives IL=∑k∈ΩLakΦkI_L=\sum_{k\in\Omega_L}a_k\Phi_k. Let ρI\rho_I denote the fraction of the original image's total energy contained in these components. The retained components cancel in I−ILI-I_L, so the residual consists exactly of the discarded frequencies. Applying the same Parseval identity to this residual gives

\begin{aligned} \rho_I &:=\frac{\sum_{k\in\Omega_L}|a_k|^2} {\sum_k|a_k|^2},\\\\[3pt] \frac{\\|I-I_L\\|_F^2}{\\|I\\|_F^2} &=\frac{\sum_{k\notin\Omega_L}|a_k|^2} {\sum_k|a_k|^2}\\\\ &=1-\frac{\sum_{k\in\Omega_L}|a_k|^2} {\sum_k|a_k|^2} =1-\rho_I. \end{aligned} \tag{2}

Thus, the fraction of energy retained at low frequencies directly determines the relative squared reconstruction error: the error is small when ρI\rho_I is close to 11, and retaining a band with more energy yields a smaller error under the same frequency-retention budget.

2. The energy–error relationship in KV reconstruction

The derivation extends to KVs because the unitary Fourier transform along the token axis also satisfies Parseval's identity. Let the K or V of an independently encoded chunk at a given layer be X∈RN×dX\in\mathbb R^{N\times d}, where NN is the number of tokens and dd is the number of channels after combining the heads. Transforming along the token axis gives X^=FX\widehat X=FX. Retaining low frequencies and applying the inverse transform gives XL=F∗MLFXX_L=F^*M_LFX. Here, F∗F^* denotes the conjugate transpose, and MLM_L is a 0/10/1 diagonal mask that retains the corresponding conjugate-symmetric set of low frequencies. The reconstruction still has the original NN positions.

For each channel cc, X:,cX_{:,c} is a one-dimensional sequence of length NN. Applying Parseval's identity (equal squared norms before and after the transform) to this sequence and its low-pass reconstruction residual gives

\begin{aligned} \\|X_{:,c}\\|_2^2 &=\sum_k|\widehat X_{k,c}|^2,\\\\ \\|X_{:,c}-(X_L)_{:,c}\\|_2^2 &=\sum_{k\notin\Omega_L}|\widehat X_{k,c}|^2. \end{aligned} \tag{3}

The two lines give the channel's total energy and the error energy from discarded frequencies. Since the KV matrix's squared Frobenius norm is the sum of squared L2 norms across channels, summing each quantity over channels gives the overall reconstruction error. Letting ρX\rho_X denote the fraction of KV energy retained at low frequencies, we obtain

\begin{aligned} \rho_X &:=\frac{\sum_{c=1}^{d}\sum_{k\in\Omega_L}|\widehat X_{k,c}|^2} {\sum_{c=1}^{d}\sum_k|\widehat X_{k,c}|^2},\\\\[3pt] \frac{\\|X-X_L\\|_F^2}{\\|X\\|_F^2} &=\frac{\sum_{c=1}^{d}\\|X_{:,c}-(X_L)_{:,c}\\|_2^2} {\sum_{c=1}^{d}\\|X_{:,c}\\|_2^2}\\\\ &=\frac{\sum_{c=1}^{d}\sum_{k\notin\Omega_L}|\widehat X_{k,c}|^2} {\sum_{c=1}^{d}\sum_k|\widehat X_{k,c}|^2} =1-\rho_X. \end{aligned} \tag{4}

Equations (2) and (4) have the same form: images are decomposed along spatial coordinates and KVs along token coordinates, and in both cases the discarded spectral energy equals the squared reconstruction error. Figure 1 in Section 3 shows the concentration of energy at low frequencies in independently encoded KVs. Equation (4) explains why this matters: low-pass filtering can remove faster-varying components while retaining the main energy structure of the independently encoded cache.

Reviewer_5UG8OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets non-prefix KV cache reuse for long-context serving, where independently encoded chunks are reused at new positions without cross-chunk attention. It proposes CacheTune, which ranks tokens offline by the magnitude of their low-pass-filtered KVs and recomputes only the top-ranked tokens in the new context. This allows the system to skip loading those KVs from CPU or disk. Experiments on summarization and multi-hop QA tasks show quality close to full recomputation, along with throughput gains.

Critical strengths
  1. This work tackles a practical problem in RAG, and non-prefix KV cache reuse is an interesting topic to study.

  2. Using the frequency structure of a chunk’s own KVs as an offline, query-independent selection signal is a new and lightweight idea.

  3. The selection signal is supported by direct evidence that low-frequency selection captures most of the tokens whose KVs change the most under full-context encoding.

  4. The evaluation covers three models, five tasks, and several storage tiers, with ablations on each component.

Critical weaknesses
  1. The first concern is whether the proposed method is broadly applicable or limited to RAG or specific systems. For example, it is unclear whether the framework can be used for common agentic tasks. The evaluation is also limited and does not cover diverse RAG benchmarks such as CRAG (Comprehensive RAG Benchmark) and FRAMES.

  2. The models used in the evaluation, Mistral v0.3-7B, Llama 3-8B, and Qwen 2.5-32B, are relatively old and do not represent modern LLMs.

  3. Another major concern is that the TTFT gains over prior selective recomputation methods are small and seem to come mostly from loading optimizations. With a GPU-resident cache, CacheTune is on par with or slightly slower than CacheSlide (Table 1). Without its loading optimizations, it is slower than CacheBlend on CPU. Since the same pipelining could be applied to the baselines, an accuracy–TTFT curve under a shared pipeline is needed to demonstrate the value of the selection method itself.

  4. Another major concern is that the paper provides little insight into why low-frequency KV magnitude identifies tokens that need repair. The evidence is mainly correlational, and plain L2-norm selection performs poorly, suggesting that the low-pass filtering step is critical. However, its role is not explained.

  5. The experiments do not include integration with a production serving engine such as vLLM or SGLang to evaluate realistic serving performance. The throughput study provides no batching details, and evaluation at higher concurrency is left as future work.

Questions for authors

I have included my questions in the weakness section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5UG8 - Part 1/3: Point-by-point responses

Thank you for your thorough and detailed review and for recognizing the novelty of our work. Your feedback has been very helpful in improving our paper. We first address your questions point by point, then explain the low-frequency mechanism in detail in Parts 2/3 and 3/3.

W1: Applicability, RAG benchmarks, and agent tasks

CacheTune applies when the same content is reused after different prefixes, including documents retrieved in RAG, tool outputs, and agent-generated reports. For example, when different agents read the same report under different role instructions and preceding contexts, CacheTune reuses its independently encoded cache and recomputes selected positions to recover cross-chunk dependencies.

We have added CRAG, FRAMES, and a GSM8K cross-agent workflow to examine reuse opportunities in real requests and the effectiveness of cross-agent reuse.

1. Additional RAG benchmarks

  • Natural request order: We evaluate all 1,335 questions in CRAG and 824 FRAMES questions. Evidence becomes reusable only after it has appeared in an earlier request. Non-prefix reuse occurs in 14.16% and 36.41% of requests in the two benchmarks, respectively, while exact prefix matching reuses only 4.98% and 2.31% of input tokens. Non-prefix reuse can therefore substantially improve the cache hit rate.
  • All evidence cached in advance: All evidence is independently encoded and cached before requests arrive, so every request can reuse its evidence.

Results cover the complete request streams. Each cell gives Full Recompute F1 → CacheTune F1; TTFT speedup, relative to Full Recompute for the same model and dataset.

Natural request order

ModelCRAGFRAMES
Mistral v0.3-7B30.60 → 29.95; 1.12×13.40 → 13.37; 1.33×
Qwen3-8B23.05 → 23.01; 1.11×9.84 → 9.82; 1.31×

All evidence cached in advance

ModelCRAGFRAMES
Mistral v0.3-7B30.60 → 27.39; 3.98×13.40 → 14.54; 4.65×
Qwen3-8B23.05 → 21.11; 3.73×9.84 → 9.79; 4.51×

Under natural request order, CacheTune’s generation quality is close to Full Recompute, but limited cached evidence constrains overall speedup. Caching all evidence in advance expands reuse, increases speedup, and makes the effects on quality more apparent. Large KV cache pools in production environments can store more reusable KVs, making such deployments closer to the all-evidence-cached condition.

2. Cross-agent workflow

Qwen3-8B acts as the analyst, critic, and judge on all 1,319 GSM8K test questions. The judge reads a previously processed report under different role instructions and prefixes. We fix and replay upstream reports and critiques so that both methods receive identical judge inputs, and measure accuracy and TTFT at the judge stage.

MethodAccuracyTTFT speedup
Full Recompute94.01%1.00×
CacheTune, 15% recomputation93.33%1.34×

All judge requests reuse cached reports, which account for 38.81% of input tokens. CacheTune’s accuracy decreases by 0.68 percentage points, with a 1.34× TTFT speedup, showing that non-prefix reuse also supports cross-agent workflows.

Manuscript revisions: Added RAG and cross-agent results in Appendices E.8.2 and E.8.3.

W2: Newer models

We have added CacheTune results on Qwen3-8B, Qwen3.5-9B, and Ministral 3 14B Instruct, covering full-attention and hybrid attention–recurrent models. The cross-attention issue in non-prefix reuse arises across models. Selective recomputation repairs these dependencies, and hardware-aware loading optimizations reduce TTFT; these principles do not depend on a particular generation of models. Our original model selection therefore emphasized model parameter count. As LLMs develop, evaluating newer models also helps assess CacheTune’s generality.

Each cell gives Full Recompute → CacheTune quality; TTFT speedup, relative to Full Recompute.

ModelSAMSum (ROUGE-L)WikiMQA (F1)MuSiQue (F1)
Qwen3-8B39.04 → 38.86; 3.81×27.17 → 25.95; 3.73×21.60 → 20.87; 4.52×
Qwen3.5-9B34.90 → 34.76; 1.39×37.37 → 36.22; 1.45×38.82 → 38.60; 1.42×
Ministral 3 14B Instruct40.91 → 39.96; 4.18×44.44 → 42.07; 4.37×39.45 → 38.59; 4.67×

Qwen3.5-9B contains 24 Gated DeltaNet layers and 8 full-attention layers. We retain full-sequence updates in the recurrent layers, selectively recompute the attention and MLP branches in full-attention layers, and reuse KVs and branch outputs at the remaining positions in those layers. Subsequent recurrent layers therefore still receive a complete sequence. The updated anonymous repository includes this adaptation.

All three models maintain good generation quality while reducing TTFT. Qwen3.5-9B’s full-sequence recurrent updates and smaller proportion of layers supporting KV reuse limit its speedup relative to full-attention models.

Manuscript revisions: Added new-model results in Appendix E.8.1.

AuthorsOpenReview ↗

Part 1/3: Point-by-point responses (continued)

W3: Loading optimizations and shared-pipeline comparison

We have added a quality–TTFT comparison in which all three methods share CacheTune’s loading pipeline. We first clarify the configurations in the original Table 1, then present the comparison.

1. Original GPU and CPU configurations

In the original Table 1:

  • CacheTune GPU: Uses the native cache reuse path without the loading optimizations in §4.2.
  • CacheTune CPU: Enables all optimizations.
  • CacheBlend and CacheSlide: Use their respective original loading and execution optimizations.

Thus, the original GPU results reflect CacheTune’s path without loading optimizations, with TTFT close to the baselines. Revised Figure 7 replaces Table 1 and compares optimized CacheTune with both baselines under GPU- and CPU-resident caches. CacheTune achieves higher quality and lower TTFT in both configurations; in the CPU configuration, its mean TTFT is 25.3% and 12.0% lower than that of CacheBlend and CacheSlide, respectively.

2. Shared loading pipeline

Using Mistral v0.3-7B, all three methods share CacheTune’s CPU cache-loading pipeline while retaining their original position-selection strategies and recomputation settings. These fixed configurations yield the following results:

DatasetMethodQualityTTFT (ms)
SAMSumCacheTune0.425581.7
CacheBlend0.409498.6
CacheSlide0.420191.7
WikiMQACacheTune0.3002134.2
CacheBlend0.2766158.9
CacheSlide0.3000138.6
MuSiQueCacheTune0.3550128.9
CacheBlend0.3260151.3
CacheSlide0.3457143.7

CacheTune reduces TTFT by 14.8%–17.1% relative to CacheBlend and 3.2%–10.9% relative to CacheSlide. Its offline indices still provide two benefits:

  • Less loading: Positions are known before requests arrive, so CacheTune loads only reused KVs, excluding those to be recomputed.
  • No online position selection: Positions are determined offline from the KV Chunk, so the recomputation indices do not need to be calculated again during execution.

These benefits allow CacheTune to retain its TTFT advantage under the shared pipeline.

Manuscript revisions: Updated §5.5 and Figure 7, and added the shared-pipeline comparison in E.7.

W4: Why low-pass filtering identifies positions that need repair

We have added an explanation connecting low-frequency reconstruction to position selection, linking the spectral observations, mathematical derivation, and recomputation results.

CacheTune is mainly inspired by image and signal reconstruction. The spectral experiments in the original paper (§3 and Figure 1 in the revision) show that independently encoded K/V representations concentrate their energy at low frequencies along the token axis, as images do. Low-frequency components are more important for reconstructing a complete image. We use this property in the design of CacheTune.

To let you first read our responses to the other concerns, we provide the full argument and experimental analysis later in Part 2/3 and Part 3/3:

  • Part 2/3: Derive the relationship between retained energy and reconstruction error from orthogonal image decomposition, extend it to KVs, and explain how the reconstruction is used for offline position ranking.
  • Part 3/3: Examine the retained information through frequency-band reconstruction and explain position scoring and its difference from raw L2 ranking.

Manuscript revisions: Moved the KV energy experiment to §3 and Figure 1, and added the derivation in Appendix E.1.

W5: Serving engine, batching, and throughput

We apologize for the confusion: CacheTune is built on vLLM. We provide CacheTune implementations based on vLLM 0.4.1 and 0.17.1 in the anonymous repository.

1. Clarification of batching and workload settings

We vary the request arrival rate, with vLLM dynamically batching pending requests. The request rates range from 0.5–32 requests/s for Mistral and 0.1–12 requests/s for Qwen. All methods use the same request workloads and scheduling limits: at most 64 sequences and 16,384 tokens per iteration.

2. Throughput comparison criterion

Figure 5 plots achieved throughput on the horizontal axis and mean TTFT on the vertical axis. For each model–task pair, we compare the maximum achieved throughput under the same TTFT limit. Queuing raises TTFT as requests increase; CacheTune reaches higher throughput at lower latency, serving more requests under the same latency requirement.

Figure 5 already covers the progression from low load toward saturation under dynamic batching. The original reference to “higher concurrency” as future work meant that we would like to extend the evaluated load range further or test the method in a production environment.

Manuscript revisions: Clarified the experimental settings and comparison criterion in the first paragraph of §5.3.

AuthorsOpenReview ↗

Part 2/3: Justification of the relationship between low-frequency energy and KV reconstruction error (W4, continued)

1. Frequency energy and reconstruction error in images

In image transform coding, an image is represented as a weighted sum over a complete orthonormal frequency basis: I=∑kakΦkI=\sum_k a_k\Phi_k, where Φk\Phi_k is a frequency basis element and ak=⟨I,Φk⟩Fa_k=\langle I,\Phi_k\rangle_F is the projection coefficient in that frequency direction.

Orthonormality ensures that different basis elements have zero inner product and each satisfies ∣Φk∣F2=1\\|\Phi_k\\|_F^2=1. Image energy is the sum of squared pixel magnitudes, so the energy of the kk-th frequency component, akΦka_k\Phi_k, is

\begin{aligned} \\|a_k\Phi_k\\|_F^2 &=\sum_{p,q}|a_k\Phi_k(p,q)|^2\\\\ &=|a_k|^2\sum_{p,q}|\Phi_k(p,q)|^2\\\\ &=|a_k|^2\\|\Phi_k\\|_F^2 =|a_k|^2, \end{aligned} \tag{1}

Here, (p,q)(p,q) ranges over the image pixels, and Equation (1) gives the energy of one frequency component. When the components are summed, orthogonality eliminates the cross terms between frequencies, giving the total image energy ∣I∣F2=∑k∣ak∣2\\|I\\|_F^2=\sum_k|a_k|^2. This is Parseval’s energy identity.

Retaining the original coefficients in the low-frequency set ΩL\Omega_L gives IL=∑k∈ΩLakΦkI_L=\sum_{k\in\Omega_L}a_k\Phi_k. Let ρI\rho_I denote the fraction of image energy retained in these components. The retained components cancel in I−ILI-I_L, so the residual consists exactly of the discarded frequencies. Applying Parseval’s identity to the residual gives

\begin{aligned} \rho_I &:=\frac{\sum_{k\in\Omega_L}|a_k|^2} {\sum_k|a_k|^2},\\\\[3pt] \frac{\\|I-I_L\\|_F^2}{\\|I\\|_F^2} &=\frac{\sum_{k\notin\Omega_L}|a_k|^2} {\sum_k|a_k|^2}\\\\ &=1-\frac{\sum_{k\in\Omega_L}|a_k|^2} {\sum_k|a_k|^2} =1-\rho_I. \end{aligned} \tag{2}

Thus, the fraction of energy retained at low frequencies directly determines the relative squared reconstruction error. When ρI\rho_I is close to 1, the relative squared error between the low-frequency reconstruction and the original image is small. With the same frequency-retention budget, retaining more energy also yields a smaller reconstruction error.

2. Extending the energy–error relationship to KVs

The derivation extends to KVs because a unitary Fourier transform along the token axis also satisfies Parseval’s identity. Let X∈RN×dX\in\mathbb R^{N\times d} denote the K or V representation at one layer of an independently encoded chunk, where NN is the number of tokens and dd is the number of channels after combining the heads. Transforming along the token axis gives X^=FX\widehat X=FX; retaining low frequencies and applying the inverse transform gives XL=F∗MLFXX_L=F^*M_LFX. Here, F∗F^* denotes the conjugate transpose, and MLM_L is a diagonal 0/10/1 mask retaining the corresponding conjugate-symmetric low-frequency set. The reconstruction still has all NN original positions.

For each channel cc, X:,cX_{:,c} is a one-dimensional sequence of length NN. Parseval’s identity equates the squared norms before and after transformation. Applying it to the sequence and its low-pass reconstruction residual gives

\begin{aligned} \\|X_{:,c}\\|_2^2 &=\sum_k|\widehat X_{k,c}|^2,\\\\ \\|X_{:,c}-(X_L)_{:,c}\\|_2^2 &=\sum_{k\notin\Omega_L}|\widehat X_{k,c}|^2. \end{aligned} \tag{3}

The first line gives the channel’s total energy; the second gives the reconstruction error from discarded frequencies. The squared Frobenius norm of the KV matrix equals the sum of its channels’ squared L2 norms, so summing each equation over channels yields the overall reconstruction error. Let ρX\rho_X denote the fraction of KV energy retained at low frequencies:

\begin{aligned} \rho_X &:=\frac{\sum_{c=1}^{d}\sum_{k\in\Omega_L}|\widehat X_{k,c}|^2} {\sum_{c=1}^{d}\sum_k|\widehat X_{k,c}|^2},\\\\[3pt] \frac{\\|X-X_L\\|_F^2}{\\|X\\|_F^2} &=\frac{\sum_{c=1}^{d}\\|X_{:,c}-(X_L)_{:,c}\\|_2^2} {\sum_{c=1}^{d}\\|X_{:,c}\\|_2^2}\\\\ &=\frac{\sum_{c=1}^{d}\sum_{k\notin\Omega_L}|\widehat X_{k,c}|^2} {\sum_{c=1}^{d}\sum_k|\widehat X_{k,c}|^2} =1-\rho_X. \end{aligned} \tag{4}

Equations (2) and (4) have the same form: for frequency decomposition over spatial coordinates in images and token coordinates in KVs, discarded spectral energy equals squared reconstruction error. The spectral experiment in Appendix E.1 of the original paper (§3 and Figure 1 in the revision) shows that actual KV energy is concentrated at low frequencies. Equation (4) explains why this matters: low-pass filtering removes rapidly varying components while retaining the main energy structure of the independently encoded cache.

CacheTune uses XLX_L to rank positions, recomputes the actual KVs at selected positions online, and reuses the original cache elsewhere. This repairs cross-chunk dependencies while retaining information at the remaining positions.

AuthorsOpenReview ↗

Part 3/3: Justification for low-frequency selection and its advantage over raw L2 ranking (W4, continued)

1. Frequency-band reconstruction experiments

To test whether the above reconstruction advantage helps preserve document semantics, we conduct frequency-band reconstruction experiments on SAMSum and REALSumm using Mistral v0.3-7B. We retain approximately 50% of either the low- or high-frequency components of the target document or dialogue KVs, reconstruct the KVs through the inverse transform, and use them for summarization, with the complete KVs as the reference. Both groups retain the DC (mean) component and have matched non-DC frequency degrees of freedom, focusing the comparison on frequency-band selection.

We measure KV reconstruction error using centered SNR and summary quality using ROUGE-L. SNR is computed after subtracting each channel’s mean along the token axis; higher values indicate smaller reconstruction errors.

With matched frequency degrees of freedom, low-frequency reconstruction yields K/V SNR values approximately 10∼12 dB10\sim 12\text{ dB} higher than high-frequency reconstruction. Its summary quality is close to that obtained with complete KVs, indicating that it preserves the original content well.

DatasetKV reconstructionK SNR ↑\uparrow (dB)V SNR ↑\uparrow (dB)ROUGE-L ↑\uparrow (0–100)
SAMSumComplete KVs----43.56
Low-frequency12.3211.5939.47
High-frequency1.481.7815.14
REALSummComplete KVs----24.06
Low-frequency13.4912.3922.57
High-frequency1.291.6612.20

Next, we explain how low-frequency reconstruction guides the selection of positions to recompute.

2. From low-pass reconstruction to position scoring

In non-prefix reuse, the discrepancy to repair is between the independently encoded cache XX and the full-context encoding XfullX^{\mathrm{full}}. Comparing the same token in the same representation coordinates, we denote its context-induced drift by Δi=Xifull−Xi\Delta_i=X_i^{\mathrm{full}}-X_i. Before XfullX^{\mathrm{full}} has been computed, offline selection must rank positions using only XX. CacheTune scores each original position by its low-pass reconstruction magnitude. Using the low-pass projection operator P=F∗MLFP=F^*M_LF introduced above, we have

\ell_i=(X_L)_i=\sum_{j=1}^{N}P_{ij}X_j, \qquad s_i=\\|\ell_i\\|_2. \tag{5}

Here, ℓi\ell_i is the representation at position ii after retaining low frequencies along the token axis and applying the inverse transform, and sis_i measures the low-frequency magnitude there. As described in §4.1, K/V scores are combined and normalized within each layer, then aggregated across layers to obtain shared recomputation indices.

3. Difference from raw L2-norm ranking

To compare this score with the raw L2 norm, let hi=Xi−ℓih_i=X_i-\ell_i denote the discarded high-frequency component. A position-wise decomposition gives

\\|X_i\\|_2^2 = \underbrace{\\|\ell_i\\|_2^2}_{s_i^2} + \underbrace{\\|h_i\\|_2^2+2\langle\ell_i,h_i\rangle} _{\text{high-frequency component and position-wise cross term}}. \tag{6}

The raw L2 norm ∣Xi∣22\\|X_i\\|_2^2 includes the low-pass structure, the high-frequency component, and their cross term at that position. Orthogonality guarantees ∑i⟨ℓi,hi⟩=0\sum_i\langle\ell_i,h_i\rangle=0 over the whole sequence, but not a zero cross term at every position. Thus, even a small fraction of high-frequency energy can change the norm ranking across positions. Low-pass scoring measures the low-frequency magnitude alone, focusing selection on the retained, slowly varying structure.

4. Comparison under the same selection budget

To examine the role of low-pass filtering in position selection, we add the following comparison. On SAMSum and MuSiQue, we fix the target chunk, preceding context, and 1515\\% selection budget. We select positions using the low-pass magnitudes or raw L2 norms of the independently encoded cache, then measure their attention to distant preceding context and normalized K/V drift under full-context encoding.

DatasetSelection methodAttention to distant preceding contextK driftV drift
SAMSumLow-pass70.51%0.83470.8760
Raw L210.81%0.26020.2727
MuSiQueLow-pass75.89%0.85000.9152
Raw L215.74%0.34900.5154

On both datasets, low-pass selection yields positions with greater attention to distant preceding context and larger K/V drift, indicating stronger attention to context under the same budget. Together with the recomputation-quality comparison in §5.4 and Figure 6 of the revised manuscript, these results support the practical value of low-pass filtering: compared with raw KV norms, it more effectively selects positions affected by context changes and improves generation quality after recomputation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers and Chairs

We thank the reviewers for their careful and detailed reviews, recognition of the strengths of our work, and constructive suggestions. We also thank the Area Chairs and Program Chairs for their time in assessing our paper and responses.

The comments gave us concrete directions for deepening the mechanism analysis, improving experimental comparisons, and clarifying applicability. We were also pleased to receive several thought-provoking comments and would be very happy to discuss them in depth.

1. Shared concern and organization of our responses

All three reviewers raised a central question: why can low-frequency structure in independently encoded KVs guide the selection of positions worth recomputing, and what role does low-pass filtering play? We provide additional mathematical derivations and experimental analyses, organized around each reviewer's particular concerns.

To make the responses easier to navigate, we divide each into three Parts:

  • Part 1/3: Point-by-point responses. We answer the comments in order, report relevant results, and identify manuscript revisions. We suggest reading this part first.
  • Part 2/3: Reconstruction rationale. Starting from image reconstruction, we derive the relationship between retained spectral energy and reconstruction error, then extend it to KV representations.
  • Part 3/3: Position selection and experimental validation. We use frequency-band reconstruction, preceding-context attention, KV drift, and actual recomputation results to explain how low-frequency structure provides position scores and how the selected positions help recover generation quality.

The latter two parts are lengthy, and we recognize that they require additional reading time and place a considerable burden on the reviewers. We sincerely apologize for this and welcome further discussion of these analyses.

2. Manuscript revisions

We mark the main revisions in blue in the submitted revised manuscript.

Main text

  • Section 3, Figure 1: We move the KV spectral-energy experiment from the original appendix into the main text.
  • Section 5.3: We clarify the settings and comparison criterion of the existing throughput experiments.
  • Section 5.5, Figure 7: We replace the original Table 1 with a quality–TTFT comparison for GPU- and CPU-resident caches, comparing CacheTune with loading optimizations enabled against the baselines.

We also add brief clarifications and references to the relevant appendices in Sections 4.1–4.2.

Appendix

  • E.1, E.3, E.5: We add low-frequency reconstruction derivations, residual-error analysis, and weighting ablations.
  • E.7, Figure 14: We add a quality–TTFT comparison under a shared CPU loading pipeline.
  • E.8: We add evaluations on newer models, CRAG, FRAMES, and a GSM8K cross-agent workflow.
  • E.9: We add quality–TTFT comparisons with EPIC-32/64.

We thank the reviewers and chairs for their time and suggestions. We are happy to clarify any remaining questions and look forward to further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.