- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper finds that the low-frequency part of the KV cache is very helpful for cross-chunk dependency recovery, and thus proposes CacheTune to enable non-prefix KV cache reuse. CacheTune uses a dynamic recomputation ratio to further reduce the latency of KV cache loading.
Critical strengths
- Figure 1 shows that recomputing the KV cache for low-frequency indices can achieve higher KV cache similarity, attention distribution, and accuracy. Based on this observation, the paper designed CacheTune.
- CacheTune determines the recomputation ratio based on the medium where the KV cache is stored.
Critical weaknesses
- The paper does not explain why low-frequency indices are more worthy of being recomputed, and the final experiments show that CacheTune does not have a significant accuracy advantage over other methods.
- The experimental setup of EPIC recomputes only the first 16 tokens, but the EPIC paper reports that recomputing the first 32 tokens yields better results. The accuracy and TTFT of EPIC-32 may be superior to those of CacheTune. Additionally, subsequent TTFT experiments did not compare against EPIC.
- The token indices recomputed by CacheTune are independent of the query, but different queries may require the recomputation of different tokens. For example, query-aware methods such as Quest and SnapKV outperform query-agnostic sparse attention methods.
Questions for authors
- Why are low-frequency tokens worth recomputing, rather than high-frequency tokens or tokens with higher attention scores?
- How does CacheFuse compare to EPIC-32 and EPIC-64?
4 comments
Part 1/3: Point-by-point responses (continued)
W2 & Q2: Comparison with EPIC-32 and EPIC-64
Thank you for your suggestion.
Our comparison with EPIC-16 follows the default configuration in the system performance evaluation of the original EPIC paper. It also provides a moderate recomputation scope that balances generation quality and recomputation cost. We have added experiments with EPIC-32 and EPIC-64, as shown below.
Each cell reports “generation quality / TTFT speedup”; the Full Recompute column reports generation quality only.
| Model | Dataset | Full Recompute | CacheTune | EPIC-16 | EPIC-32 | EPIC-64 |
|---|---|---|---|---|---|---|
| Mistral | SAMSum | 42.93 | 42.55 / 4.02× | 37.99 / 4.81× | 38.92 / 3.13× | 38.98 / 1.87× |
| WikiMQA | 32.71 | 30.02 / 4.22× | 27.12 / 5.94× | 27.53 / 4.53× | 27.51 / 2.63× | |
| MuSiQue | 37.50 | 35.50 / 4.21× | 29.15 / 5.53× | 29.17 / 4.06× | 29.20 / 2.34× | |
| Llama | SAMSum | 40.21 | 37.92 / 3.94× | 33.74 / 4.79× | 34.64 / 3.23× | 34.57 / 1.93× |
| WikiMQA | 25.61 | 23.74 / 4.07× | 21.05 / 6.04× | 22.47 / 4.85× | 22.41 / 2.94× | |
| MuSiQue | 23.61 | 21.57 / 4.12× | 17.95 / 4.71× | 17.13 / 3.79× | 18.09 / 2.29× |
Increasing EPIC's budget to 32/64 tokens yields limited and inconsistent quality gains, while its TTFT speedup continues to decrease. CacheTune still achieves higher generation quality than EPIC-32/64. EPIC-32 has lower TTFT on WikiMQA for both models; CacheTune has lower TTFT in the other four settings and in all comparisons with EPIC-64, providing a more favorable quality–latency trade-off.
Analysis: EPIC mainly recomputes tokens at the beginning of each cached chunk to mitigate the attention-sink effect introduced by independent encoding. This fixed block-start selection is simple and efficient, but may miss other positions within the chunk that are strongly affected by context. Expanding the block-start recomputation region may therefore fail to cover these positions effectively.
Subsequent storage-tier experiments: The subsequent TTFT experiments with CPU/SSD/HDD caches primarily evaluate cache offloading and loading optimizations, so they focus on methods with corresponding implementations. The original EPIC implementation does not provide a multi-tier offloading pipeline and was therefore not included in these experiments.
Manuscript revisions: Added comparisons with EPIC-32/64 in Appendix E.9.
W3: Can query-independent selection work across different queries?
Thank you for your question. Our original experiments already validated CacheTune's effectiveness across queries and compared it with a query-aware non-prefix reuse method:
- Effectiveness across queries: Figure 1(c) in the original manuscript (Figure 2(c) in the revision) and Appendix E.2 specifically evaluated the effectiveness of the same offline-selected recomputation positions across different queries.
- Comparison with a query-aware method: Figure 3 of the original manuscript (Figure 4 in the revision) compares CacheTune with ProphetKV. The experimental results show that CacheTune achieves an average relative improvement of 4.84% in generation quality over ProphetKV. Because query-aware selection introduces more online overhead, CacheTune also achieves lower TTFT.
CacheTune is effective across queries because it repairs cross-chunk dependencies missing from independently encoded KV Chunks:
- Select recomputation positions offline; update KVs online. CacheTune uses the low-frequency structure of KVs to prioritize positions that are more likely to exhibit large drift when the context changes. The positions are fixed offline, while the repaired KVs are generated online using the current context.
- Different queries can benefit from recovering the same dependencies. Although different queries seek different information, they may all rely on cross-chunk attention omitted during independent encoding. The same selected positions can therefore serve multiple queries.
- Preserve query-specific information access. CacheTune retains KVs at all positions, so the model can still use query-dependent attention to access the information each query needs. This combines cache repair across queries with information access tailored to a specific query.
Part 2/3: Justification of the relationship between low-frequency energy and KV reconstruction error (W1 & Q1 continued)
1. Image energy and reconstruction error
Image transform coding represents an image as a weighted sum of a complete orthonormal frequency basis: , where is a frequency basis element and is the corresponding projection coefficient.
Orthonormality means that distinct basis elements have zero inner product and each satisfies . Image energy is the sum of squared pixel magnitudes. The energy of the -th frequency component is therefore
\begin{aligned} \\|a_k\Phi_k\\|_F^2 &=\sum_{p,q}|a_k\Phi_k(p,q)|^2\\\\ &=|a_k|^2\sum_{p,q}|\Phi_k(p,q)|^2\\\\ &=|a_k|^2\\|\Phi_k\\|_F^2 =|a_k|^2, \end{aligned} \tag{1}
Here, ranges over image pixels. Equation (1) identifies each coefficient's squared magnitude as its component's energy. When components are summed, orthogonality cancels cross terms between frequencies, giving Parseval's energy identity .
This relationship lets us directly calculate the error of low-frequency reconstruction. Retaining the original coefficients in the low-frequency set gives . Let denote the fraction of total image energy contained in these components. The retained components cancel in , so the residual consists exactly of the discarded frequencies. Applying the same Parseval identity to this residual gives
\begin{aligned} \rho_I &:=\frac{\sum_{k\in\Omega_L}|a_k|^2} {\sum_k|a_k|^2},\\\\[3pt] \frac{\\|I-I_L\\|_F^2}{\\|I\\|_F^2} &=\frac{\sum_{k\notin\Omega_L}|a_k|^2} {\sum_k|a_k|^2}\\\\ &=1-\frac{\sum_{k\in\Omega_L}|a_k|^2} {\sum_k|a_k|^2} =1-\rho_I. \end{aligned} \tag{2}
Thus, the retained low-frequency energy fraction directly determines relative squared reconstruction error. When is close to 1, the error relative to the original image is small; under the same frequency budget, retaining a band with more energy gives a smaller error.
2. Extending image reconstruction to KV reconstruction
This derivation extends to KVs because the energy relationship depends on orthonormal transforms preserving squared norms, regardless of whether the signal is a two-dimensional image or a one-dimensional sequence. The unitary Fourier transform along the token axis has the same property. Let the Key or Value tensor at one layer of an independently encoded chunk be , where is the number of tokens and is the number of channels after combining all heads. Keys use their pre-RoPE representations. Transforming along the token axis gives , and retaining low frequencies followed by the inverse transform gives . Here, is the conjugate transpose, and is a diagonal mask retaining the corresponding conjugate-symmetric low-frequency set. The reconstruction still has the original positions.
For each channel , is a one-dimensional sequence of length . Applying Parseval's identity separately to this sequence and its low-pass reconstruction residual gives
\begin{aligned} \\|X_{:,c}\\|_2^2 &=\sum_k|\widehat X_{k,c}|^2,\\\\ \\|X_{:,c}-(X_L)_{:,c}\\|_2^2 &=\sum_{k\notin\Omega_L}|\widehat X_{k,c}|^2. \end{aligned} \tag{3}
The two lines give the channel's total energy and reconstruction error from discarded frequencies. The squared Frobenius norm sums the channels' squared L2 norms, so summing total and error energy over channels gives the overall KV reconstruction error. With denoting the retained low-frequency energy fraction, we obtain
\begin{aligned} \rho_X &:=\frac{\sum_{c=1}^{d}\sum_{k\in\Omega_L}|\widehat X_{k,c}|^2} {\sum_{c=1}^{d}\sum_k|\widehat X_{k,c}|^2},\\\\[3pt] \frac{\\|X-X_L\\|_F^2}{\\|X\\|_F^2} &=\frac{\sum_{c=1}^{d}\\|X_{:,c}-(X_L)_{:,c}\\|_2^2} {\sum_{c=1}^{d}\\|X_{:,c}\\|_2^2}\\\\ &=\frac{\sum_{c=1}^{d}\sum_{k\notin\Omega_L}|\widehat X_{k,c}|^2} {\sum_{c=1}^{d}\sum_k|\widehat X_{k,c}|^2} =1-\rho_X. \end{aligned} \tag{4}
Equations (2) and (4) have the same form: images are decomposed along spatial coordinates and KVs along token coordinates, and in both cases the discarded spectral energy equals the squared reconstruction error.
Revised §3, Figure 1 (original Appendix E.1, Figure 9) shows low-frequency energy concentration in KVs. Equation (4) explains why low-pass filtering can remove rapid variations while retaining the independent cache's main energy structure.
Part 3/3: Justification for selecting low-frequency positions for recomputation (W1 & Q1 continued)
1. Frequency-band reconstruction experiments
To test whether this reconstruction advantage preserves document semantics, we conduct frequency-band reconstruction experiments with Mistral v0.3-7B on SAMSum and REALSumm as additional evidence in this response.
We retain approximately 50% of either the low-frequency or high-frequency components of the target document or dialogue KVs, reconstruct the KVs through the inverse transform, and use them for summarization, with the complete KVs as the reference. Both groups retain the DC (mean) component and use the same number of non-DC frequency degrees of freedom, so the comparison focuses on the effect of frequency-band selection.
We measure KV reconstruction error using centered SNR and evaluate the resulting summaries using ROUGE-L. SNR is calculated after subtracting each channel's mean along the token axis; higher values indicate smaller reconstruction errors.
| Dataset | KV reconstruction | K SNR (dB) | V SNR (dB) | ROUGE-L (0–100) |
|---|---|---|---|---|
| SAMSum | Complete KVs | -- | -- | 43.56 |
| Low-frequency | 12.32 | 11.59 | 39.47 | |
| High-frequency | 1.48 | 1.78 | 15.14 | |
| REALSumm | Complete KVs | -- | -- | 24.06 |
| Low-frequency | 13.49 | 12.39 | 22.57 | |
| High-frequency | 1.29 | 1.66 | 12.20 |
With matched frequency degrees of freedom, low-frequency reconstruction yields K/V SNR values approximately higher than high-frequency reconstruction, and summary quality is closer to that of complete KVs. SNR measures representation error; summary quality tests whether reconstructed representations still support summarizing document content. Both support the role of low-frequency structure in preserving content and summary quality.
2. From low-frequency scores to recomputation positions
The preceding derivation and experiments motivate extracting low-frequency structure. We next test whether it identifies positions worth recomputing.
In non-prefix reuse, the discrepancy to repair is between the independently encoded cache and full-context encoding . Comparing the same token in the same representation coordinates, we denote its context-induced drift by . Offline selection must rank positions before has been computed, using only . CacheTune scores each original position by the magnitude of its low-pass reconstruction. Using the low-pass projection operator , we have
\ell_i=(X_L)_i=\sum_{j=1}^{N}P_{ij}X_j, \qquad s_i=\\|\ell_i\\|_2. \tag{5}
Here, is the representation at original position after retaining low frequencies along the token axis and applying the inverse transform; measures its low-frequency magnitude. As in §4.1, K/V scores are combined and normalized within each layer, then aggregated across layers into shared recomputation indices.
“Low-frequency positions” means original token positions with large low-pass reconstruction magnitudes. Reconstruction supplies offline scores; online execution recomputes selected KVs and reuses the remaining original cache.
How selected positions relate to context-induced changes
To examine how the selected positions read added preceding context, we consider how this context updates target-token representations through self-attention and affects K/V in subsequent layers. On SAMSum and MuSiQue, we fix the target chunk, preceding context, and 15% selection budget, select positions with the highest low-pass or high-pass scores from the independent cache, then measure attention to distant preceding context and normalized K/V drift under full-context encoding.
| Dataset | Selection | Attention to distant preceding context | K drift | V drift |
|---|---|---|---|---|
| SAMSum | Low-pass | 70.51% | 0.8347 | 0.8760 |
| High-pass | 15.72% | 0.3340 | 0.3791 | |
| MuSiQue | Low-pass | 75.89% | 0.8500 | 0.9152 |
| High-pass | 19.68% | 0.4189 | 0.5368 |
On both datasets, low-pass-selected positions allocate more attention to distant preceding context and exhibit greater K/V drift, reflecting larger context-related changes in their cached representations. This attention is used for post-hoc analysis of context dependencies; the indices are determined before full-context encoding.
Low-frequency reconstruction magnitudes can be computed from the independent cache and, in the comparisons above, prioritize positions with stronger interactions with preceding context and larger context-induced drift, supporting their use for offline selection. Figure 2 of the paper (original Figure 1) further shows that, under the same recomputation budget, updating these positions restores the full-context attention distribution and answer quality more effectively than high-frequency selection.
Response to Reviewer mjWG - Part 1/3: Point-by-point responses
Thank you for your questions and suggestions. In the revised manuscript, we have added an explanation of low-frequency selection and comparisons with EPIC-32/64. We first address your comments point by point, then present the detailed reasoning behind low-frequency selection in Parts 2/3 and 3/3. We hope this addresses all your questions.
W1 & Q1: Why select low-frequency positions, and how much quality does CacheTune gain over existing methods?
Thank you for your question. CacheTune is inspired by image and signal reconstruction: when image energy is concentrated at low frequencies, retaining these components yields a small relative squared reconstruction error. Revised §3, Figure 1 (original Appendix E.1, Figure 9) shows that independently encoded K/V also concentrate their energy at low frequencies along the token axis. We therefore use this low-frequency structure to guide recomputation of a small set of positions and restore cross-chunk context dependencies missing from independent encoding.
To let you read our responses to the other concerns first, we present the detailed reasoning behind recomputing low-frequency positions in Parts 2/3 and 3/3:
We first address your concerns about CacheTune's limited gains over existing methods and why we do not use attention scores to select recomputation indices.
(1) Generation quality gains over existing methods
Figure 4 of the revised manuscript (Figure 3 in the original) compares generation quality across three models and five datasets. The table below reports CacheTune's gains over each method.
The gains over CacheSlide are smaller, while CacheTune achieves higher generation quality than the other methods listed in the table in all evaluated configurations.
In non-prefix reuse, cross-chunk attention inevitably arises, so we cannot fully recover the model's inference accuracy. Recomputed KVs are also affected by errors that propagate across layers from the remaining, directly reused KVs, further reducing accuracy.
In our experiments, CacheTune retains approximately 95% of Full Recompute's generation quality on average by selecting only 15% of token positions offline for recomputation, bringing its quality close to full recomputation.
The compared methods already achieve good accuracy, leaving limited theoretical room for improvement, yet CacheTune still achieves higher generation quality. We consider it valuable to address the complex problem of non-prefix reuse with a lightweight but effective method such as CacheTune.
(2) Why not use attention scores directly?
We distinguish between two sources of attention scores:
Thus, selecting recomputation positions directly from attention scores makes it difficult to achieve both effective selection and low online overhead.
Manuscript revisions: Moved the KV energy experiment to §3, Figure 1, and added the derivation from image reconstruction to KV reconstruction in Appendix E.1.