- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CORE, a KV-cache compression method that uses a shared allocation distribution for both retention and compensation. CORE distills an offline teacher combining query utility and log-determinant coverage into a lightweight indexer. At inference, Top-B selection determines retained KV states, while the excluded allocation mass and conditional weights control latent-memory writes. Experiments across three backbones show consistent average improvements under aggressive compression.
Critical strengths
- The mass–direction decomposition provides a clear rationale for coordinating KV selection and memory writing through one distribution. This design avoids inconsistent weighting between the two channels and eliminates a separate write-weight predictor.
- CORE achieves the highest RULER average in all evaluated 75% and 90% compression settings across three backbones and two context lengths. LongBench and LongBench-v2 subset provide additional evidence of effectiveness beyond the main synthetic benchmark.
- The paper examines coverage supervision, boundary loss, memory compensation, write weights, and independent allocations. Training-seed and training-source sensitivity analyses, together with memory-reconstruction diagnostics, strengthen the empirical support for the proposed components.
Critical weaknesses
- The student approximates a coverage-regularized teacher rather than the attention distribution of future queries. Table 6 illustrates this distinction: at 25% compression, excluded allocation mass is 0.120, whereas actual evicted attention mass is 0.3251. Theorem 2 accounts for this discrepancy through mismatch terms, but these terms limit its predictive usefulness.
- CORE’s shared scorer receives Stage I teacher distillation, whereas the independent write scorer is initialized only at Stage II and trained through reconstruction loss. Table 3 consequently does not fully isolate the benefit of sharing the allocation.
- Mass-independent stability applies to linear writes; the full recurrence and readout bounds still depend on normalization denominators. The analysis fixes queries and memory atoms, while the reconstruction diagnostic covers only four write events. These results do not capture the feedback through which generation errors alter subsequent queries and KV states.
Questions for authors
What precisely is the calibration target? Can the authors report allocation–attention mass discrepancies across layers, heads, and future queries, and explain how these discrepancies affect compensation quality?
Does CORE retain its advantage if the independent write scorer is initialized from the shared scorer and receives equivalent distillation supervision and training resources?
Can the authors report reconstruction error, correction alignment, and task accuracy as functions of eviction count during long generation, including a full-KV reference? Do the findings hold with Qwen’s thinking mode enabled?
5 comments
Table R1. Allocation–attention discrepancy and compensation after a single write.
| Dimension | Layer / head / subsequent token position | Mean excluded allocation mass | Mean true evicted attention mass | Mean absolute mass discrepancy | Compensation gain (%) ↑ |
|---|---|---|---|---|---|
| Layer | Layer 4 | 0.1106 | 0.2866 | 0.1772 | 32.61 |
| Heads | Layer 4: 1 / 9 / 17 / 25 | Same as above | 0.2879 / 0.2774 / 0.2960 / 0.3059 | 0.1794 / 0.1682 / 0.1859 / 0.1957 | 26.29 / 29.78 / 33.03 / 33.67 |
| Layer | Layer 12 | 0.1255 | 0.3585 | 0.2338 | 32.46 |
| Heads | Layer 12: 1 / 9 / 17 / 25 | Same as above | 0.3517 / 0.3598 / 0.3698 / 0.3512 | 0.2275 / 0.2345 / 0.2452 / 0.2260 | 31.36 / 31.38 / 32.52 / 33.35 |
| Layer | Layer 20 | 0.1167 | 0.3105 | 0.1949 | 33.24 |
| Heads | Layer 20: 1 / 9 / 17 / 25 | Same as above | 0.3185 / 0.3040 / 0.3107 / 0.3156 | 0.2019 / 0.1879 / 0.1966 / 0.1993 | 32.64 / 32.39 / 34.99 / 33.36 |
| Layer | Layer 28 | 0.1272 | 0.3448 | 0.2185 | 32.01 |
| Heads | Layer 28: 1 / 9 / 17 / 25 | Same as above | 0.3382 / 0.3408 / 0.3539 / 0.3438 | 0.2113 / 0.2140 / 0.2278 / 0.2173 | 32.67 / 33.37 / 31.82 / 32.85 |
| Position | Tokens 1–4 | 0.1200 | 0.3142 | 0.1954 | 31.10 |
| Position | Tokens 5–8 | 0.1200 | 0.3301 | 0.2110 | 30.19 |
| Position | Tokens 9–12 | 0.1200 | 0.3215 | 0.2026 | 30.67 |
| Position | Tokens 13–16 | 0.1200 | 0.3346 | 0.2155 | 29.82 |
Despite discrepancies between excluded allocation mass and true evicted attention mass, compensation yields positive average reconstruction gains across the reported layers, query heads, and subsequent query intervals, with a 32.58% mean groupwise relative squared-error reduction. Mismatch may affect written information’s relevance to later queries, but mass discrepancy alone does not determine compensation quality: query-dependent readout and gating use stored information to reconstruct residuals. These results show that CORE remains effective in reducing reconstruction error under the evaluated mismatch conditions.
Following your suggestion, we have clarified the term at its first occurrence, explicitly identified the calibration target in Sec. 3.1, and extended the original diagnostic evaluation to include results across layers, heads, and subsequent query positions in Appendix C.2.
W2/Q2. Shared allocation under aligned initialization and supervision
Inspired by your constructive suggestion, we added a comparison to more fully isolate the benefit of sharing the allocation. We initialized the independent write scorer from the shared scorer and provided equivalent teacher-allocation distillation supervision and comparable training resources. Evaluation followed the Llama-3.1-8B setting in Table 3, using RULER at 4K and 16K context lengths with 75% and 90% KV eviction ratios.
Table R2. Shared versus independent writing with aligned initialization and teacher supervision.
| Variant | RULER-4K / 75% ↑ | RULER-4K / 90% ↑ | RULER-16K / 75% ↑ | RULER-16K / 90% ↑ |
|---|---|---|---|---|
| CORE: shared allocation | 80.60 | 62.44 | 74.98 | 68.48 |
| Original independent writer | 79.37 | 61.82 | 73.02 | 67.43 |
| Independent writer with aligned initialization and teacher supervision | 79.55 | 62.04 | 73.41 | 67.85 |
The aligned independent writer improves over the original variant, while CORE retains better task performance. These results indicate that the original performance gap is not entirely attributable to differences in supervision and further support the effectiveness of shared allocation. We thank the reviewer again for helping us strengthen this key comparison.
To strengthen the original comparison in Table 3, we have added the comparison with aligned initialization and supervision to Sec. 5.5, with training details in Appendix E.4.
W3/Q3. Stability and compensation during long autoregressive generation
We sincerely thank you for your thoughtful and constructive feedback, which highlights an important direction for deepening our theoretical analysis. We respond from two perspectives:
-
Scope of the existing stability analysis.
Changes in the current correction may affect subsequent queries and KV states, altering later allocations and memory updates. Accounting for feedback between memory perturbations and evolving generation states would better characterize long-generation stability. As you noted, analysis with common candidate memory atoms and fixed queries does not capture this feedback.
[continued below]
The original analysis uses these conditions to isolate the effects of allocation perturbations. Proposition 1 shows that mass-preserving linear writes avoid inverse-mass amplification from conditional normalization; Corollary 8 (Appendix B.5.2) covers repeated memory updates and normalized readout, retaining denominator dependencies. We also considered extending this analysis to include generation feedback, but small output perturbations can change discrete token choices, complicating bounds on subsequent state divergence and its long-term accumulation. Appendix F of our original submission already identifies full autoregressive trajectory analysis as future work. We therefore combine this conditional analysis of the benefits of mass-preserving writing with actual autoregressive evaluation, where CORE’s task-performance advantages under repeated eviction provide initial empirical support for the design.
-
Existing generation evaluations and additional process diagnostics.
Table 2 already evaluates repeated eviction during actual autoregressive generation. With Qwen3-14B on AIME25 and Math500, using 75% initial eviction and four recompression intervals, CORE achieves the highest accuracy among the compared methods in all eight settings. These results provide task-level evidence under evolving queries and KV states.
Inspired by your constructive suggestion, we report reconstruction error, correction alignment, and accuracy across cumulative eviction counts to assess sustained compensation during long generation and its relation to task performance.
We evaluate Qwen3-14B on all 500 Math500 problems in both thinking modes, with one response per problem per mode and a 16,384-token limit. In this additional diagnostic, CORE initially evicts 25% of KV entries and fixes the retention budget at the number of entries remaining after this compression. The cache is then recompressed to this budget every 128 generated tokens. Counting initial compression as eviction 1, we record diagnostics at stages 4/16/64/128 and compare CORE and full-KV accuracy at matched token budgets at stage ends, before the next compression. At each checkpoint, reconstruction diagnostics include only problems that reach that checkpoint and have valid post-eviction queries, using up to 16 queries per problem. Same-query reference outputs are computed using the complete historical KV archive from the same generation trajectory. All three reconstruction diagnostics follow the definitions, grouping, and aggregation protocol in Appendix C.2. Accuracy covers all 500 problems, retaining final answers for early-finished samples and counting those without valid answers at the token cutoff as incorrect.
Table R3. Compensation and task performance under repeated eviction on Math500.
Thinking mode Eviction stage mean relative squared reconstruction error ↓ mean correction alignment ↑ mean relative correction magnitude CORE accuracy (%) ↑ full-KV accuracy (%) ↑ Disabled 4 0.4517 0.8723 0.4190 33.0 54.2 16 0.4089 0.8803 0.4932 52.0 83.6 64 0.3837 0.8846 0.5134 66.0 87.8 128 0.3703 0.8891 0.5177 70.4 87.8 Enabled 4 0.4051 0.8896 0.4551 5.4 6.0 16 0.3701 0.9040 0.4941 37.8 42.0 64 0.3262 0.9128 0.5570 68.6 89.0 128 0.3142 0.9180 0.5624 74.8 94.8 Across the sampled checkpoints, compensation consistently lowers the mean relative squared reconstruction error compared with the uncompensated output, while mean correction alignment remains high. Together, these observations indicate that CORE’s accumulated memory remains useful for correcting current eviction residuals after repeated updates. This extends the reconstruction finding in Appendix Fig. 7 to long autoregressive generation, where generation errors can alter subsequent queries and KV states, supporting continued compensation effectiveness under these evolving conditions. This finding also holds with Qwen’s thinking mode enabled.
Building on the original repeated-eviction evaluation, we have added long-generation diagnostics under both thinking modes to Appendix C.2 and clarified the scope of the stability analysis in Sec. 4 and Appendix F.
We sincerely appreciate the care you have taken in examining both the strengths and the boundaries of CORE. Your feedback provides concrete ways to improve the calibration explanation, the sharing comparison, and the evaluation of long-generation behavior. We hope these clarifications and the completed, verified additions will make the scope and evidence easier to assess, and we would be glad to clarify any remaining points.
Dear Reviewer rcj1,
We sincerely thank you for your insightful and constructive feedback. We are grateful for your recognition of our design rationale, strong results, and thorough evaluation. Your thoughtful questions help us clarify and strengthen our work. To facilitate your review, we summarize our responses and revisions below.
W1. Allocation–attention mismatch and the predictive value of Theorem 2
Thank you for your insightful comments. Your observations are well reasoned and theoretically substantive, highlighting a key concern about our analysis. We respond as follows:
Purpose and scope of Theorem 2.
Theorem 2 is intended to provide a theoretical analysis of the expected pre-compensation output error, with an upper bound that holds under the stated assumptions. As with generalization-error bounds in machine learning, it provides a theoretical constraint on error rather than an exact prediction of future error. As noted in Sec. 4 and App. B.2.2 of our original submission, unknown mean future attention limits direct online evaluation, and the empirical bound does not establish generalization to unobserved queries.
Further results supporting error analysis in the original Table 6.
Thank you for your careful attention to Table 6 of our original submission (Table 8 in the revised manuscript). The table reports the mismatch-adjusted bound, measured error, and their ratio, alongside the allocation–attention discrepancy. Across the four compression settings, the aggregated normalized bounds are 0.370–0.930, below the generic normalized upper bound of 1, while the measured-error-to-bound ratios are 0.580–0.792. These results show that, in the evaluated settings, the bound still provides a tighter aggregate error constraint than the generic bound after accounting for mismatch, supporting the bound’s value for offline error analysis despite the allocation–attention discrepancy.
We sincerely thank you for this insightful and forward-looking perspective. Your emphasis on the practical predictive value of theoretical analysis provides valuable guidance for our further algorithmic and theoretical development. We have explicitly added improving the theory’s predictive usefulness and strengthening its role in guiding algorithm design as key directions in the “Limitations” section of the revised manuscript. Thank you for helping us improve the current manuscript and identify valuable directions for continued research.
Accordingly, building on Theorem 2 and the original Table 6 audit, we have expanded the scope discussion in Sec. 4 and Appendix C.1 and added future research directions to Appendix F.
Q1. Calibration target and the effect of allocation–attention mismatch on compensation
Thank you for your careful review. The calibration target is the teacher allocation combining query utility and set-level log-determinant coverage, with entropy regularization. We have clarified the term at its first occurrence and explicitly identified the calibration target in Sec. 3.1.
Inspired by your suggestion, we added a diagnostic experiment on allocation–attention mass discrepancies and compensation gains. Following Appendix C, we evaluated Llama-3.1-8B on 104 RULER-16K sequences, each with one 25% KV eviction, reset memory, and allocation based only on pre-eviction information. Measurements cover layers 4/12/20/28 and the next 16 original-sequence tokens, without further eviction. Aggregates include all heads; head-level rows report preselected heads 1/9/17/25, and query positions are grouped into four-token intervals. All indices are one-based.
Compensation gain is the percentage reduction in squared reconstruction error summed over the specified heads and positions, computed per sequence–layer group before aggregation. Overall results cover all 16 positions.
[continued below]