acceptodds
Under review as a conference paper at ICLR 2027

CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache

10 views

Abstract

Long-context decoding is increasingly constrained by key–value (KV) cache memory and bandwidth. Existing fixed-budget compression methods typically separate retention from compensation, while a retention ranking specifies neither discarded attention mass nor the direction of induced output error. We start from an exact factorization: eviction error equals evicted attention times the between the evicted centroid and retained output, highlighting the importance of set-level coverage in retention and mass-preserving memory writing. We introduce (verage Calibration and Evicted-Mass distribution for KV Cache), which distills an offline allocation combining query utility and log-determinant coverage into a lightweight cache-aware indexer. At inference, one calibrated distribution drives both channels: its Top- ordering retains complementary KV states, while its excluded allocation mass and conditional weights parameterize latent-memory writes without a separate write-weight predictor or online log-determinant evaluation. Our analysis provides a four-term pre-compensation error certificate, characterizes non-additive coverage interactions, and establishes mass-independent write stability with hierarchical bounds through recurrent and query-adaptive normalization. Across three backbones, CORE exceeds the strongest RULER baseline by up to 3.78 points at 90% compression; LongBench and repeated-eviction evaluations further demonstrate strong effectiveness and decoding efficiency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache eviction, coverage calibration, latent Memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rcj1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CORE, a KV-cache compression method that uses a shared allocation distribution for both retention and compensation. CORE distills an offline teacher combining query utility and log-determinant coverage into a lightweight indexer. At inference, Top-B selection determines retained KV states, while the excluded allocation mass and conditional weights control latent-memory writes. Experiments across three backbones show consistent average improvements under aggressive compression.

Critical strengths
  1. The mass–direction decomposition provides a clear rationale for coordinating KV selection and memory writing through one distribution. This design avoids inconsistent weighting between the two channels and eliminates a separate write-weight predictor.
  2. CORE achieves the highest RULER average in all evaluated 75% and 90% compression settings across three backbones and two context lengths. LongBench and LongBench-v2 subset provide additional evidence of effectiveness beyond the main synthetic benchmark.
  3. The paper examines coverage supervision, boundary loss, memory compensation, write weights, and independent allocations. Training-seed and training-source sensitivity analyses, together with memory-reconstruction diagnostics, strengthen the empirical support for the proposed components.
Critical weaknesses
  1. The student approximates a coverage-regularized teacher rather than the attention distribution of future queries. Table 6 illustrates this distinction: at 25% compression, excluded allocation mass is 0.120, whereas actual evicted attention mass is 0.3251. Theorem 2 accounts for this discrepancy through mismatch terms, but these terms limit its predictive usefulness.
  2. CORE’s shared scorer receives Stage I teacher distillation, whereas the independent write scorer is initialized only at Stage II and trained through reconstruction loss. Table 3 consequently does not fully isolate the benefit of sharing the allocation.
  3. Mass-independent stability applies to linear writes; the full recurrence and readout bounds still depend on normalization denominators. The analysis fixes queries and memory atoms, while the reconstruction diagnostic covers only four write events. These results do not capture the feedback through which generation errors alter subsequent queries and KV states.
Questions for authors

What precisely is the calibration target? Can the authors report allocation–attention mass discrepancies across layers, heads, and future queries, and explain how these discrepancies affect compensation quality?

Does CORE retain its advantage if the independent write scorer is initialized from the shared scorer and receives equivalent distillation supervision and training resources?

Can the authors report reconstruction error, correction alignment, and task accuracy as functions of eviction count during long generation, including a full-KV reference? Do the findings hold with Qwen’s thinking mode enabled?

5 comments

Dear Reviewer rcj1,

We sincerely thank you for your insightful and constructive feedback. We are grateful for your recognition of our design rationale, strong results, and thorough evaluation. Your thoughtful questions help us clarify and strengthen our work. To facilitate your review, we summarize our responses and revisions below.

ConcernMain responseRevision locations
W1. Mismatch terms and predictive usefulnessExplicit theoretical scope; Mismatch-adjusted bounds remain tighter than the generic bound in the evaluated aggregates.Sec. 4; Apps. C.1, F
Q1. Calibration target and compensation qualityCalibration targets the teacher allocation; compensation yields gains despite mismatch.Introduction; Sec. 3.1; App. C.2
W2/Q2. Benefit of sharing the allocationCORE retains its advantage under aligned training conditions.Sec. 5.5; App. E.4
W3/Q3. Long generation and Qwen’s thinking modeExplicit stability scope; Compensation yields positive average gains at tested checkpoints in both thinking modes.Sec. 4; Apps. C.2, F

W1. Allocation–attention mismatch and the predictive value of Theorem 2

Thank you for your insightful comments. Your observations are well reasoned and theoretically substantive, highlighting a key concern about our analysis. We respond as follows:

  • Purpose and scope of Theorem 2.

    Theorem 2 is intended to provide a theoretical analysis of the expected pre-compensation output error, with an upper bound that holds under the stated assumptions. As with generalization-error bounds in machine learning, it provides a theoretical constraint on error rather than an exact prediction of future error. As noted in Sec. 4 and App. B.2.2 of our original submission, unknown mean future attention limits direct online evaluation, and the empirical bound does not establish generalization to unobserved queries.

  • Further results supporting error analysis in the original Table 6.

    Thank you for your careful attention to Table 6 of our original submission (Table 8 in the revised manuscript). The table reports the mismatch-adjusted bound, measured error, and their ratio, alongside the allocation–attention discrepancy. Across the four compression settings, the aggregated normalized bounds are 0.370–0.930, below the generic normalized upper bound of 1, while the measured-error-to-bound ratios are 0.580–0.792. These results show that, in the evaluated settings, the bound still provides a tighter aggregate error constraint than the generic bound after accounting for mismatch, supporting the bound’s value for offline error analysis despite the allocation–attention discrepancy.

We sincerely thank you for this insightful and forward-looking perspective. Your emphasis on the practical predictive value of theoretical analysis provides valuable guidance for our further algorithmic and theoretical development. We have explicitly added improving the theory’s predictive usefulness and strengthening its role in guiding algorithm design as key directions in the “Limitations” section of the revised manuscript. Thank you for helping us improve the current manuscript and identify valuable directions for continued research.

Accordingly, building on Theorem 2 and the original Table 6 audit, we have expanded the scope discussion in Sec. 4 and Appendix C.1 and added future research directions to Appendix F.


Q1. Calibration target and the effect of allocation–attention mismatch on compensation

Thank you for your careful review. The calibration target is the teacher allocation combining query utility and set-level log-determinant coverage, with entropy regularization. We have clarified the term at its first occurrence and explicitly identified the calibration target in Sec. 3.1.

Inspired by your suggestion, we added a diagnostic experiment on allocation–attention mass discrepancies and compensation gains. Following Appendix C, we evaluated Llama-3.1-8B on 104 RULER-16K sequences, each with one 25% KV eviction, reset memory, and allocation based only on pre-eviction information. Measurements cover layers 4/12/20/28 and the next 16 original-sequence tokens, without further eviction. Aggregates include all heads; head-level rows report preselected heads 1/9/17/25, and query positions are grouped into four-token intervals. All indices are one-based.

Compensation gain is the percentage reduction in squared reconstruction error summed over the specified heads and positions, computed per sequence–layer group before aggregation. Overall results cover all 16 positions.

[continued below]

Table R1. Allocation–attention discrepancy and compensation after a single write.

DimensionLayer / head / subsequent token positionMean excluded allocation massMean true evicted attention massMean absolute mass discrepancyCompensation gain (%) ↑
LayerLayer 40.11060.28660.177232.61
HeadsLayer 4: 1 / 9 / 17 / 25Same as above0.2879 / 0.2774 / 0.2960 / 0.30590.1794 / 0.1682 / 0.1859 / 0.195726.29 / 29.78 / 33.03 / 33.67
LayerLayer 120.12550.35850.233832.46
HeadsLayer 12: 1 / 9 / 17 / 25Same as above0.3517 / 0.3598 / 0.3698 / 0.35120.2275 / 0.2345 / 0.2452 / 0.226031.36 / 31.38 / 32.52 / 33.35
LayerLayer 200.11670.31050.194933.24
HeadsLayer 20: 1 / 9 / 17 / 25Same as above0.3185 / 0.3040 / 0.3107 / 0.31560.2019 / 0.1879 / 0.1966 / 0.199332.64 / 32.39 / 34.99 / 33.36
LayerLayer 280.12720.34480.218532.01
HeadsLayer 28: 1 / 9 / 17 / 25Same as above0.3382 / 0.3408 / 0.3539 / 0.34380.2113 / 0.2140 / 0.2278 / 0.217332.67 / 33.37 / 31.82 / 32.85
PositionTokens 1–40.12000.31420.195431.10
PositionTokens 5–80.12000.33010.211030.19
PositionTokens 9–120.12000.32150.202630.67
PositionTokens 13–160.12000.33460.215529.82

Despite discrepancies between excluded allocation mass and true evicted attention mass, compensation yields positive average reconstruction gains across the reported layers, query heads, and subsequent query intervals, with a 32.58% mean groupwise relative squared-error reduction. Mismatch may affect written information’s relevance to later queries, but mass discrepancy alone does not determine compensation quality: query-dependent readout and gating use stored information to reconstruct residuals. These results show that CORE remains effective in reducing reconstruction error under the evaluated mismatch conditions.

Following your suggestion, we have clarified the term at its first occurrence, explicitly identified the calibration target in Sec. 3.1, and extended the original diagnostic evaluation to include results across layers, heads, and subsequent query positions in Appendix C.2.


W2/Q2. Shared allocation under aligned initialization and supervision

Inspired by your constructive suggestion, we added a comparison to more fully isolate the benefit of sharing the allocation. We initialized the independent write scorer from the shared scorer and provided equivalent teacher-allocation distillation supervision and comparable training resources. Evaluation followed the Llama-3.1-8B setting in Table 3, using RULER at 4K and 16K context lengths with 75% and 90% KV eviction ratios.

Table R2. Shared versus independent writing with aligned initialization and teacher supervision.

VariantRULER-4K / 75% ↑RULER-4K / 90% ↑RULER-16K / 75% ↑RULER-16K / 90% ↑
CORE: shared allocation80.6062.4474.9868.48
Original independent writer79.3761.8273.0267.43
Independent writer with aligned initialization and teacher supervision79.5562.0473.4167.85

The aligned independent writer improves over the original variant, while CORE retains better task performance. These results indicate that the original performance gap is not entirely attributable to differences in supervision and further support the effectiveness of shared allocation. We thank the reviewer again for helping us strengthen this key comparison.

To strengthen the original comparison in Table 3, we have added the comparison with aligned initialization and supervision to Sec. 5.5, with training details in Appendix E.4.


W3/Q3. Stability and compensation during long autoregressive generation

We sincerely thank you for your thoughtful and constructive feedback, which highlights an important direction for deepening our theoretical analysis. We respond from two perspectives:

  • Scope of the existing stability analysis.

    Changes in the current correction may affect subsequent queries and KV states, altering later allocations and memory updates. Accounting for feedback between memory perturbations and evolving generation states would better characterize long-generation stability. As you noted, analysis with common candidate memory atoms and fixed queries does not capture this feedback.

[continued below]

The original analysis uses these conditions to isolate the effects of allocation perturbations. Proposition 1 shows that mass-preserving linear writes avoid inverse-mass amplification from conditional normalization; Corollary 8 (Appendix B.5.2) covers repeated memory updates and normalized readout, retaining denominator dependencies. We also considered extending this analysis to include generation feedback, but small output perturbations can change discrete token choices, complicating bounds on subsequent state divergence and its long-term accumulation. Appendix F of our original submission already identifies full autoregressive trajectory analysis as future work. We therefore combine this conditional analysis of the benefits of mass-preserving writing with actual autoregressive evaluation, where CORE’s task-performance advantages under repeated eviction provide initial empirical support for the design.

  • Existing generation evaluations and additional process diagnostics.

    Table 2 already evaluates repeated eviction during actual autoregressive generation. With Qwen3-14B on AIME25 and Math500, using 75% initial eviction and four recompression intervals, CORE achieves the highest accuracy among the compared methods in all eight settings. These results provide task-level evidence under evolving queries and KV states.

    Inspired by your constructive suggestion, we report reconstruction error, correction alignment, and accuracy across cumulative eviction counts to assess sustained compensation during long generation and its relation to task performance.

    We evaluate Qwen3-14B on all 500 Math500 problems in both thinking modes, with one response per problem per mode and a 16,384-token limit. In this additional diagnostic, CORE initially evicts 25% of KV entries and fixes the retention budget at the number of entries remaining after this compression. The cache is then recompressed to this budget every 128 generated tokens. Counting initial compression as eviction 1, we record diagnostics at stages 4/16/64/128 and compare CORE and full-KV accuracy at matched token budgets at stage ends, before the next compression. At each checkpoint, reconstruction diagnostics include only problems that reach that checkpoint and have valid post-eviction queries, using up to 16 queries per problem. Same-query reference outputs are computed using the complete historical KV archive from the same generation trajectory. All three reconstruction diagnostics follow the definitions, grouping, and aggregation protocol in Appendix C.2. Accuracy covers all 500 problems, retaining final answers for early-finished samples and counting those without valid answers at the token cutoff as incorrect.

    Table R3. Compensation and task performance under repeated eviction on Math500.

    Thinking modeEviction stagemean relative squared reconstruction error Rˉ\bar R ↓mean correction alignment ρˉ\bar\rho ↑mean relative correction magnitude rˉ\bar rCORE accuracy (%) ↑full-KV accuracy (%) ↑
    Disabled40.45170.87230.419033.054.2
    160.40890.88030.493252.083.6
    640.38370.88460.513466.087.8
    1280.37030.88910.517770.487.8
    Enabled40.40510.88960.45515.46.0
    160.37010.90400.494137.842.0
    640.32620.91280.557068.689.0
    1280.31420.91800.562474.894.8

    Across the sampled checkpoints, compensation consistently lowers the mean relative squared reconstruction error compared with the uncompensated output, while mean correction alignment remains high. Together, these observations indicate that CORE’s accumulated memory remains useful for correcting current eviction residuals after repeated updates. This extends the reconstruction finding in Appendix Fig. 7 to long autoregressive generation, where generation errors can alter subsequent queries and KV states, supporting continued compensation effectiveness under these evolving conditions. This finding also holds with Qwen’s thinking mode enabled.

Building on the original repeated-eviction evaluation, we have added long-generation diagnostics under both thinking modes to Appendix C.2 and clarified the scope of the stability analysis in Sec. 4 and Appendix F.


We sincerely appreciate the care you have taken in examining both the strengths and the boundaries of CORE. Your feedback provides concrete ways to improve the calibration explanation, the sharing comparison, and the evaluation of long-generation behavior. We hope these clarifications and the completed, verified additions will make the scope and evidence easier to assess, and we would be glad to clarify any remaining points.

Reviewer_GCbgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a KV cache compression method named CORE, which can address the bottleneck of memory use and slow speeds caused by the KV cache during long-context LLM inference. By utilizing a single scoring distribution to simultaneously select retained KVs and compensate for the information of evicted KVs, CORE significantly outperforms existing baselines at a high compression ratio of 90% while maintaining decoding efficiency.

Critical strengths

(1) The authors provide a relatively rigorous theoretical derivation for KV cache compression error decomposition along with stability proofs, giving the method solid theoretical grounding. (2) The experiments are relatively comprehensive compared to related work, with evaluations conducted across multiple models and baselines.

Critical weaknesses

(1) The method requires training an additional cache-aware indexer for each model. Compared to training-free approaches, this is a limitation and a practical concern for real-world deployment. In the experiments, the indexer's architecture is very simple, the input feature space is hand-crafted, and it is trained only on LongAlpaca-12k. Since the authors did not evaluate CORE's performance in more complex yet common scenarios (such as multi-turn dialogues in agentic workflows), I am concerned about how the authors can guarantee the robustness and generalizability of CORE. Although academic research in this area frequently relies on the benchmarks and training data used in this paper, the context lengths and scenario complexities of current frontier models far exceed the scale covered here. Given that KV cache compression has been studied for several years and most of the relevant methods are actually not practical, I hope the authors can take the realities of frontier LLMs into greater consideration.

(2) The models evaluated by the authors all rely on the GQA architecture, without validation on models utilizing MLA (e.g., GLM-5.3-Flash) or CSA (e.g., DeepSeek-V4-Flash), which limits the scope of applicability. While GQA is indeed adopted by most mainstream models and commonly used across academic works in this field, validating the method on a wider range of architectures is crucial to determining whether CORE can be practically deployed in industry. I hope the authors can conduct further experiments to address this.

Questions for authors

The title in the uploaded PDF ("ONE DISTRIBUTION, TWO CHANNELS: COVERAGE-CALIBRATED KV RETENTION AND COMPENSATION") differs from the metadata title registered on OpenReview ("CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache"). While the abstracts match, please ensure they are synchronized in the revised PDF or camera-ready version.

2 comments

Dear Reviewer GCbg,

We sincerely thank you for your detailed and constructive review. We appreciate your positive assessment of our theoretical grounding and comprehensive evaluation. To save you time and make our responses easier to follow, we summarize our responses and revisions below.

ConcernMain responseRevision locations
W1. Training requirementsClarified offline adaptation and module reuse.App. D.1
W1. Training sources and practical workloadsHighlighted the training-source comparison and added BFCL multi-turn evaluation using existing CORE weights.Sec. 5.1; App. E.5; App. E.7
W2. Applicability beyond GQAEvaluated GLM-4.7-Flash with its native MLA cache and clarified the evaluation scope.App. D.1; App. E.7; App. F
Q1. Title consistencyUpdated the OpenReview title to match the PDF.OpenReview metadata

W1. Training cost and generalization to practical workloads

Thank you for your careful feedback on these practical considerations. We respectfully clarify that CORE’s offline-trained modules are reusable and that our original submission already included training-source comparisons. Inspired by your constructive suggestion, we have added a multi-turn evaluation, which further supports CORE’s applicability. We address your concerns below:

  • CORE is adapted offline, and trained modules can be reused.

    As you noted, CORE requires additional model adaptation compared with training-free methods. However, training is performed offline, and the resulting indexer and compensation modules can be reused across requests and deployment instances for the same backbone and compatible configuration, without retraining for each request or downstream task. For sustained model use under a limited cache budget, the initial training cost can be amortized over continued use, making this trade-off attractive when preserving task quality is particularly important. When quality requirements are less stringent, training-free methods offer some cost advantages.

  • Existing training-source comparisons show relatively stable aggregate performance.

    Your concern about dependence on the training source also encourages us to make the existing evidence more prominent. Table 14 in Appendix E.5 of our original submission already compares CORE trained separately on LongAlpaca-12k, OpenMathReasoning, and SWE-smith-trajectories, showing relatively stable aggregate performance across the tested general-text, mathematical-reasoning, and software-engineering sources. We have highlighted this comparison more clearly in the main discussion.

  • Additional multi-turn tool-use evaluation examines the generalization of existing weights.

    Our existing long-context and repeated-compression experiments examine long inputs and continued decoding. Your suggestion motivated us to evaluate multi-turn interactions more directly. We have therefore added a BFCL multi-turn tool-use evaluation with Qwen3-14B, reusing the existing CORE weights without retraining or tuning.

    Inspired by your constructive suggestion, we evaluate the multi_turn_base subset. CAPKV, IndexMem, and CORE each retain 3K token positions after compression, including protected tokens, and use the same compression triggers. All methods use identical prompt and generation configurations, execute tool calls, and incorporate tool feedback. Within each task, subsequent turns reuse the previously compressed cache and any associated compensation memory, processing only newly added inputs. The results are presented in Table R1.

    Table R1. BFCL basic multi-turn tool-use results with Qwen3-14B.

    MethodSuccessful tasksComplete-task success rate (%) ↑Task-averaged turn success rate (%) ↑
    Full KV cache80 / 20040.070.8
    CAPKV57 / 20028.558.6
    IndexMem63 / 20031.562.4
    CORE72 / 20036.067.9

    CORE improves complete-task success over CAPKV and IndexMem by 7.5 and 4.5 percentage points, respectively, and task-averaged turn success by 9.3 and 5.5 percentage points. These results show that CORE maintains an advantage over the evaluated compression baselines at both the complete-task and turn levels in multi-turn tool use, providing further evidence that its existing weights generalize to this interactive setting.

Thank you for encouraging us to connect the evaluation more closely to practical use. Longer sessions, concurrent users, and complex tool failures remain directions for further validation, and we would welcome collaboration with industry partners to evaluate CORE under these workloads.

[continued below]


W2. Applicability beyond GQA

Thank you for acknowledging the relevance of GQA and offering this concrete suggestion for broader architectural validation. We agree that evaluating emerging attention architectures is important for understanding CORE’s practical applicability.

Following your suggestion, we reviewed public deployment recipes for the models you mentioned, including configurations using four B200 GPUs for GLM-5.3-Flash and four H200 GPUs for the original DeepSeek-V4-Flash release. Reproducing these deployments and completing CORE’s cache adaptation, training, and controlled evaluation would require resources well beyond our current capacity. To provide direct evidence within our available resources, we have additionally evaluated CORE on GLM-4.7-Flash, which uses MLA, while preserving its native latent-cache representation.

We freeze the backbone and adapt the indexer and compensation modules using the original training data and two-stage training procedure. We evaluate RULER-4K and RULER-16K at 75% and 90% KV eviction rates. IndexMem and CORE use matched retained-cache budgets and identical evaluation settings, with recompression every 128 generated tokens. The results are presented in Table R2.

Table R2. MLA adaptation results on GLM-4.7-Flash.

MethodKV eviction rateRULER-4K average ↑RULER-16K average ↑
Full cache0%97.4495.83
IndexMem75%86.580.0
CORE75%90.887.0
IndexMem90%55.061.0
CORE90%64.070.0

At 75% eviction, CORE improves over IndexMem by 4.3 points on RULER-4K and 7.0 points on RULER-16K. At 90% eviction, the improvement increases to 9.0 points at both lengths. CORE thus achieves a consistent advantage across both context lengths, with larger gains under the tighter cache budget. These results support CORE’s applicability to the tested MLA model while preserving its native latent-cache representation.

Thank you for this thoughtful and technically informed suggestion. By connecting architectural validation with practical deployment needs, you have helped us broaden the evaluation and strengthen its relevance to both research and practice. We would welcome collaboration with industry and research teams to extend the evaluation to GLM-5.3-Flash, DeepSeek-V4-Flash, and other frontier models with greater computational resources and real deployment workloads.


Q1. Consistency between the PDF and OpenReview titles

Thank you for your careful attention to the submission details alongside your assessment of the technical content. We appreciate you pointing out this inconsistency. We initially registered the title “CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cache.” While refining the manuscript, we changed the PDF title to “ONE DISTRIBUTION, TWO CHANNELS: COVERAGE-CALIBRATED KV RETENTION AND COMPENSATION,” as it more directly conveys our central contribution: coordinating KV retention and memory compensation through a single coverage-calibrated distribution.

We sincerely apologize for overlooking the corresponding metadata update. We have now updated the OpenReview title to match the PDF. Thank you again for your thorough and professional review, which helped us identify and correct this oversight and improve the accuracy and consistency of our submission.


We appreciate your thoughtful review and practical perspective. Your suggestions have helped us clarify CORE’s adaptation requirements and extend its evaluation to multi-turn tool use and MLA. We hope these additions address your concerns and better demonstrate CORE’s practical value. We welcome further discussion and would be happy to clarify any remaining points.

Reviewer_bq7aOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that existing KV cache compression methods typically design information retention and post-eviction compensation separately, while importance ranking alone can retain redundant information and cannot adequately describe the amount and composition of the evicted information. An exact error decomposition shows that the attention output error caused by eviction depends on two factors: the total attention mass of the evicted entries and the difference between the evicted information and the retained output. Effective compression therefore requires not only retaining important information but also accounting for information complementarity and compensating for eviction losses accordingly. Based on this analysis, the paper proposes CORE, which learns a shared weight distribution that accounts for both importance and coverage. This distribution is used both to select the KV entries to retain and to write evicted information into memory, unifying retention and compensation. Across RULER, LongBench, and repeated-eviction experiments on three models, CORE performs particularly well under aggressive compression, improving the RULER score by up to 3.78 points over the strongest baseline at a 90% KV eviction rate.

Critical strengths
  1. A shared distribution unifies KV retention and eviction compensation. This establishes an explicit connection between retention and compensation without requiring an additional write-weight predictor. Comparisons between shared and independent distributions also support the effectiveness of this design.
  2. Set coverage modeling is incorporated into a lightweight retention policy. A teacher distribution combining relevance and a log-determinant coverage objective is distilled into a lightweight scorer. This enables online selection to use coverage and redundancy information within the cache without computing the log-determinant online, while achieving good performance under aggressive compression.
Critical weaknesses
  1. The design has practical value but is closer to an incremental improvement on existing frameworks. The error identity itself is insufficient to support strong claims of conceptual novelty.
  2. It is the analytical identity that is exact; the actual algorithm remains an approximate design inspired by that identity. The current theory is insufficient to attribute the empirical gains to a guaranteed error-minimization mechanism.
Questions for authors
  1. Under the same KV budget and with similar total true attention mass assigned to evicted entries, please compare output errors with and without the coverage objective to test whether the gains indeed come from information complementarity.
  2. Please provide the independent writer with the same teacher supervision as the shared distribution to distinguish whether the improvement comes from weight sharing itself or from differences in supervision.
  3. Please clarify that “coverage-calibrated” refers to allocation calibration under the coverage objective, rather than an accurate estimate of true attention for future queries or of eviction error.
4 comments

Dear Reviewer bq7a,

We sincerely thank you for your detailed and constructive review. We appreciate your recognition of our unified retention–compensation design and lightweight coverage-aware policy. We address your questions and suggestions below. To save you time and make our responses easier to follow, we summarize our responses and the corresponding revisions incorporated into the manuscript below.

ConcernMain responseRevision locations
W1. Contribution beyond existing frameworksClarified the role of the error identity and emphasized the core methodological contributions.Introduction
W2. Exact decomposition and error minimizationAdded a conditional analysis connecting learned allocation and actual memory correction to reconstruction-error reduction.App. B.6
Q1. Contribution of information complementarityAdded comparisons of coverage variants under the same KV budget and matched true evicted attention mass.Sec. 5.4; App. C.1
Q2. Benefit of shared allocationAdded a comparison showing that CORE retains its advantage under aligned initialization and teacher supervision.Sec. 5.5; App. E.4
Q3. Meaning of “coverage-calibrated”Explicitly defined calibration as allocation adjustment under the utility-and-coverage teacher objective.Introduction; Sec. 3.1

W1. Contribution beyond existing frameworks

We appreciate your recognition of the practical value of our design.We partly agree with your assessment, in that some aspects of CORE build on and extend existing approaches. Our core methodological contribution, however, is learning a coverage-aware distribution that jointly controls KV retention and memory writing. Under a fixed budget, a ranking alone determines the retained set but does not uniquely specify the weights for writing evicted information into memory. CORE distills query utility and set-level coverage into a lightweight indexer, using the resulting distribution’s ranking to select retained entries and its excluded probability mass and conditional weights to determine the strength and composition of memory writes.

The error identity follows directly from partitioning the normalized attention output into retained and evicted contributions, and its derivation is straightforward. We use it as an analytical starting point and a design principle, and do not regard the identity itself as a conceptual innovation.

Your impression that we claim conceptual novelty for the error identity itself may have arisen from unclear wording in our manuscript, particularly the first contribution statement in the Introduction of our original submission.

  • We start from an exact eviction-error identity that factorizes output deviation into evicted mass and directional gap, exposing the joint requirements of retention and compensation, showing normalized aggregation readouts satisfy this exact factorization, and establishing its converse characterization.

The original wording may not have clearly distinguished the analytical starting point from the subsequent analysis. We have revised it as follows.

  • Building on the mass–direction decomposition of eviction error, we clarify the joint requirements of retention and compensation by examining evicted mass and directional gap, establish the decomposition’s validity for normalized aggregation readouts, and prove a converse characterization under the stated assumptions.

We appreciate your comment, which has helped us articulate our contributions and their boundaries more precisely.

W2. Exact decomposition versus guaranteed error minimization

Thank you for your insightful assessment. We agree that our original analysis can be strengthened by explicitly connecting the exact identity to error reduction in CORE’s combined retention and compensation process. Its guarantees concern the pre-compensation error bound, teacher–student allocation fidelity, and memory-write stability, leaving a gap between these properties and error reduction from the combined retention and compensation process. CORE’s effectiveness is supported by empirical evidence, which does not depend on a theoretical guarantee of error minimization. However, a theoretical attribution of these gains requires the additional connection you have identified.

To strengthen this theoretical connection and help alleviate your concern, we have developed a conditional joint-error analysis linking CORE’s learned allocation and actual memory correction to its expected squared reconstruction error. Starting from the exact mass–direction identity, we derive sufficient conditions under which the combined retention and compensation process achieves lower error than a reference strategy with the same cache budget.

[continued below]

Setup. Fix a layer, a candidate cache C={1,…,N}\mathcal C=\{1,\ldots,N\}, an admissible budget 1≤B<N1\le B<N, and a common subsequent-query distribution. All comparisons use common reference QKV states and protected-token conventions. Each strategy uses one retained set shared across heads within the layer, selected before the evaluated queries. Let Sp=TopB⁡(πTC)S_p=\operatorname{TopB}(\pi_T^{\mathcal C}), Sq=Skeep=TopB⁡(πθC)S_q=S_{\mathrm{keep}}=\operatorname{TopB}(\pi_\theta^{\mathcal C}), and let SrefS_{\mathrm{ref}} be an admissible reference set of size BB.

For E=C∖SE=\mathcal C\setminus S, Eq. (1) gives ΔoS(h)=πE(h)(vˉE(h)−oattn(h)(S))\Delta o_S^{(h)}=\pi_E^{(h)}(\bar v_E^{(h)}-o_{\mathrm{attn}}^{(h)}(S)). Concatenating these residuals, define Lret(S)=Eq∥ΔoS∥22\mathcal{L}_{\mathrm{ret}}(S)=\mathbb{E}_{q} \lVert \Delta o_{S} \rVert_{2}^{2} and Lmem(S)=Eq∥ΔoS−cS∥22\mathcal{L}_{\mathrm{mem}}(S)=\mathbb{E}_{q} \lVert \Delta o_{S}-c_{S} \rVert_{2}^{2}, where cSc_{S} is the strategy's memory correction. These are unnormalized squared Euclidean errors in the pre-WOW_O space. In particular, Lret\mathcal L_{\mathrm{ret}} is the expected sum of squared head-wise mass–direction products. Assume positive retained attention mass and finite expectations; query arguments are suppressed below.

Allocation and retention error. Let δi=1[i∈Sq]−1[i∈Sref]\delta_i=\mathbf1[i\in S_q]-\mathbf1[i\in S_{\mathrm{ref}}], J=Sq△SrefJ=S_q\triangle S_{\mathrm{ref}}, and kref=∣Sq∖Sref∣k_{\mathrm{ref}}=|S_q\setminus S_{\mathrm{ref}}|. With full-cache attention weights πi(h)\pi_i^{(h)} and PSq(h)=∑i∈Sqπi(h)P_{S_q}^{(h)}=\sum_{i\in S_q}\pi_i^{(h)}, define

\delta o_{\mathrm{attn}}^{(h)} &= \frac{\sum_i\delta_i\pi_i^{(h)} \bigl(v_i^{(h)}-o_{\mathrm{attn}}^{(h)}(S_{\mathrm{ref}})\bigr)} {P_{S_q}^{(h)}},\\ s_i^{\mathrm{ex}} &= 2\mathbb E_q\sum_h \frac{\pi_i^{(h)}}{P_{S_q}^{(h)}} \left\langle \Delta o_{S_{\mathrm{ref}}}^{(h)}, v_i^{(h)}-o_{\mathrm{attn}}^{(h)}(S_{\mathrm{ref}}) \right\rangle. \end{aligned}$$ The first expression is exactly $o_{\mathrm{attn}}^{(h)}(S_q)-o_{\mathrm{attn}}^{(h)}(S_{\mathrm{ref}})$. Hence, expanding $\Delta o_{S_q}=\Delta o_{S_{\mathrm{ref}}}-\delta o_{\mathrm{attn}}$ yields \$\mathcal L_{\mathrm{ret}}(S_q)-\mathcal L_{\mathrm{ret}}(S_{\mathrm{ref}}) =-\sum_i\delta_i s_i^{\mathrm{ex}}+\mathcal Q_{\mathrm{swap}}\$, where $\mathcal Q_{\mathrm{swap}}=\mathbb E_q\|\delta o_{\mathrm{attn}}\|_2^2$. For the final teacher score $s_{T,i}=u_i+\beta d_i(w^{(R-1)})$, define $\Delta_T=\sum_{i\in S_p}s_{T,i}-\sum_{i\in S_{\mathrm{ref}}}s_{T,i}$ and $\mathcal E_{\mathrm{sel}}=\sum_{i\in S_p}s_{T,i}-\sum_{i\in S_q}s_{T,i}$. Both are nonnegative because $S_p$ maximizes the teacher-score sum over admissible sets. We fix $\lambda_{\mathrm{align}}=1$ and $\nu_{\mathrm{align}}=0$ throughout this analysis and define $\eta_{\mathrm{align}}=\max_{i\in J}|s_i^{\mathrm{ex}}-s_{T,i}|$, with $\eta_{\mathrm{align}}=0$ if $J$ is empty. Since $\sum_i\delta_i s_{T,i}=\Delta_T-\mathcal E_{\mathrm{sel}}$ and $\sum_i|\delta_i|=2k_{\mathrm{ref}}$, we obtain $$\begin{aligned} \mathcal G_{\mathrm{ret}} &= \Delta_T-\mathcal E_{\mathrm{sel}} -2k_{\mathrm{ref}}\eta_{\mathrm{align}} -\mathcal Q_{\mathrm{swap}},\\ \mathcal L_{\mathrm{ret}}(S_q) -\mathcal L_{\mathrm{ret}}(S_{\mathrm{ref}}) &\le-\mathcal G_{\mathrm{ret}}. \end{aligned}$$ Thus, **$\mathcal G_{\mathrm{ret}}>0$ guarantees lower pre-compensation error**, connecting the utility-and-coverage teacher score, student selection, and the exact mass–direction error. **Actual compensation and joint error.** Use CORE's actual correction $c_{S_q}=g_{\chi}(q)m_{S_q}(q)$ from Eqs. (10)–(12). Define $\mathcal{G}_{\mathrm{mem}}(S)=2\mathbb{E}_{q}\langle\Delta o_{S},c_{S}\rangle-\mathbb{E}_{q}\lVert c_{S}\rVert_{2}^{2}$. The exact identity $\mathcal{L}_{\mathrm{mem}}(S)=\mathcal{L}_{\mathrm{ret}}(S)-\mathcal{G}_{\mathrm{mem}}(S)$ then gives $$\boxed{ \begin{aligned} \mathcal L_{\mathrm{mem}}(S_q) -\mathcal L_{\mathrm{mem}}(S_{\mathrm{ref}}) &\le-\mathcal M_{\mathrm{joint}},\\ \mathcal M_{\mathrm{joint}} &= \mathcal G_{\mathrm{ret}} +\mathcal G_{\mathrm{mem}}(S_q) -\mathcal G_{\mathrm{mem}}(S_{\mathrm{ref}}). \end{aligned} }$$ Therefore, **$\mathcal M_{\mathrm{joint}}>0$ guarantees lower post-compensation error**. For an uncompensated reference, $c_{S_{\mathrm{ref}}}=0$. Neither condition requires the mass and direction factors to decrease separately. This is a conditional comparison at fixed reference states, not a global error-minimization guarantee. **Diagnostic protocol.** We use Llama-3.1-8B-Instruct and 104 RULER-16K sequences (eight per subtask), evaluating four layers and up to 32 subsequent queries per sequence–layer state. All methods retain 25% of candidate KV positions, including protected positions, with one shared retained set per layer. They use common reference QKV states and query positions, with one eviction and, where applicable, one write to zero-initialized memory. [continued below]

Hatted quantities replace expectations with averages over the common query sample within each state, with squared reconstruction errors computed in the pre-WOW_{O} space without normalization. For each state and reference method, we fix λalign=1\lambda_{\mathrm{align}}=1 and νalign=0\nu_{\mathrm{align}}=0 and compute η^align=max⁡i∈J∣s^iex−sT,i∣\widehat{\eta}_{\mathrm{align}}=\max_{i\in J}\lvert\widehat{s}_{i}^{\mathrm{ex}}-s_{T,i}\rvert, taking it as zero when JJ is empty. The teacher-score advantage, student selection loss, krefk_{\mathrm{ref}}, and Q^swap\widehat{\mathcal{Q}}_{\mathrm{swap}} are computed directly from the corresponding retained sets and common query samples. Errors and bounds are computed separately for each state and then averaged with equal weights across the same states; G^ret=0\widehat{\mathcal{G}}_{\mathrm{ret}}=0 when JJ is empty. We report the following four quantities.

  • Pre-compensation error L^ret\widehat{\mathcal L}_{\mathrm{ret}}. The average squared reconstruction error after eviction and before memory correction.
  • Post-compensation error L^mem\widehat{\mathcal L}_{\mathrm{mem}}. The average squared reconstruction error after applying the actual memory correction. For H₂O, this equals its pre-compensation error because no correction is applied.
  • Error reduction by CORE ΔL^\widehat{\Delta\mathcal{L}}. The reference method's post-compensation error minus CORE's, namely L^mem(Sref)−L^mem(Sq)\widehat{\mathcal{L}}_{\mathrm{mem}}(S_{\mathrm{ref}})-\widehat{\mathcal{L}}_{\mathrm{mem}}(S_{q}). Positive values favor CORE.
  • Joint-gain lower bound M^joint\widehat{\mathcal M}_{\mathrm{joint}}. The empirical retention-gain lower bound plus the difference in actual compensation gains, computed using the expressions above.

Table R1. Joint-error diagnostic at 75% KV eviction.

MethodPre-compensation error L^ret\widehat{\mathcal L}_{\mathrm{ret}} ↓Post-compensation error L^mem\widehat{\mathcal L}_{\mathrm{mem}} ↓Error reduction by CORE ΔL^\widehat{\Delta\mathcal L}Joint-gain lower bound M^joint\widehat{\mathcal M}_{\mathrm{joint}}
CORE0.30270.2119——
H₂O0.36180.36180.14990.1025
IndexMem0.33260.25230.04040.0214

Table R1 shows that CORE achieves the lowest reconstruction error both before and after memory compensation among the compared methods. Its pre-compensation error is lower than H₂O and IndexMem by 0.0591 and 0.0299, respectively. Memory compensation further reduces CORE’s error by 0.0908, compared with 0.0803 for IndexMem, yielding final error reductions of 0.1499 and 0.0404 over the two references.

The positive empirical joint-gain lower bounds are consistent with the observed error reductions. Together, these results provide empirical support for the conditional analysis, showing that CORE combines lower retention error with effective memory compensation to reduce the joint reconstruction error in the evaluated setting.

Q1. Coverage under matched true evicted attention mass

Your suggestion pinpoints a subtle but important issue in attributing the gains to coverage, and we greatly appreciate the care and technical insight behind this proposed control. It helps us assess the contribution of information complementarity more directly. We have therefore compared the two selectors under the same KV budget and closely matched true evicted attention mass.

We follow the setting of Fig. 5, using common reference QKV states and disabling memory compensation. We compare CORE with a selector trained without the teacher’s coverage objective (β=0\beta=0), keeping the remaining settings unchanged. Each pair corresponds to the same sequence, layer, and attention head. We compute the mean true evicted attention mass over the same subsequent queries and retain pairs whose absolute mass difference is both at most 0.01 and at most 5% of their mean mass. We then compare the normalized directional factor and pre-compensation output error on these matched pairs.

Table R2. Coverage ablation under the same KV budget and matched true evicted attention mass.

KV eviction rateMethodpˉ(E)\bar p(E)D~E\widetilde D_E ↓RetErr ↓
25%Without coverage objective0.36240.96590.3521
25%CORE0.36430.66820.2580
50%Without coverage objective0.60430.97940.5997
50%CORE0.60620.76440.4728
75%Without coverage objective0.73800.83310.6186
75%CORE0.74170.70900.5278
90%Without coverage objective0.90170.78210.7074
90%CORE0.90200.61760.5591

[continued below]

CORE reduces RetErr by 26.73%, 21.16%, 14.68%, and 20.96% at the four eviction rates, respectively, while also achieving a lower normalized directional factor in every setting. Its mean true evicted attention mass is slightly higher in all four comparisons, so the lower error cannot be attributed simply to a reduction in the reported mean evicted mass.

These matched comparisons support the contribution of information complementarity to CORE’s retention quality. The coverage objective improves the composition of retained information, with error reductions that extend beyond preserving more attention mass alone. We have included this diagnostic in Appendix C.1 and discussed its results alongside Fig. 5. Your suggestion has helped us separate the two factors in the error decomposition more clearly.


Q2. Shared allocation under aligned supervision

Inspired by your constructive suggestion, we added a comparison to more fully isolate the benefit of sharing the allocation. We initialized the independent write scorer from the shared scorer and provided equivalent teacher-allocation distillation supervision and comparable training resources. Evaluation followed the Llama-3.1-8B setting in Table 3, using RULER at 4K and 16K context lengths with 75% and 90% KV eviction ratios.

Table R3. Shared versus independent writing with aligned initialization and teacher supervision.

VariantRULER-4K / 75% ↑RULER-4K / 90% ↑RULER-16K / 75% ↑RULER-16K / 90% ↑
CORE: shared allocation80.6062.4474.9868.48
Original independent writer79.3761.8273.0267.43
Independent writer with aligned initialization and teacher supervision79.5562.0473.4167.85

The aligned independent writer improves over the original variant, while CORE retains better task performance. These results indicate that the original performance gap is not entirely attributable to differences in supervision and further support the effectiveness of shared allocation. We thank the reviewer again for helping us strengthen this key comparison.

Q3. Meaning of “coverage-calibrated”

We greatly appreciate your thoughtful and precise attention to this terminology. Your comment reflects a careful reading of both our method and its theoretical scope, and helps us distinguish the intended allocation target from quantities that readers might otherwise assume it estimates.

We agree that the calibration target should be explicit from the term’s first appearance. We have added the following definition in the Introduction and made the same distinction explicit in Section 3.1.

“Coverage-calibrated” refers to allocation weights learned under a teacher objective combining query utility and set-level coverage. These weights jointly guide KV retention and memory writing; they do not represent accurate estimates of future-query attention or eviction error.

We are grateful for the care and technical precision behind your suggestion. It helps align our terminology with the actual learning objective and the scope of our theoretical claims, improving the clarity of the paper and helping readers interpret the learned allocation correctly.


We are grateful for your careful and technically insightful review. Your questions have helped us clarify CORE’s contributions, strengthen the error analysis, and improve the experimental controls for coverage and shared allocation. We hope these additions address your concerns and would welcome further discussion of any points that merit clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Rebuttal Summary for Reviewer and AC Discussion

Dear Area Chair and Reviewers,

We sincerely thank you for your thoughtful reviews and constructive feedback. All three reviews initially gave an overall rating of 3, and the ratings remain at 3. After reviewing our response and additional experiments, reviewer rcj1 acknowledged our response addressing their concerns, and maintained their positive assessment. To provide a concise overview, we summarize the contributions recognized by reviewers, our responses to their concerns, and the subsequent feedback.


Contributions Recognized by Reviewers

  • Significance and problem formulation. CORE coordinates retention and compensation through a shared coverage-aware allocation. bq7a highlighted their “explicit connection,” and rcj1 recognized the “clear rationale” for coordinating them. GCbg noted that CORE “can address the bottleneck of memory use and slow speeds.”

  • Technical design and theoretical grounding. CORE distills set-level coverage into a lightweight indexer and analyzes eviction error and memory-write stability. bq7a highlighted its “lightweight retention policy”; GCbg recognized the “relatively rigorous theoretical derivation” and “solid theoretical grounding.”

  • Empirical validation. bq7a noted that CORE “performs particularly well under aggressive compression”; rcj1 highlighted the “highest RULER average” across all evaluated 75% and 90% compression settings. GCbg described the experiments as “relatively comprehensive.”


Reviewer Concerns → Responses → Outcomes

  • Reviewer rcj1 — 3 (Presentation: 3; Informativeness: 3)

    • Concerns: Calibration and allocation–attention mismatch, Theorem 2’s predictive value, shared allocation under equivalent supervision, and long-generation stability with Qwen’s thinking mode.
    • → Response: Clarified calibration and theoretical scope; added mismatch diagnostics, controlled comparisons, and repeated-eviction results in both thinking modes, with full-KV accuracy as a reference. Results support shared allocation’s advantage and compensation effectiveness at the evaluated checkpoints.
    • → Confirmation: The reviewer confirmed that the mismatch concern was addressed, acknowledged the additional results, and concluded: “Therefore, I maintain my positive assessment.”
  • Reviewer GCbg — 3 (Soundness: 3)

    • Concerns: Training requirements, multi-turn generalization, validation beyond GQA, and title consistency.
    • → Response: Clarified module reuse and training-source evidence; added BFCL and native-MLA evaluations. CORE outperforms the evaluated BFCL compression baselines without retraining or tuning, and outperforms IndexMem in the tested MLA settings. We also aligned the OpenReview and PDF titles.
    • → Status: The reviewer has not yet posted a follow-up response; the overall rating remains 3.
  • Reviewer bq7a — 3

    • Concerns: Contribution beyond existing frameworks, the theory–method connection, information complementarity, shared allocation, and calibration.
    • → Clarification: Identified shared coverage-aware allocation as the methodological contribution and the error identity as an analytical starting point. Calibration targets the utility-and-coverage teacher allocation.
    • → Additional evidence: Added conditional joint-error analysis and comparisons showing lower pre-compensation error with coverage at the same KV budget and closely matched true evicted attention mass. Under aligned initialization and teacher supervision, with comparable training resources, CORE retains its advantage in all four evaluated RULER settings.
    • → Status: The reviewer has not yet posted a follow-up response; the overall rating remains 3.

Summary for the Area Chair

The reviews recognize CORE’s shared retention–compensation design, theoretical grounding, and experimental results. rcj1 maintained their positive assessment after reviewing the additional evidence. We welcome further discussion with bq7a and GCbg about the explanations and results provided in our responses.

We have incorporated the clarifications, additional analyses, and experimental results into the revised manuscript. The responses specify the theoretical scope and report the full-KV accuracy reference, and their opening tables identify the corresponding revision locations. We hope this summary helps the AC consider the revised paper together with the complete rebuttal and supporting evidence.

We sincerely thank you again for the time and care you have devoted to reviewing and discussing our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.