acceptodds
Under review as a conference paper at ICLR 2027

PAGE: Partition-Aware Gated KV-Cache Eviction

Abstract

KV-cache eviction can do more than compress. In long-context LLMs, keeping only some cached tokens sometimes matches or exceeds full-cache accuracy, because many redundant prefill tokens otherwise dilute attention away from the tokens that carry the answer. This benefit is not uniform, and evicting the wrong tokens drops accuracy to zero on tasks that need precise retrieval, so the useful question is not only which tokens to keep but whether to evict this input at all. We show that one label-free number computed from the prefill attention, the drop between early and late layers in how much attention heads agree on which tokens to read, predicts per input, before any decoding, which of the two cases an input falls under. We build this into PAGE (Partition-Aware Gated Eviction), a wrapper that runs any SnapKV-style evictor when the drop is large and keeps the full cache when it is small, with no training, labels, or fine-tuning. PAGE is a safety mechanism rather than a compressor, so we measure it by the failures it prevents. It cuts the harm rate on capacity-bound inputs from 0.75 to 0.026, and on multi-key retrieval with Mistral-7B plain SnapKV falls from 99% to 0% as the budget shrinks while PAGE holds it at 89%. Elsewhere it passes the base evictor through unchanged, which is the intended behaviour and is what we observe in 8 of 16 cells. The prediction is one-sided, identifying evictable inputs reliably and resolving the single-near-tie-distractor boundary only with per-input information beyond the mean drop. The head-agreement drop orders tasks the same way across four model families, but the threshold does not transfer, and an unlabeled roughly 100-input pilot per model is needed outside Qwen2.5 and Mistral. Realized compression is 1.8 to 3.4 against a nominal 16 budget and decays toward unity by batch 16, and at matched memory a trained evictor outperforms PAGE. Code is available at https://anonymous.4open.science/r/PAGE-018239.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Key-Value Cache Eviction, Memory-Efficient Inference, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GqTUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper observes that the effect of KV-cache eviction in long-context LLMs is input-dependent: for some inputs, evicting redundant prefill tokens mitigates attention dilution and matches or even exceeds full-cache accuracy, whereas for inputs that rely on precise retrieval of a few critical tokens, eviction causes accuracy to collapse. Accordingly, the authors partition inputs into dilution-prone and capacity-bound classes, and show that a single label-free scalar computed from prefill attention, namely the early-to-late-layer drop in pairwise top-k (Jaccard) head agreement, D, predicts per input and before any decoding which class an input belongs to. Building on this, they propose PAGE (Partition-Aware Gated Eviction), a training-free wrapper that invokes any SnapKV-style evictor when D exceeds a threshold and retains the full KV cache otherwise. PAGE is positioned as a safeguard against harmful eviction rather than a compressor. Experiments show that on the capacity-bound task in RULER, PAGE reduces the harm rate under 16× compression from 0.75 to 0.026.

Critical strengths
  1. The problem addressed is valuable and complements the shortcomings of current methods.
  2. The gating signal is simple and the method is modular, applicable to any SnapKV-style evictor.
Critical weaknesses
  1. Limited novelty. Both the dilution mechanism and the head-agreement signal come from prior work; the contribution mainly amounts to using them as a threshold gate around existing evictors.
  2. The threshold τ is set empirically and transfers poorly across models, requiring recalibration for each new model.
  3. Limited datasets. The main experiments are conducted on RULER, and validation beyond RULER is small in scale, so generalization to other tasks remains insufficiently validated.
  4. Limited model coverage. The main experiments focus on Qwen2.5 and Mistral-7B, with only limited validation on other model families, lacking validation on larger and more recent models, as well as on newer attention mechanisms and architectures (e.g., hybrid models, Multi-head Latent Attention (MLA), and DeepSeek Sparse Attention (DSA)).
Questions for authors
  1. How does PAGE perform on tasks beyond RULER?
  2. How does PAGE perform on larger and more recent models?
  3. For hybrid models or models with newer attention mechanisms such as MLA and DSA, does PAGE remain effective?
2 comments
AuthorsOpenReview ↗

Response to Reviewer GqTU

Thank you for the review. We clarify the model and workload coverage below.

Weakness 1: Novelty

PAGE adds an input-level routing decision to existing eviction methods. Its prefill head-agreement signal identifies a failure-prone task class without labels at inference and outperforms the evaluated gates based on SnapKV, H2O, and OBCache scores (Appendix I.5, Table 41). It achieves 93.8% of a hindsight task-level policy's gain, allowing fractional routing within the baseline's final task to match the closed fraction (Appendix I.6, Table 42). The dilution mechanism and head-agreement statistic build on prior work, as acknowledged in the paper. Our response to Reviewer rro6 discusses these comparisons in more detail.

Weakness 2: Threshold transfer

The standardized gate accounts for differences in the scale of DD across models. It separates NIAH-MK3 from the other tasks in seven Qwen2.5, Mistral, Llama, and Yi cells, including the two held-out model families (Appendix A.3.4). Its selection gain is at least as large as the fixed threshold's on every model in Table 36, increasing from +0.3pp to +12.4pp on Qwen3-4B. Further results cover Qwen3-14B (Appendix I.7, Table 43), two MLA models, and three hybrid models (Appendix I.8, Table 44).

This supports transfer on the reported models under the evaluated calibration conditions. Applying the standardized threshold requires an unlabeled pilot of approximately 100 inputs per model, although fitting the standardized cutoff used task labels. Pilot composition affects calibration (Appendix I.4, Table 40), so we do not claim a universal threshold. Section 6 also documents failures of the fixed threshold τ\tau of 0.07.

Weakness 3 and Question 1: Beyond RULER

The three LongBench subtasks in Table 12 test classification before decoding. They do not establish selection over a random gate. We added a pre-registered experiment on NQ multi-document QA and BABILong to test selection on real documents. These tasks are harmed by eviction but have among the highest D scores in their workloads, and the gate achieves no positive selection (Appendix I.9, Table 45, “Selection” and “Holm p” columns). These near-tie tests do not establish positive selection. The other real-document runs (Appendix I.11, Tables 47- 49) show isolated gains on L-Eval, hotpotqa and QuALITY that do not hold on a second model.

Weakness 4 and Question 2: larger and more recent models

The submitted evaluation already included Qwen2.5-32B, Qwen3-4B, Llama-3.1-8B, and Yi-1.5-9B (Appendix A.4.7). We now add explicit selection controls. The standardized gate gains +12.0pp, +12.4pp, and +4.0pp over random fallback on the first three models, respectively (Table 36).

We also added a pre-registered Qwen3-14B experiment (Appendix I.7, Table 43). The fixed threshold opens on 3% of inputs, so its gain is largely due to full-cache fallback. The standardized gate retains the full cache for 84% of NIAH-MK3 inputs, uses approximately half the cache overall, and gains +7.5pp [+5.9, +9.0] over the random gate.

Question 3: hybrid models, MLA, and DSA (Appendix I.8)

  • MLA: On DeepSeek-V2-Lite-Chat and GLM-4.7-Flash, DD separates NIAH-MK3 from the other tasks with AUCs of 1.00 and 0.99. Selection gains averaged over budgets are +8.1pp [+5.4, +10.7] and +12.0pp [+10.7, +13.2], respectively (Appendix I.8, Table 44).
  • Hybrid models: We compute DD and apply eviction in the layers that maintain a key-value cache. DD separates NIAH-MK3 on Olmo-Hybrid-7B and Qwen3-Next-80B-A3B, which combine linear and full attention, and on LFM2-24B-A2B, which uses short convolutions. Selection gains averaged over budgets range from +6.8pp to +10.2pp, with all intervals above zero. Selection is also positive at b=1/16b = 1/16 on each model (Appendix I.8, Table 44). The placement of cached attention layers also limits architectural transfer (Section 6).
  • DSA: We have not evaluated DSA because no available open checkpoint with DSA-type attention fits our hardware (Appendix I.8, “Implementation details”).

We hope these results answer your questions and would be grateful if you could consider raising your score.

AuthorsOpenReview ↗

Discussion Reminder

Thank you again for your time. We hope our responses have addressed your concerns. We would appreciate your reassessment and are happy to clarify anything further.

Reviewer_rro6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PAGE applies a threshold to the difference in head agreement between early and late layers to decide whether to evict KV-cache entries or retain the full cache for each input.

Critical strengths

The motivation is understandable, and the experiments show that avoiding eviction on some inputs can reduce severe accuracy degradation.

Critical weaknesses
  1. Heuristic measurement. Using head agreement to measure eviction safety is heuristic and lacks convincing theoretical justification. OBCache [1] derives its scores from the perturbation of attention outputs caused by pruning. No comparably rigorous connection between agreement and eviction effects is established here.

  2. Limited novelty. Head agreement is an existing statistic, and applying it to KV-cache eviction does not provide a clear new insight.

  3. Limited decision scope. The binary gate only chooses between eviction at a preset budget and retaining the full cache. It does not determine how much eviction is safe or how to allocate the budget across layers, and harmful gate-open cases remain.

  4. Data-dependent calibration and missing scoring baselines. SnapKV-style like H2O and OBCache scores can all be aggregated into input-level signals, standardized, and threshold-calibrated to construct alternative gates. This pipeline is not specific to agreement, and without these comparisons its claimed advantage remains unestablished. Standardization does not justify a universal safety threshold. How −0.69 was selected, and whether thresholds and accuracy–memory results remain stable when calibration samples, task proportions, or sample sizes change, require stronger evidence.

[1] OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inference

Questions for authors
  1. What advantage does agreement-based gating provide over calibrated gates using aggregated SnapKV-style, H2O, or OBCache scores, activation changes, and random gating at a matched gate-open rate?

  2. How was −0.69 selected? When calibration samples, task proportions, or sample size change, how stable are the effective threshold, harmful-eviction rate, accuracy, and retained-cache size?

4 comments
AuthorsOpenReview ↗

Response to Reviewer rro6

Thank you for suggesting the matched-open-rate comparison.

Weakness 4 and Question 1: Other gates at a matched open rate

In the pre-registered experiment, every gate closes the same number of inputs as DD. We fixed signal orientations on Qwen2.5-1.5B before evaluating the other models (Section 4.1, Appendix I.5, Table 41). The alternatives use retained SnapKV, H2O, or OBCache score mass, activation change, next-token KL divergence, or top-1 agreement. We also include random fallback. Activation change is the relative change in the final hidden state at the first decoding step under pruned versus full-cache inference.

  • DD achieves larger gains than these alternatives on the three primary models and the optional fourth model, Qwen2.5-14B.
  • Score-based gates perform worse than random fallback on two of the three primary models. On Mistral-7B, their gains are +1.0 to +2.8pp, compared with +8.0pp for random fallback and +18.7pp for DD.
  • On Qwen2.5-3B and Mistral-7B, activation change is the closest alternative, trailing DD by 1.6 and 2.1pp, respectively. It requires an additional forward pass per budget.

Table 41 also reports exploratory comparisons with three prefill statistics we examined before pre-registration. Under the registered comparison rule, DD has an advantage over entropy drop and top-32 attention share, but no consistent advantage over its early-layer agreement term. They tie on two models. DD leads by 3.9pp on Qwen2.5-3B, while early-layer agreement leads by 1.1pp on Mistral-7B.

These comparisons support head agreement as a signal for distinguishing NIAH-MK3 (Table 41 and Figure 5). They do not establish general per-input harm prediction, which we test separately in Appendix I.1, Table 33.

Weakness 1: Heuristic signal without a derivation comparable to OBCache

OBCache (Gu et al., ICML 2026) derives token-retention scores from changes in attention outputs. PAGE decides whether to apply eviction to an input. We connect these two decisions by testing gates built from aggregated token scores at the same open rate as DD. DD achieves larger gains on all four models (Appendix I.5, Table 41). The matched-open-rate experiment provides direct empirical evidence for this routing decision. The mechanistic interpretation in Section 5 and Appendices B-D is separate from that comparison, whose conclusions depend on the measured gating outcomes.

Weakness 2: Novelty

PAGE adds an input-level routing decision that complements token-retention methods. The prefill head-agreement signal requires no task labels at inference, outperforms the evaluated score-based gates, and achieves 93.8% [92.0, 95.4] of a hindsight task-level policy's gain. That baseline allows fractional routing within the final task to match PAGE's closed fraction (Appendix I.6, Table 42). Section 1 acknowledges head agreement as an existing statistic. We also correct the separate claim that PAGE outperforms every task-level policy.

Weakness 3: Decision scope

The binary gate decides whether to evict, leaving token retention and layer allocation to the base method. The PyramidKV rows of Table 2 demonstrate this composition. Table 29 reports residual harm, and Appendix A.2.5 describes the NIAH-MK2 false negative. On Mistral-7B, cwe accuracy falls from 0.84 with the full cache to 0.00 at b=1/16b = 1/16 while the gate opens on every input (Appendix I.10, Table 46). Section 6 explains that false negatives can reduce accuracy, while false positives reduce compression.

Question 2: Selection and stability of −0.69 (Appendix I.4)

  • Selection: On five Qwen and Mistral cells, we consider midpoints between sorted standardized scores. We maximize the number of cells in which NIAH-MK3's mean falls below the threshold and other task means fall above it, excluding niah_multivalue. Ties are resolved by balanced accuracy, then by the lowest threshold. Fitting requires task labels, but applying the fitted threshold does not (Table 8 and Appendix I.4).
  • Fitted value: Thresholds from −1.306 to −0.587 separate all five cells. Leave-one-cell-out refits give −0.693 to −0.881 (Table 39).
  • Pilot sampling: The threshold's standard deviation across resampled pilots is at most 0.005 for 100 inputs and 0.010 for 20 inputs (Table 40).
  • Task composition: Raising the pilot's NIAH-MK3 share from 0% to 50% changes the threshold by a factor of 1.3 on Mistral-7B and approximately 11 on Qwen2.5-3B. On Qwen2.5-3B at b=1/16b = 1/16, accuracy falls from 0.60 to 0.39, harm rises from 0.33 to 0.57, and retained cache falls from 0.53 to 0.26 (Table 40).

The threshold in DD units depends on the pilot mean and standard deviation, which change with task composition. Standardization therefore does not guarantee a universal threshold or transfer across workloads. Section 3, “Calibration,” and Appendix I.4, Table 40, discuss this dependence.

We hope this addresses your concerns. We would be grateful if you could consider raising your score.

AuthorsOpenReview ↗

Discussion Reminder

We would like to kindly follow up on our rebuttal responses. We hope we have addressed your concerns and would appreciate your thoughts before the November 18 (AoE) deadline.

Reviewer_rro6OpenReview ↗

Response

Thank you for the response and the additional baseline and calibration experiments. My main concerns remain unresolved.

  1. Limited novelty and lack of theoretical insight. PAGE repurposes an existing head-agreement statistic as a binary eviction gate. The added value of using the difference between early and late layers remains unclear. In Table 41, the difference beats early-layer agreement alone on one model, ties on two, and loses on Mistral. Using head agreement as a measure of eviction safety remains heuristic. As Section 5 acknowledges, the theoretical analysis does not validate the gate under the measured conditions.

  2. Limited decision scope. PAGE either retains the full cache or applies eviction at a preset budget. The gate leaves the choice of a safe cache budget and its allocation across layers unresolved. In Table 46, PAGE allows eviction on every common-words extraction sample for Mistral-7B. Accuracy falls from 84% with the full cache to 0% when SnapKV retains 1/16 of the cache.

  3. Insufficient generalization. PAGE's query-aware gate uses the question when computing head agreement. A query-agnostic comparison using only the context would help determine how much the gate relies on the question to identify the task type. The gain over random gating at the same gate-open rate drops from 8.3 to 0.5 percentage points when NIAH-MK3 is excluded (Table 34). On NQ and BABILong, PAGE trails the random gate in all six primary tests (Table 45). Changing the calibration task mix also lowers Qwen2.5-3B accuracy from 60.1% to 38.7% on the same evaluation set (Table 40). Together, these results leave the gate's reliability across workloads unclear.

I maintain my original recommendation.

AuthorsOpenReview ↗

Additional Response to Reviewer rro6

Thank you for the follow-up. We address each point below and clarify what our results support.

Early-to-late difference: We agree that the early layers provide most of the separation for NIAH-MK3. In Table 41, DD and aearlya_{\text{early}} perform equally well on Qwen2.5-1.5B and Qwen2.5-14B. On Qwen2.5-3B, DD does 3.9pp better, whereas aearlya_{\text{early}} does 1.1pp better on Mistral-7B (also consistent with the layer-wise results in Appendix A.3.3), where NIAH-MK3 has the lowest early-third agreement across all four models. The importance of early-layer agreement is further confirmed in Llama-3.1-8B, where the late layers reduce the difference in DD between NIAH-MK3 and VT to just +0.007. Collectively, these results do support head agreement as a useful gating signal.

Theory and score-based gates: As stated in Section 5, our theoretical analysis helps explain the mechanism, but it does not directly establish the effectiveness of the gate. We therefore rely on the controlled experiments in Table 41, where all gates are compared at the same open rate. The gains are measured over plain eviction and averaged across the budget grid. Under this setup, gates based on the retained score mass of SnapKV, H2O, and OBCache at b=1/8b=1/8 perform worse than random gating on three of the four models. In contrast, DD performs better than random gating on all four, which supports our choice of signal on score-based gates, we tested.

Decision scope: PAGE is designed to make a binary decision about whether eviction should be applied. The eviction budget and its distribution across layers are still determined by the base method, as illustrated by the PyramidKV results in Table 2. This also means that PAGE does not guarantee protection against every harmful eviction. For example, it fails to detect harmful eviction for cwe on Mistral-7B. We report this case in Table 46 and Section 6 to make this limitation clear.

Query dependence: We compute DD using the last 32 prompt positions (Section 3), which is also the window used by SnapKV-style methods for token scoring. Since the question appears at the end of the user message in our setup, both PAGE and the base method have access to the same information. Thus, using the question is part of the method's design rather than an additional source of information available only to PAGE. However, we agree that this does not tell us whether DD captures the task mainly from the question or from the context.

Workload transfer: Tables 34, 45 and 48 report these results in full. Together they show that the gate helps where a workload contains a task family with the NIAH-MK3 pattern, including outside RULER. L-Eval on Qwen3-14B is one such case. Its topic retrieval task has the lowest mean D of six tasks, and its accuracy falls from 0.887 to 0.427 at b = 1/16. The gate keeps 84.7% of these questions at full cache, and selection over the random gate is +3.8pp [+2.8, +4.9] (Table 48). Where a workload has no such task family, transfer is not shown, as Section 6 states.

Calibration (Table 40): The pilot mixture affects the trade-off between accuracy and memory savings. On Qwen2.5-3B, PAGE protects every NIAH-MK3 input when its share in the pilot ranges from 0% to 25%. However, as this share increases, the gate becomes more aggressive on the other inputs, with the open rate rising from 0.665 to 0.996. As a result, retained cache decreases from 0.53 to 0.30, while accuracy drops from 0.601 to 0.424. The 25% share also matches the evaluation mixture.

When the pilot share increases further to 50%, PAGE begins evicting 14.7% of NIAH-MK3 inputs, and accuracy falls to 0.387. This shows that the choice of pilot mixture matters not only for memory savings but also for protecting sensitive inputs. As discussed in Section 3, the threshold can therefore be transferred without labels only when the pilot is representative of the deployment workload.

Overall, PAGE provides a label-free way to identify an eviction-sensitive task family and decide when to preserve the full cache. We evaluate this approach against random and score-based gates at matched open rates. While the results support its usefulness in certain settings, we do not claim that PAGE can predict harmful eviction for every task or that a single threshold will work across different workloads.

Reviewer_sW1MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a simple heuristic for predicting whether KV-cache eviction will harm downstream task accuracy. The proposed method, PAGE, measures the decrease in attention-head agreement from early to late layers, where agreement is defined by the overlap between heads’ top-attended token positions. If this decrease is below a threshold, PAGE retains the full KV cache; otherwise, it applies an existing eviction method.

Although the paper addresses an interesting problem, it requires substantial improvement in methodological justification, presentation, and experimental validation. In particular, the connection between the proposed statistic and eviction-induced accuracy loss is insufficiently established, and the reported task-class separation does not directly demonstrate reliable prediction of eviction harm at the individual-input level.

Critical strengths

The paper studies an underexplored topic in token eviction. When to apply token eviction has received limited attention, although predicting its impact on accuracy is essential for achieving the memory and latency benefits of token eviction methods without compromising downstream task accuracy.

Critical weaknesses
  1. Presentation and organization: The paper requires substantial improvements in clarity and organization. For instance, the definition of the central PAGE score could be improved: a self-contained mathematical formulation of the attention aggregation, head-pair agreement, and layer grouping would make the method easier to understand. More broadly, the main text does not provide a clear progression from the proposed statistic to its motivation and supporting evidence. Definitions, qualifications, and experimental details are scattered across the main text and appendices, making it difficult to identify which results support each claim. The paper should more clearly distinguish task-class separation, prediction of per-input eviction harm, and the accuracy–memory trade-off of the resulting policy.

  2. Experimental interpretation: Figure 3 and Table 2 show accuracy improvements from adding PAGE to existing eviction methods, but these comparisons use the same nominal eviction budget rather than the same achieved cache usage. Since PAGE retains the full cache whenever the gate closes, it can use substantially more memory than the baseline. These results therefore demonstrate accuracy preservation through selective full-cache fallback, but do not by themselves establish an improvement under a controlled memory budget. Although Figure 4 provides a matched-cache comparison, this distinction should be central to the presentation and interpretation of the headline results. Furthermore, directly evaluating the score against per-input accuracy loss from eviction would better establish its predictive value; task-class separation alone is insufficient.

Questions for authors
  1. Can the authors provide direct evidence that the proposed PAGE score DD predicts the task success rate under eviction? The manuscript currently claims that applying PAGE generally improves downstream task accuracy. However, these results include the effect of skipping eviction. I was unable to find direct evidence that DD predicts the likelihood of task success under eviction.

  2. Is it possible to demonstrate PAGE’s predictive effectiveness on tasks beyond the RULER benchmark? RULER is primarily synthetic and therefore may not adequately represent PAGE’s overall applicability to more general tasks. I would appreciate results for PAGE on more general long-context evaluation datasets.

4 comments
AuthorsOpenReview ↗

Response to Reviewer sW1M

Thank you for the review. We address harm prediction and cache usage below.

Question 1 and Weakness 2: Does DD predict success under eviction?

We test this directly in Appendix I.1. Among inputs answered correctly with the full cache, we ask whether low DD identifies those whose answers become incorrect after eviction. Table 33 compares DD with a task-identity baseline and reports both pooled and within-task AUCs.

  • Across tasks, the pooled AUC is 0.69-0.96 on the four 4K models. The baseline uses task labels and harm rates estimated on held-out inputs. It has a higher AUC in 12 of 16 comparisons, with the largest difference on Mistral-7B at b=1/16b = 1/16: 0.69 for DD versus 0.95 for task identity (Table 33).

  • Within tasks, discrimination is less consistent. The AUC pooled over budgets ranges from 0.415 to 0.652 by cell, and the budget-level interval includes 0.5 in 37 of 44 cell-budget pairs (Table 33).

Task-class discrimination is useful for routing when harm is concentrated by task, as it is in this RULER evaluation. DD identifies that structure without task labels. The within-task results do not establish consistent discrimination between harmed and safe inputs, although the small number of harmed inputs limits statistical power. We make this distinction explicit in the abstract and Section 4.1.

Weakness 2: Nominal budget versus achieved cache

PAGE retains an accuracy advantage over random fallback at the same open rate. The matched-cache comparison then shows where its advantage over plain eviction persists when memory use is controlled.

  • Selection over random fallback is +8.3pp [+7.5, +9.1], significant in 14 of 16 cells after Benjamini-Hochberg correction (Appendix I.2, Tables 34 and 35). Random fallback accounts for 63.5% [61.3, 66.0] of the +22.9pp headline gain, so retaining the full cache alone does not explain PAGE's performance.
  • At matched cache usage, PAGE outperforms plain SnapKV at 4K once approximately one-third of the cache is retained, with a crossover of 0.55 on Qwen2.5-3B and a maximum advantage of +19.1pp. At b=1/16b = 1/16, it underperforms on all four models (Appendix I.3, Figure 4, and Tables 37–38).

The harm reduction from 0.75 to 0.026 in Table 29 is a result at the same nominal budget, as Section 4.1 now states. Because PAGE retains the full cache for some inputs, its accuracy gains must be considered alongside its memory use.

Question 2: Beyond RULER

The submission includes three LongBench subtasks on Qwen2.5-14B (Table 12). Before decoding, DD assigns lcc to the capacity-bound class, repobench-p to the mixed class, and hotpotqa to the evict-robust class. These results evaluate task classification rather than selection over a random gate.

We added a pre-registered test on NQ multi-document QA and BABILong to evaluate selection on real documents (Appendix I.9, Table 45). Eviction harms these retrieval tasks, but their DD scores are among the highest in their workloads, so the gate generally stays open. Selection is negative in all six primary tests, and neither dataset meets the pre-registered success criterion.

These near-tie tests do not establish positive selection. Appendix I.11 lists the other real-document runs (Tables 47- 49). Three are positive at b=1/16b = 1/16: Qwen3-14B on L-Eval with the standardized gate, +3.8pp [+2.8, +4.9] (Table 48), Qwen2.5-3B on hotpotqa with the fixed threshold, +2.9pp [+0.9, +5.2], and the Mistral-7B QuALITY quote arm, +2.1pp [+0.0, +4.1] (Table 47). None holds on a second model. hotpotqa selection is −1.0pp [−1.9, −0.3] on Qwen2.5-14B (Table 49), the QuALITY result does not replicate on Qwen2.5-14B (Table 47), and the other four L-Eval models lose or show no difference (Table 48). On RULER, excluding NIAH-MK3 reduces selection to +0.5pp [+0.1, +0.9] (Tables 34 and 35).

Weakness 1: Presentation

Section 3, “Head-Agreement Drop DD,” gives the definitions of window attention, top-k sets, pairwise Jaccard similarity, and layer bins before defining DD in Equation (3). Section 4 separates the evidence for task-class discrimination (Tables 3 and 6), per-input harm prediction (Appendix I.1, Table 33), and the accuracy–memory trade-off (Figure 4 and Appendix I.3).

We hope this addresses your concerns and would be grateful if you could consider raising your score.

AuthorsOpenReview ↗

Discussion Reminder

Thank you for your feedback. We have addressed your comments in our rebuttal and would appreciate it if you could revisit our responses. Please let us know if you have any further questions or concerns.

Thank you for the additional experiments and for explicitly acknowledging the limitations. The new analyses address my questions more directly, but they do not resolve my main concern about the predictive validity of the PAGE score.

For instance, the additional results show that PAGE’s selection gain over random fallback is limited to +0.5pp on RULER when NIAH-MK3 is excluded, suggesting that the selection benefit is largely concentrated in a single task within the benchmark. The additional LongBench results raise a similar concern: the benefit of PAGE is inconsistent across models and does not establish reliable generalization across tasks and models.

In my view, the manuscript requires further refinement of both the method and its evaluation to establish whether PAGE consistently improves the accuracy–memory trade-off over existing token eviction methods.

AuthorsOpenReview ↗

Additional response to reviewer sW1M

Thank you for the follow-up. We respond on predictive validity and on the accuracy-memory trade-off with the results that bear on them.

Predictive validity of D: Table 33 tests D directly against per-input harm. An input counts as harmed if it is answered correctly with the full cache and incorrectly after eviction. When inputs from all tasks are considered together, D ranks harmed inputs below unharmed ones with an AUC of 0.69 to 0.96 on the four 4K models, and it does so without task labels. D therefore predicts harm at the level of the task type rather than the individual input. On RULER, where harm is concentrated in particular tasks, this is enough for the gate to route inputs, because D identifies the failure-prone task class before any decoding.

Concentration on NIAH-MK3: Table 34 splits the gain into two parts. The first is what any gate obtains by keeping the full cache equally often, measured with a random gate. The second is the extra gain from D choosing which inputs to protect. This extra gain concentrates on NIAH-MK3, the task on which eviction fails most severely. On Mistral-7B, plain SnapKV accuracy on this task falls from 0.99 to 0.00 at b = 1/16 (Figure 3, left). The same task is detected on every model in Tables 43 and 44. With the standardized threshold, the gain over a random gate is +7.5pp [+5.9, +9.0] on Qwen3-14B (Table 43), +8.1pp and +12.0pp on two MLA models, and +6.8pp to +10.2pp on three hybrid models (Table 44).

LongBench and other real documents: Tables 47 to 49 report all of these runs, with each LongBench model shown separately in Tables 47 and 49. The results are consistent with the gate helping when a workload contains a task that behaves like NIAH-MK3. L-Eval on Qwen3-14B is such a case, with a gain of +3.8pp [+2.8, +4.9] over a random gate (Table 48). When a workload has no such task family, the transfer is not shown, as Section 6 states.

Accuracy-memory trade-off: We agree that gains should be judged at equal memory use. PAGE wraps an existing evictor rather than replacing it, so the fair and controlled comparison is the same evictor at retained cache (Appendix I.3). Averaged over the eight budgets, SnapKV with the gate is more accurate than SnapKV alone at equal retained cache on all four 4K models, by +3.1 to +10.3pp with every interval above zero, and by up to +19.1pp at a single budget (Table 38). The gain appears once roughly a third of the cache is kept (0.55 on Qwen2.5-3B). At the smallest budget, b = 1/16, SnapKV alone is ahead, because the inputs kept at full cache then account for most of the memory, as Section 4.1 states.

In summary, PAGE's contribution is a label-free safeguard for an eviction-sensitive task family, tested against random and score-based gates at a matched open rate and against plain eviction at matched achieved cache. We do not claim a general per-input harm predictor.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

Thank you for the reviews. Appendix I reports the added experiments and analyses. All changes are shown in blue. Tables 1-32 retain their numbering and values, apart from Table 4's corrected gate-open column.

Additional evidence:

  • Selection beyond fallback: PAGE gains +8.3pp [+7.5, +9.1] over a random gate at the same open rate. This selection gain is significant in 14 of the 16 cells in Table 2 (Appendix I.2, Tables 34 and 35).
  • Alternative gating signals: In a pre-registered comparison at a matched open rate, DD achieves larger gains than gates built from aggregated SnapKV, H2O, and OBCache scores or activation change on all four tested models. Score-based gates often perform worse than random fallback (Section 4.1, “Comparison with Score-Based Gates,” and Appendix I.5, Table 41).
  • Transfer across models and architectures: With the standardized threshold, selection gains are +7.5pp [+5.9, +9.0] on Qwen3-14B (Appendix I.7, Table 43), +8.1pp and +12.0pp on two MLA models, and +6.8pp to +10.2pp on three hybrid models (Appendix I.8, Table 44).
  • Task-level routing: Without task labels at inference, PAGE achieves 93.8% [92.0, 95.4] of a hindsight task-level policy's gain at the same closed fraction. The baseline allows fractional routing within the final task to match that fraction (Appendix I.6, Table 42).

What these results establish:

  • Harm prediction: DD primarily distinguishes task classes in this evaluation. Evidence that it predicts harm within a task is limited (Appendix I.1, Table 33). The abstract and Section 4.1 distinguish these two capabilities.
  • Cache usage: The headline gains and harm reduction compare methods at the same nominal eviction budget. We also compare PAGE with random fallback at the same open rate and with plain eviction at matched cache usage (Appendix I.2, Tables 34 - 35, and Appendix I.3, Tables 37 - 38).
  • Workload transfer: The pre-registered near-tie retrieval test finds no positive selection over a random gate. These tasks have among the highest D scores in their workloads despite being harmed by eviction (Appendix I.9, Table 45). The RULER selection gain is concentrated on NIAH-MK3. Excluding that task reduces it to +0.5pp [+0.1, +0.9] (Tables 34 and 35). The other real-document runs (Appendix I.11, Tables 47- 49) include isolated positive results but do not establish consistent transfer across models.

The score-based gate comparison, Qwen3-14B experiment, and near-tie retrieval test were pre-registered. The original near-tie registration was hashed before validation, with an execution-only amendment before the main runs. Table 45 uses a post-hoc add-one correction to finite-bootstrap p-values before Holm adjustment. Estimates, intervals, and significance decisions are unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.