acceptodds
Withdrawn from ICLR 2027

PagePrune: Bridging the Pruning–Execution Gap in Paged-Attention KV Cache

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Long-context large language model (LLM) inference is increasingly constrained by the growing Key-Value (KV) cache, which consumes substantial GPU memory and incurs repeated memory traffic during decoding. Existing KV-cache pruning methods reduce the retained cache size, but often overlook how the surviving KV entries are organized and accessed by attention kernels, limiting the resulting end-to-end (E2E) gains. We present **PagePrune**, an execution-aware, page-granular KV-cache pruning system for vLLM that jointly optimizes KV selection and execution layout. PagePrune periodically ranks KV pages using exponentially smoothed attention importance and retains high-utility pages under a configurable budget, while protecting critical prefix and recent context. It further aligns pruning decisions with attention-kernel tiles and performs compact-once repacking to expose the retained cache as a contiguous working set. We integrate PagePrune into vLLM and evaluate it across long-context language, vision question answering, and video-language workloads. Empirically, PagePrune substantially reduces KV-cache memory footprint while achieving up to **7.91×** E2E speedup over vanilla vLLM with comparable task quality. Experiments and analysis demonstrate that effective KV-cache pruning requires not only selecting what to retain, but also organizing how the retained cache is executed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, VLLM, Paged KV Cache, Pruning, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ue24OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed an execution-aware, page-granular KV-cache pruning system built natively on vLLM's PagedAttention, named PagePrune. On paged serving systems, evicting KV pages reduces memory capacity but does not reduce memory traffic, because the attention kernel iterates positionally over the block table and therefore still traverses evicted regions. Every G decoding steps, PagePrune ranks pages by exponentially-smoothed attention mass, retains the top-τ fraction plus sink/recent page protection, expands the keep-set to the kernel's 8-page tile boundaries, frees the remaining physical blocks, and rebuilds the block table densely over the survivors (keeping a logical-index map so compacted entries can be mapped back to original positions for RoPE). Evicted pages are recomputed from stored token inputs before the next re-ranking round. Evaluated on different models and benchmarks, the paper reports up to 3.86× (text), 5.22× (VQA), and 7.91× (video summarization) end-to-end speedups over vanilla vLLM at roughly comparable quality.

Critical strengths
  1. the paper is well written and easy to understand.
  2. The core observation is real and well-evidenced, as in Figure 1.
  3. The ablation well supports the motivation. Page scoring alone cuts memory (73.67 → 53.39 GB on DREAM1K) yet reduces throughput (77.36 → 70.90 tok/s) — a clean demonstration that capacity savings without layout repair can be net-negative, since the kernel still walks the scattered span plus selection overhead. The gains appear only once tile alignment and compaction are added.
  4. Deployability. Native vLLM integration with no custom sparse kernel and no changes to the hot attention loop.
  5. Extensive experiments and ablation study are conducted, including different benchmarks and models plus sensitivity studies over pruning interval G, tile size K, and batch size, and an explicit recomputation-overhead analysis.
Critical weaknesses
  1. The novelty seems incremental. The contribution is the combination and the gap framing, not a new selection principle, for example attention-mass page ranking (cf. Quest), EMA smoothing, sink + recent protection (cf. StreamingLLM/H2O), physical eviction (cf. PagedEviction), rounding retention to hardware granularity.
  2. Page importance is estimated from the final layer's attention and applied to prune all layers' KV, yet attention patterns differ markedly across layers (early layers are far more diffuse). This choice of design is not justified.
  3. The natural alternative to table rebuild — a per-tile eviction bitmask checked in-kernel, skipping fully-dead tiles each step — is not compared. The paper argues by assertion that amortized rebuild (once per G steps) beats per-step checks, but the reviewer cannot tell how much of the win comes from skipping dead tiles (achievable either way) versus dense sequential layout (compaction-specific, better for prefetching/locality).
Questions for authors
  1. Why use final-layer attention to drive pruning for every layer? What does a per-layer scoring ablation show?
  2. Did you evaluate a tile-eviction bitmask in the kernel (skip-if-dead per step) against compact-once? What is the measured per-step check cost versus the amortized rebuild cost?
  3. For Figure 1b: how do 18 retained pages compact to 18 tiles under 8-page tiling? And was the 85-tile scattered span measured on PagedEviction or or on PagePrune without compaction?
  4. How were μ = 0.5 and the sink/recent protection (1 + 2 pages = 48 tokens) chosen, and is 48 tokens of recency sufficient for the reasoning-heavy LongBench-v2 tasks?
4 comments

We sincerely thank the reviewer for carefully reading our manuscript and providing valuable comments and suggestions. Below, we respond to each concern point-by-point.

W1: The novelty seems incremental. The contribution is the combination and the gap framing, not a new selection principle, for example attention-mass page ranking (cf. Quest), EMA smoothing, sink + recent protection (cf. StreamingLLM/H2O), physical eviction (cf. PagedEviction), rounding retention to hardware granularity.

Response to W1: The main contribution of PagePrune lies in our co-design mechanism and the pruning–execution gap we close, rather than the isolated existence of ranking, smoothing, or eviction. For example, token-level eviction methods such as H2O, SnapKV, and StreamingLLM only care about which tokens are important, and ignore the page and tile structure of the paged KV cache, so the surviving state is neither released at page granularity nor aligned to the tiles that the attention kernel actually reads. Quest ignores memory release altogether. It keeps the full KV cache retained and only bounds which pages the kernel should read. PagedEviction releases physical blocks but ignores whether the surviving pages form a tile-executable set, so the backend still traverses a fragmented layout. An importance update must produce a protected, tile-executable set, release the corresponding physical KV capacity, and shorten the structure traversed by the backend.

W2: Page importance is estimated from the final layer's attention and applied to prune all layers' KV, yet attention patterns differ markedly across layers (early layers are far more diffuse). This choice of design is not justified. Q1: Why use final-layer attention to drive pruning for every layer? What does a per-layer scoring ablation show?

Response to W2 and Q1: We score pages with attention from the final layer. Early-layer attention is diffuse and yields low-contrast, weakly discriminative page scores, while the final layer concentrates mass on a few pages and, being closest to the output, ranks pages by their direct contribution to the next token. One shared mask therefore applies this output-driven criterion to every layer rather than layer-specific noise, and it simultaneously enables one compact shared layout across layers. Table 16, with additional results provided during rebuttal, confirms this choice empirically. The final layer achieves the best measured accuracy (12.82), throughput (188.21 tok/s), and latency (1.22 s) at the same 41.5 GB memory, outperforming layers 8, 17, and 26. The choice is thus both conceptually motivated and empirically grounded.

W3: The natural alternative to table rebuild — a per-tile eviction bitmask checked in-kernel, skipping fully-dead tiles each step — is not compared. The paper argues by assertion that amortized rebuild (once per G steps) beats per-step checks, but the reviewer cannot tell how much of the win comes from skipping dead tiles (achievable either way) versus dense sequential layout (compaction-specific, better for prefetching/locality). Q2: Did you evaluate a tile-eviction bitmask in the kernel (skip-if-dead per step) against compact-once? What is the measured per-step check cost versus the amortized rebuild cost?

Response to W3 and Q2: Following your suggestion, we provide this baseline in our revised Table 6 with G=1G=1, which indicates that the excessively frequent updates introduce additional computational latency, which negatively impacts the efficiency of the inference process.

Q3: For Figure 1b: how do 18 retained pages compact to 18 tiles under 8-page tiling? And was the 85-tile scattered span measured on PagedEviction or or on PagePrune without compaction?

Response to Q3: The 85-tile scattered span is measured from PagePrune, tile-aligned set before compact-once, not from other methods.

Q4: How were μ\mu = 0.5 and the sink/recent protection (1 + 2 pages = 48 tokens) chosen, and is 48 tokens of recency sufficient for the reasoning-heavy LongBench-v2 tasks?

Response to Q4: Table 17 sweeps the EMA coefficient on DREAM1K with Qwen3-VL-4B-Instruct over μ\mu, where μ\mu = 0.5 attains the highest ROUGE-L and the best reported throughput. Figure 4 ablates the two structural safeguards on long context benchmarks, which shows the safeguards are necessary. PagePrune includes not only safeguards but also page-level KV cache selection based on attention scores.

W2 and Q1 Thanks for extending the Table 16, my question is resolved.

W3 and Q2 Thanks for running the G=1 baseline, but it is not the baseline I proposed - it reruns the full pipeline every step, whereas a per-tile bitmask is a single bit test per tile, reusing the same G=64 selection. However, your reply to Reviewer xGpd mentions a mask-only mode using Quest's backend. Could you report mask-only throughput versus compact-once throughput at matched retained sets? That is the comparison Q2 asked for: it would separate how much of the win comes from skipping dead tiles (which the mask already achieves) from dense sequential layout (which only compaction achieves). Table 4 doesn't settle this as written, since its rows differ in memory (53.39 vs 61.67 vs 41.51 GB on DREAM1K) as well as layout, so the retained sets aren't matched.

Q3 Thanks for the clarification. On the first half, I withdraw the arithmetic question, I had misread "18" as retained pages.

Q4 Thanks for the response, my question is resolved.

We thank the reviewer for your following-up! We carefully checked the runs behind Table 4, and definitely agree with you that the metric definition needs to be stated more explicitly. The reported Memory entries are the peak of a device-level memory counter on the GPU, not the memory of our process alone, so they fold in memory held by co-located processes on the same card. In the run you identifies, the reading is dominated by the memory of other co-located processes resident at that time, which our execution logs record. In order to eliminate any confusion, we have re-measured the GPU memory on a clean GPU card and revised Table 4 accordingly, as shown in blue. Specifically, on DREAM1K, the +Scoring and +Compact rows now read 52.23 GB and 52.23 GB, and on EgoSchema they read 64.26 GB and 64.26 GB. The two rows hold exactly the same device-level memory because they share exactly the same retained set, which also confirms that compaction itself neither adds nor removes memory.

Regarding mask-only throughput versus compact-once throughput at matched retained sets, we would first like to clarify the structure of pageprune, which we believe is the source of the confusion. The three modules are structurally dependent rather than independently switchable. Scoring, which produces the per-page mask, determines the candidate set. Tile alignment rounds this set to kernel boundaries. Compaction packs the resulting tiles into a dense sequential layout. Compaction operates on the retained set produced by the upstream stages that are enabled before it. In the +Compact row of Table 4 the only upstream stage is scoring, so compaction there takes the retained set produced by scoring as its input. In the full pipeline it instead takes the tile-aligned set produced by scoring and tile alignment together. In either case, compact-once run in isolation, with no upstream stage, has no well-defined retained set to match against. If no pages were pruned, there would be no dead tiles to skip and no fragmented layout to repair. In Table 4, the +Scoring row is exactly the mask-only configuration you refers to, and the +Compact row is compact-once applied to precisely the same retained set, so these two rows provide the requested comparison. Mask-only throughput of 70.90 tokens per second versus compact-once throughput of 81.73 tokens per second at matched retained sets, with task quality unchanged and memory now visibly identical. Since disabling upstream stages makes downstream ablations ill-defined, the cumulative ablation in Table 4 is the only valid decomposition under these dependencies.

We are not sure whether this response fully addresses the question you care about, and we welcome further discussion on this point. We would also be happy to provide any additional experiments that could help clarify it.

Dear reviewer,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Reviewer_xGpdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PagePrune, a runtime KV cache pruning method for paged attention serving systems. It combines attention-based page scoring with exponential smoothing, protection of prefix and recent pages, tile-aligned retention, and block-table compaction. The method is implemented in vLLM and evaluated on language and VL benchmarks, reporting accuracy, memory usage, throughput, and latency. The paper reports up to 7.91x e2e speedup over vanilla vLLM.

Critical strengths

The paper addresses a practical serving problem. Translating KV cache pruning into actual execution savings is important, and the vLLM integration allows the paper to evaluate end-to-end behavior. The experiments also cover multiple model families, language and VL tasks, and evaluate both accuracy and serving efficiency. Component ablations and parameter sweeps provide useful evidence about the behavior of the complete implementation.

Critical weaknesses
  • The baseline comparisons mix algorithmic and systems advantages without adequately controlling for either. Against Quest, PagePrune changes both the selection policy and the cache-management pipeline, so the reported gains may come from systems optimizations rather than a better sparse attention algorithm. Against LMCache, the gains may instead come from additional sparsity and reduced KV accesses rather than a better systems design. A fair assessment should compare PagePrune’s sparse-attention policy with Quest under the same execution backend. Its systems optimizations should be compared with LMCache under a common sparse-attention policy and comparable KV workloads. Without these controls, the reported end-to-end gains do not establish whether PagePrune improves the sparse-attention algorithm, the serving system, or primarily benefits from combining capabilities that are not jointly enabled in the baselines.

  • The added value of the proposed selection rule is not sufficiently established. All pruned variants in Table 4 use the proposed scoring rule. Table 4 ablates components of PagePrune, but keeps its selection rule throughout. It therefore does not establish whether the proposed scoring rule is necessary, or whether an existing selection rule would achieve similar results. So it remains unclear whether the proposed scorer preserves better quality at matched actual retained page budgets, or achieves the same quality with a smaller budget.

Questions for authors
  1. Under the same optimized attention backend, does PagePrune achieve better accuracy/efficiency than Quest? For example, can it preserve the same task quality while using fewer KV pages for attention? This comparison would help distinguish an advantage of the selection algorithm from gains due to the cache management pipeline.

  2. What sparse attention policy is enabled in the LMCache baseline, if any? When LMCache and PagePrune use the same sparse attention policy and KV workloads, what latency/throughput/memory advantages remain from PagePrune’s systems optimizations? Could you also specify the enabled LMCache features and cache-reuse conditions?

3 comments

We sincerely thank the reviewer for their careful reading of our manuscript and for providing valuable comments and suggestions. Below, we respond to each concern point-by-point.

W1: The baseline comparisons mix algorithmic and systems advantages without adequately controlling for either.

Response to W1: We isolate the algorithmic dimension by running PagePrune in mask-only mode, which uses the same execution backend of Quest, so the higher accuracy of our method over Quest proves the algorithmic advantage is real and not borrowed from our eviction, recovery, or compaction pipeline. We isolate the systems dimension through Table 4, which keeps the retained set identical to the mask-only baseline yet still gains throughput from scoring alone. Scoring alone cuts the retained KV set at unchanged quality but slightly dips throughput to 70.90 tokens per second, because the kept pages are scattered. Adding only compact-once, with the same retained set, raises throughput to 81.73 tokens per second by repairing the layout. The full pipeline then reaches 188.21 tokens per second, at essentially the same retention and flat quality. The systems benefit thus comes from the execution-aware layout pipeline, not from which pages are selected. These results and analysis confirm that our method delivers a genuine systems benefit independent of which sparse policy is used.

W2: The added value of the proposed selection rule is not sufficiently established.

Response to W2: Table 4 is a moduler ablation, not a scorer ablation. Tile alignment and compact-once act on the keep mask produced by pruning, so without pruning neither mechanism exists and every row must share this pruning. That scorer is deliberately minimal, plain top-k over smoothed attention mass, which makes the pipeline method agnostic and the reported modular gains independent of the selection rule. We do not claim the pruning as novel. Its value is established separately against Quest, PagedEviction, and Random Selection, where PagePrune preserves better quality.

Q1: Under the same optimized attention backend, does PagePrune achieve better accuracy/efficiency than Quest? For example, can it preserve the same task quality while using fewer KV pages for attention? This comparison would help distinguish an advantage of the selection algorithm from gains due to the cache management pipeline.

Response to Q1: We compare both methods on the same attention backend, with PagePrune in mask-only mode to isolate selection from cache management. On long-context retrieval, PagePrune maintains near-perfect accuracy under compression, while Quest degrades, demonstrating better KV selection at comparable or smaller retention budgets. On ActCap, both preserve vanilla task quality, but PagePrune achieves 115.86 token/s versus 28.21 token/s for Quest. These results establish that PagePrune's advantage stems not only from cache management but also from a more effective selection strategy.

Q2: What sparse attention policy is enabled in the LMCache baseline, if any? When LMCache and PagePrune use the same sparse attention policy and KV workloads, what latency/throughput/memory advantages remain from PagePrune’s systems optimizations? Could you also specify the enabled LMCache features and cache-reuse conditions?

Response to Q2: LMCache is a KV reuse and offloading layer attached through the vLLM KV transfer connector, which stores and retrieves the complete KV cache, never discards tokens or pages, and exposes no selection mask. Retention ratio is an output of PagePrune, not an independent hyperparameter that can be held fixed across two systems without changing what those systems are. All comparisons across experiments are based on the same hardware environment, rather than sparse attention. We ensure our comparison is fair.

Dear reviewer,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Dear reviewer,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

Reviewer_48NHOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents PagePrune, a page-level KV-cache pruning framework for long-context inference in vLLM. It combines attention-based page selection with tile-aligned retention, block-table compaction and periodic recovery of evicted KV states. Experiments are broad, and cover text, vision-language, and video-language workloads. The paper reports substantial gains in memory reductions and E2E speedup.

Critical strengths
  1. The paper highlights an important problem: how retained KV pages are organized affects whether pruning translates into faster inference. Jointly considering page selection and attention-kernel execution provides a useful perspective on Kv cache optimization.
  2. The evaluation span multiple models, model sizes, and modalities. This is complemented by ablations on scoring layers, retention budgets, pruning intervals, and tile sizes. This breadth helps examine the method’s applicability and the effects of its design choices.
Critical weaknesses
  1. Inconsistency between algorithm and results. The stated algorithm must retain at least the target fraction τ\tau, since safeguards and tile alignment only expand the selected set. However, Table 15 reports 18.69% retention at τ=0.2\tau=0.2 and 9.26% at τ=0.1\tau=0.1. The authors should reconcile these results with the algorithm and the stated retention metric.
  2. Insufficiently specified KV recovery and overhead accounting. Reconstructing evicted multilayer KV states requires preceding context, but the paper does not explain dependency handling, preserved inputs, batching, or whether reconstruction is exact. Table 7’s overhead percentages also do not reconcile with the main-table latencies. E.g., (0.040/1.22=3.28%), rather than 1.8%.
  3. Unclear throughput definition and pruning schedule. Several throughput-latency products are approximately two tokens, including across different batch sizes, which requires explanation given the stated aggregate decode-throughput definition. If these workloads generate only a few tokens, the authors should clarify when pruning occurs under the default 64-step interval, including whether an initial pruning operation occurs before decoding.
  4. Unclear peak-memory reductions. Restoring all previously evicted pages before rescoring appears to recreate the full per-request KV working set, yet substantial reductions in maximum GPU memory are reported. The authors should provide memory timelines covering prefill and restoration, distinguish occupied KV memory from reserved memory, and specify whether restoration is simultaneous or streamed.
  5. Layout benefits are not cleanly isolated. Table 4 does not establish that compaction configurations use identical retained KV sets, while Figure 3 shows abrupt memory changes that tile expansion alone cannot explain. Non-monotonic throughput is plausible, but supporting the proposed mechanism requires comparisons with fixed retained pages, actual KV footprints, executed tile counts, and separate kernel and management timings.
  6. Performance dropoffs. InfiniteBench-4B accuracy falls from 16.17 to 13.13 (-19%), while NIAH-8B accuracy falls from 100% to 90.9%. The authors should quantify “comparable quality”, or avoid overclaiming the performance results.
  7. Unquantified retention expansion from tile alignment. Selecting one page in every tile preserves the entire cache, potentially eliminating pruning benefits, but the paper does not establish how often near-full retention occurs across workloads. Beyond Table 15’s single-workload sweep, the authors should report pre-/post-alignment retention and tile occupancy distribution. Furthermore, as the retained set is shared across layers, layerwise retained attention mass would be more informative than separate layerwise retention probabilities.
  8. Limited justification and profiling of final layer scoring. Sec 3 describes attention-statistic collection without an additional query–key pass, and Table 16 supports final-layer scoring on one model and workload, but the collection overhead and generality of this choice remain unclear. Broader layer-selection evidence, layerwise attention coverage, and an explanation of Table 16’s large latency differences would strengthen the justification.
  9. Missing hardware details. The paper does not explicitly report the number of GPUs or tensor/pipeline parallel settings for each experiment. The author should also clarify whether memory measurements are per GPU or aggregated.
Questions for authors

Please see weaknesses, thanks.

5 comments

We sincerely thank the reviewer for carefully reading our manuscript and providing valuable comments and suggestions. Below, we respond to each concern point-by-point.

W1: Inconsistency between algorithm and results.

Response to W1: The two τ\tau values are taken from a separate diagnostic calibration in which the minimum-retention and minimum-hot-page floors are disabled to expose the unconstrained effect of τ\tau. We have corrected the table so that every row reports the default configuration, under which τ=0.2\tau=0.2 and τ=0.1\tau=0.1 both retain 23.60 percent, satisfying the bound like all remaining rows.

W2: Insufficiently specified KV recovery and overhead accounting.

Response to W2: The recovery mechanism keeps the KV cache exact while saving memory. Every percentage of E2E latency in Table 7 is computed within the same controlled measurement, namely the per-request recovery time divided by that same run's mean E2E latency. For DREAM1K with Qwen3-VL-4B-Instruct, the 0.04 seconds of recovery divided by that configuration's own mean E2E latency of about 2.22 seconds yields exactly the reported 1.8 percent. The 1.22 seconds the reviewer divides by is a latency quoted from the main results table, which is measured under a different experimental configuration with different experiment runs. We respectfully remind the reviewer that dividing a recovery time taken from one experiment by a latency taken from another experiment should be an invalid cross-experiment comparison, where the two tables report latencies under different experiment runs.

W3: Unclear throughput definition and pruning schedule.

Response to W3: We follow the standard aggregate serving throughput used in the official vLLM serving benchmark and in the serving literature, namely the total number of output tokens generated across all evaluated requests divided by the total wall clock time of the run [1,2]. We sincerely remind the reviewer that our PagePrune targets long-context workloads, where the input prompt alone already produces a large KV cache of lots of pages during prefill given a long video or document as the input (usually longer than 64 tokens). Therefore, the first pruning decision is applied to this prefill KV cache, before the first output token is generated, and it is the prompt pages that are scored, aligned, and released. The interval of 64 steps governs how often the retention decision is refreshed during decoding, starting with the initial decision.

W4: Unclear peak-memory reductions.

Response to W4: Following your suggestion, we have revised this confusing description around Line 155, 188 and 213. Specifically, pruned pages are not restored before rescoring. First, eviction frees only the GPU KV blocks while every page retains a lightweight importance record made of its accumulated attention scores and a cached per-page key summary, so a pruned page can be re-ranked without its KV being fetched back. Second, at each decision we recover only the pruned pages that re-enter the retained set under the budget, and we do so in a streamed manner by merging a request's re-entering pages into one contiguous suffix that is re-prefilled, so the retained set stays bounded by the retention target and never reconstructs the full context. We add a memory timeline in Appendix G of our revised manuscript.

W5: Layout benefits are not cleanly isolated.

Response to W5: Tile alignment may retain slightly more KV pages than pure pruning, but this overhead is offset by improved layout regularity. Fragmented pages incur scattered gathers and skip-mask overhead at every decode step, whereas tile-aligned, compacted pages form a denser working set. The modules are structurally dependent rather than independently switchable. Pruning determines the candidate set, tile alignment rounds it to kernel boundaries, and compaction packs the resulting pages. Thus, disabling upstream stages makes downstream ablations ill-defined. Table 4 therefore uses cumulative ablations, the only valid decomposition under these dependencies. Crucially, the compact once configuration increases throughput from 70.90 to 81.73 token/s at unchanged task quality. This directly demonstrates that execution layout contributes independently to performance, beyond reducing the retained KV footprint.

[1] Zirui Liu, Jiayi Yuan, Hongye Jin, Shaochen Zhong, Zhaozhuo Xu, Vladimir Braverman, Beidi Chen, and Xia Hu. Kivi: A tuning-free asymmetric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02750, 2024c.

[2] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody H Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E Gonzalez, et al. Sglang: Efficient execution of structured language model programs. Advances in neural information processing systems, 37:62557–62583, 2024.

W6: Performance dropoffs.

Response to W6: We respectfully disagree that the highlighted cases are representative of PagePrune's overall quality--efficiency trade-off. Across the 14 accuracy-based configurations in Tables 1 and 2, PagePrune exhibits an average accuracy change of only −0.48-0.48 percentage points (pp) relative to the vLLM baseline, with 12/14 configurations within 1 pp. In comparison, Quest and PagedEviction show larger average accuracy decreases of −1.55-1.55 and −1.37-1.37 pp, respectively. Meanwhile, PagePrune achieves a 2.54×2.54\times geometric-mean E2E speedup, compared with 1.01×1.01\times, 1.52×1.52\times, and 1.60×1.60\times for Quest, PagedEviction, and LMCache. Similar quality–efficiency trends are observed in Table 3, where PagePrune maintains comparable ROUGE-L scores while achieving substantial E2E speedups.

We acknowledge the larger degradation on InfiniteBench-4B (−3.04-3.04 pp, or −18.8%-18.8\% relative) and NIAH-8B (−9.1-9.1 pp). However, these cases should be considered alongside the complete evaluation: PagePrune achieves a 2.80×2.80\times geometric-mean E2E speedup across all 18 configurations, outperforming the strongest competing baseline in 15/18 cases. Moreover, its NIAH-8B accuracy of 90.9%90.9\% substantially exceeds Quest (26.2%26.2\%) and PagedEviction (45.2%45.2\%). These quantitative results support our characterization of comparable quality on most evaluated workloads, rather than uniform lossless preservation. We will clarify this distinction and explicitly report the aggregate quality differences in the revised manuscript.

W7: Unquantified retention expansion from tile alignment.

Response to W7: For retention expansion, tile alignment trades additional retention for tile-level execution rather than enforcing τ\tau as a strict upper bound. At τ=0.3\tau=0.3, our sweep reports 33.36% actual retention, reducing KV memory from 1.52 to 0.43 GB. The full system also reduces peak GPU memory from 73.67 to 41.51 GB on DREAM1K and from 78.35 to 55.39 GB on EgoSchema. These results demonstrate that substantial pruning benefits remain with alignment, but do not quantify the frequency of near-full retention. That question requires pre-/post-alignment retention and tile occupancy statistics. The target ratio alone cannot isolate expansion because safeguards also add pages.

For cross-layer coverage, a shared mask gives identical page-retention ratios across layers, not identical retained attention mass. The appropriate diagnostic is each layer’s full-context attention mass assigned to the same final retained set, measured before renormalization over retained pages. Our scoring-layer ablation supports final-layer scoring through downstream quality and efficiency, but answers a different question—which layer to use for selection, rather than how well that selection covers every layer.

W8: Limited justification and profiling of final layer scoring.

Response to W8: The statistics are per-page key summaries maintained incrementally as tokens arrive, so no additional query-key pass is performed. Scoring is running only at pruning decisions, at a measured cost of 0.1 to 3.8 percent of E2E time. To test generality, we add additional results about EgoSchema on Qwen3-VL-4B-Instruct to Table 16. We observe that scoring layers 8, 17, and 26 all give 73.20 percent accuracy, and all layers and vanilla produce identical predictions on every sample. In our method, final-layer scores effectively predict the pages consumed when generating the next token and need the fewest recomputes.

W9: Missing hardware details.

Response to W9: Each individual reported inference run uses a single serving process on one NVIDIA Pro 6000 GPU (95GB GPU memory) with tensor parallelism = 1, pipeline parallelism = 1 and data parallelism = 1. We have mentioned this hardware information in lines 254–255.

Dear reviewer,

Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.

We sincerely look forward to your feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.