- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposed an execution-aware, page-granular KV-cache pruning system built natively on vLLM's PagedAttention, named PagePrune. On paged serving systems, evicting KV pages reduces memory capacity but does not reduce memory traffic, because the attention kernel iterates positionally over the block table and therefore still traverses evicted regions. Every G decoding steps, PagePrune ranks pages by exponentially-smoothed attention mass, retains the top-τ fraction plus sink/recent page protection, expands the keep-set to the kernel's 8-page tile boundaries, frees the remaining physical blocks, and rebuilds the block table densely over the survivors (keeping a logical-index map so compacted entries can be mapped back to original positions for RoPE). Evicted pages are recomputed from stored token inputs before the next re-ranking round. Evaluated on different models and benchmarks, the paper reports up to 3.86× (text), 5.22× (VQA), and 7.91× (video summarization) end-to-end speedups over vanilla vLLM at roughly comparable quality.
Critical strengths
- the paper is well written and easy to understand.
- The core observation is real and well-evidenced, as in Figure 1.
- The ablation well supports the motivation. Page scoring alone cuts memory (73.67 → 53.39 GB on DREAM1K) yet reduces throughput (77.36 → 70.90 tok/s) — a clean demonstration that capacity savings without layout repair can be net-negative, since the kernel still walks the scattered span plus selection overhead. The gains appear only once tile alignment and compaction are added.
- Deployability. Native vLLM integration with no custom sparse kernel and no changes to the hot attention loop.
- Extensive experiments and ablation study are conducted, including different benchmarks and models plus sensitivity studies over pruning interval G, tile size K, and batch size, and an explicit recomputation-overhead analysis.
Critical weaknesses
- The novelty seems incremental. The contribution is the combination and the gap framing, not a new selection principle, for example attention-mass page ranking (cf. Quest), EMA smoothing, sink + recent protection (cf. StreamingLLM/H2O), physical eviction (cf. PagedEviction), rounding retention to hardware granularity.
- Page importance is estimated from the final layer's attention and applied to prune all layers' KV, yet attention patterns differ markedly across layers (early layers are far more diffuse). This choice of design is not justified.
- The natural alternative to table rebuild — a per-tile eviction bitmask checked in-kernel, skipping fully-dead tiles each step — is not compared. The paper argues by assertion that amortized rebuild (once per G steps) beats per-step checks, but the reviewer cannot tell how much of the win comes from skipping dead tiles (achievable either way) versus dense sequential layout (compaction-specific, better for prefetching/locality).
Questions for authors
- Why use final-layer attention to drive pruning for every layer? What does a per-layer scoring ablation show?
- Did you evaluate a tile-eviction bitmask in the kernel (skip-if-dead per step) against compact-once? What is the measured per-step check cost versus the amortized rebuild cost?
- For Figure 1b: how do 18 retained pages compact to 18 tiles under 8-page tiling? And was the 85-tile scattered span measured on PagedEviction or or on PagePrune without compaction?
- How were μ = 0.5 and the sink/recent protection (1 + 2 pages = 48 tokens) chosen, and is 48 tokens of recency sufficient for the reasoning-heavy LongBench-v2 tasks?
4 comments
W2 and Q1 Thanks for extending the Table 16, my question is resolved.
W3 and Q2 Thanks for running the G=1 baseline, but it is not the baseline I proposed - it reruns the full pipeline every step, whereas a per-tile bitmask is a single bit test per tile, reusing the same G=64 selection. However, your reply to Reviewer xGpd mentions a mask-only mode using Quest's backend. Could you report mask-only throughput versus compact-once throughput at matched retained sets? That is the comparison Q2 asked for: it would separate how much of the win comes from skipping dead tiles (which the mask already achieves) from dense sequential layout (which only compaction achieves). Table 4 doesn't settle this as written, since its rows differ in memory (53.39 vs 61.67 vs 41.51 GB on DREAM1K) as well as layout, so the retained sets aren't matched.
Q3 Thanks for the clarification. On the first half, I withdraw the arithmetic question, I had misread "18" as retained pages.
Q4 Thanks for the response, my question is resolved.
We thank the reviewer for your following-up! We carefully checked the runs behind Table 4, and definitely agree with you that the metric definition needs to be stated more explicitly. The reported Memory entries are the peak of a device-level memory counter on the GPU, not the memory of our process alone, so they fold in memory held by co-located processes on the same card. In the run you identifies, the reading is dominated by the memory of other co-located processes resident at that time, which our execution logs record. In order to eliminate any confusion, we have re-measured the GPU memory on a clean GPU card and revised Table 4 accordingly, as shown in blue. Specifically, on DREAM1K, the +Scoring and +Compact rows now read 52.23 GB and 52.23 GB, and on EgoSchema they read 64.26 GB and 64.26 GB. The two rows hold exactly the same device-level memory because they share exactly the same retained set, which also confirms that compaction itself neither adds nor removes memory.
Regarding mask-only throughput versus compact-once throughput at matched retained sets, we would first like to clarify the structure of pageprune, which we believe is the source of the confusion. The three modules are structurally dependent rather than independently switchable. Scoring, which produces the per-page mask, determines the candidate set. Tile alignment rounds this set to kernel boundaries. Compaction packs the resulting tiles into a dense sequential layout. Compaction operates on the retained set produced by the upstream stages that are enabled before it. In the +Compact row of Table 4 the only upstream stage is scoring, so compaction there takes the retained set produced by scoring as its input. In the full pipeline it instead takes the tile-aligned set produced by scoring and tile alignment together. In either case, compact-once run in isolation, with no upstream stage, has no well-defined retained set to match against. If no pages were pruned, there would be no dead tiles to skip and no fragmented layout to repair. In Table 4, the +Scoring row is exactly the mask-only configuration you refers to, and the +Compact row is compact-once applied to precisely the same retained set, so these two rows provide the requested comparison. Mask-only throughput of 70.90 tokens per second versus compact-once throughput of 81.73 tokens per second at matched retained sets, with task quality unchanged and memory now visibly identical. Since disabling upstream stages makes downstream ablations ill-defined, the cumulative ablation in Table 4 is the only valid decomposition under these dependencies.
We are not sure whether this response fully addresses the question you care about, and we welcome further discussion on this point. We would also be happy to provide any additional experiments that could help clarify it.
Dear reviewer,
Thank you once again for your valuable comments on our submission. As the discussion phase is approaching its end, we would like to kindly confirm whether we have addressed all of your concerns. Should there be any remaining questions requiring further clarification, please do not hesitate to let us know. If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.
We sincerely look forward to your feedback.
We sincerely thank the reviewer for carefully reading our manuscript and providing valuable comments and suggestions. Below, we respond to each concern point-by-point.
W1: The novelty seems incremental. The contribution is the combination and the gap framing, not a new selection principle, for example attention-mass page ranking (cf. Quest), EMA smoothing, sink + recent protection (cf. StreamingLLM/H2O), physical eviction (cf. PagedEviction), rounding retention to hardware granularity.
Response to W1: The main contribution of PagePrune lies in our co-design mechanism and the pruning–execution gap we close, rather than the isolated existence of ranking, smoothing, or eviction. For example, token-level eviction methods such as H2O, SnapKV, and StreamingLLM only care about which tokens are important, and ignore the page and tile structure of the paged KV cache, so the surviving state is neither released at page granularity nor aligned to the tiles that the attention kernel actually reads. Quest ignores memory release altogether. It keeps the full KV cache retained and only bounds which pages the kernel should read. PagedEviction releases physical blocks but ignores whether the surviving pages form a tile-executable set, so the backend still traverses a fragmented layout. An importance update must produce a protected, tile-executable set, release the corresponding physical KV capacity, and shorten the structure traversed by the backend.
W2: Page importance is estimated from the final layer's attention and applied to prune all layers' KV, yet attention patterns differ markedly across layers (early layers are far more diffuse). This choice of design is not justified. Q1: Why use final-layer attention to drive pruning for every layer? What does a per-layer scoring ablation show?
Response to W2 and Q1: We score pages with attention from the final layer. Early-layer attention is diffuse and yields low-contrast, weakly discriminative page scores, while the final layer concentrates mass on a few pages and, being closest to the output, ranks pages by their direct contribution to the next token. One shared mask therefore applies this output-driven criterion to every layer rather than layer-specific noise, and it simultaneously enables one compact shared layout across layers. Table 16, with additional results provided during rebuttal, confirms this choice empirically. The final layer achieves the best measured accuracy (12.82), throughput (188.21 tok/s), and latency (1.22 s) at the same 41.5 GB memory, outperforming layers 8, 17, and 26. The choice is thus both conceptually motivated and empirically grounded.
W3: The natural alternative to table rebuild — a per-tile eviction bitmask checked in-kernel, skipping fully-dead tiles each step — is not compared. The paper argues by assertion that amortized rebuild (once per G steps) beats per-step checks, but the reviewer cannot tell how much of the win comes from skipping dead tiles (achievable either way) versus dense sequential layout (compaction-specific, better for prefetching/locality). Q2: Did you evaluate a tile-eviction bitmask in the kernel (skip-if-dead per step) against compact-once? What is the measured per-step check cost versus the amortized rebuild cost?
Response to W3 and Q2: Following your suggestion, we provide this baseline in our revised Table 6 with G=1, which indicates that the excessively frequent updates introduce additional computational latency, which negatively impacts the efficiency of the inference process.
Q3: For Figure 1b: how do 18 retained pages compact to 18 tiles under 8-page tiling? And was the 85-tile scattered span measured on PagedEviction or or on PagePrune without compaction?
Response to Q3: The 85-tile scattered span is measured from PagePrune, tile-aligned set before compact-once, not from other methods.
Q4: How were μ = 0.5 and the sink/recent protection (1 + 2 pages = 48 tokens) chosen, and is 48 tokens of recency sufficient for the reasoning-heavy LongBench-v2 tasks?
Response to Q4: Table 17 sweeps the EMA coefficient on DREAM1K with Qwen3-VL-4B-Instruct over μ, where μ = 0.5 attains the highest ROUGE-L and the best reported throughput. Figure 4 ablates the two structural safeguards on long context benchmarks, which shows the safeguards are necessary. PagePrune includes not only safeguards but also page-level KV cache selection based on attention scores.