- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
GrowPage is an online per-request KV-capacity controller for long-output reasoning serving. It maintains short- and long-timescale exponential moving averages of pre-RoPE queries. At a cache-capacity boundary, the two summaries induce attention working sets over historical keys; the difference between their mean working-set ratios is used as a demand-trend signal. If demand appears to broaden, the request grows by one PagedAttention block; otherwise, historical KVs are selected and compacted to create one block of free space while retaining the current physical allocation. The method is implemented on nano-vLLM with Zipage-style asynchronous compression and evaluated on two 8B reasoning models, five benchmarks, and mixed workloads.
Critical strengths
-
Practical integration with a paged serving runtime. The paper goes beyond a retention score and incorporates its capacity controller into page allocation and asynchronous compaction. It also considers preemption and request recovery.
-
Informative controls for the demand signal. The future-demand correlation and counterfactual branches support the proposed signal. The random and inverse policies further indicate that the timing of capacity decisions matters.
Critical weaknesses
-
The novelty claim omits a closely related adaptive-budget method. GVote (arXiv:2509.03136) also predicts future-query demand to avoid a manually fixed KV budget. I regard this as very close to the motivation of GrowPage, yet it is neither discussed nor compared. Without a clear distinction and controlled comparison, I am not convinced by the claim that existing methods keep total per-request capacity fixed.
-
I could not reconcile the serving results. My calculation from Tables 1, 5, and 7 gives an average DeepSeek-R1/AMC23 output of about 6,848 tokens. Under the stated single-A100 setting, roughly 54 GiB remains for KV storage; at 128 KiB per token, FullKV can keep only about 60 such requests resident. The reported aggregate throughput would therefore require about 51 ms/token per resident request, but Table 7 reports 1,027 ms/token. I may be missing an undocumented scheduling convention, but the paper does not provide enough information to close this roughly 20x gap. The overhead analysis is also incomplete because preempted requests require re-prefill, while Figure 8 reports only selected per-event costs. I therefore do not find the headline throughput gains verifiable from the submitted results.
Questions for authors
-
What initial and maximum per-request capacities were used for GrowPage? How does the comparison with Zipage change when average KV use and resident concurrency are matched?
-
How is TPOT defined, and what time intervals does it include? Could the authors explain how the DeepSeek-R1/AMC23 FullKV throughput is consistent with the reported TPOT and memory budget?
-
Does the reported overhead include the cumulative cost of preemption and re-prefill? How do these operations affect end-to-end latency under concurrent serving?
-
How does GrowPage differ technically from GVote? A comparison under the same runtime and resource constraints would materially affect the novelty assessment.
4 comments
Response to K1Za (2/2)—KV Capacity, Zipage, and GVote
Q3. KV capacity settings and comparison with Zipage
We selected GrowPage's initial per-request KV capacities with reference to budgets evaluated in prior work, including Zipage and R-KV.
| Workload | Initial KV capacity (tokens) |
|---|---|
| MATH500 | 2048 |
| AMC23 | 2048 |
| AIME24 | 4096 |
| LiveCodeBench v1 | 4096 |
| GSM8K | 1024 |
| Mixed workload (Fig. 5) | 1024 |
The implementation supports an optional per-request capacity cap. We left it unset in the reported experiments, so growth is limited by available pages in the shared GPU KV pool. We will clarify these settings in the revised paper.
We added a 3072-token Zipage point to the Qwen3-8B mixed-workload experiment in Fig. 5 to compare the methods at similar mean resident concurrency.
| Method | Per-request KV budget (tokens) | Mean KV-resident requests | Pass@1 (%) | TPS (tok/s) |
|---|---|---|---|---|
| Zipage-3072 | 3072 (fixed) | 127 | 81.4 | 2191 |
| GrowPage | 3442 (mean) | 121 | 84.9 | 2213 |
GrowPage improves Pass@1 by 3.5 percentage points at comparable throughput (+1.0%). In Fig. 5, it also approaches Zipage-8192's Pass@1 (84.9% vs. 85.1%) while delivering about 1.9× the throughput.
Q4. Technical distinction and nano-vLLM comparison with GVote
Thank you for pointing out GVote (Tang et al., arXiv:2509.03136). We overlooked this closely related work and will cite it and compare the two approaches in the revised manuscript.
Novelty. GrowPage’s core contribution is to treat KV cache capacity as a resource shared by concurrent requests and adjust each request’s page allocation online during decoding.
- GVote compresses prompt KV, while GrowPage compresses accumulated KV history. GrowPage also compresses KV from generated tokens, which can account for most of the cache when reasoning outputs far exceed their prompts.
- GVote sets prompt KV retention at prefill, while GrowPage decides whether to grow during decoding. Observation 2 shows that attention demand can shift substantially within a request, making it difficult to size a budget for the entire decode from prefill information alone. When a request fills its current allocation, GrowPage uses observed decoding queries to decide whether to request another page or compact within its current capacity.
- GVote requires adaptations for standard prefix caching and PagedAttention, while GrowPage uses both.
- Prefix caching. GVote selects KV entries based on the full prompt. Requests sharing a token prefix can therefore produce different compacted pages, making those pages difficult to reuse through standard prefix matching.
- PagedAttention. GVote can retain a different number of tokens per KV head, while Nano-vLLM allocates physical pages shared by all heads of a request. Reclaiming physical pages through per-head compression requires adapting the page layout and allocator.
Same-runtime comparison on AMC23. To address the reviewer’s request, we reimplemented GVote from the paper in Nano-vLLM because a public reference implementation was unavailable. We refer to this version as GVote-Prefill (paged adaptation). Gvote_prefill JSON log
- Prefix caching. We disabled prefix caching for the GVote adaptation because the compacted pages depend on the full prompt.
- Paged KV layout. For each request, we kept GVote’s selections for each KV head in each layer and used the largest retained count across layers and heads as the common cache length. We filled shorter selections with additional valid KV entries ranked by the actual prefill query’s attention scores, packed the results into whole pages, and released the surplus pages.
- Position handling. We preserved the original RoPE positions and continued advancing logical positions during decoding, while using the compacted physical length for cache addressing.
| Method (Qwen3-8B, AMC23) | Pass@1 (%) ↑ | Throughput (tokens/s) ↑ | Elapsed time (h) ↓ |
|---|---|---|---|
| GVote-Prefill (paged adaptation) | 91.6 | 1162 | 2.71 |
| GrowPage | 91.4 | 2261 | 1.42 |
GrowPage achieves 1.95× the throughput and about 48% less elapsed time, with a Pass@1 difference of only 0.2 percentage points. On AMC23, prompts are short relative to the generated reasoning traces, so prompt compression can reduce only a limited share of total KV usage.
The published GVote method uses a head-wise variable-length layout. In our paged adaptation, the additional KV entries used to equalize head lengths participate in attention, so the adaptation may affect both accuracy and serving efficiency.
We believe that algorithmic contribution, compatibility with existing inference engines, and measured end-to-end serving gains are all important criteria for evaluating KV cache compression.
Thank you for the detailed responses and additional experiments. The TPOT definition and inclusion of preemption costs address my main concerns about the serving measurements, and the technical distinction from GVote is clearer. However, the Zipage comparison does not fully match average KV usage, and the GVote adaptation limits attribution of the throughput gains. Overall, the rebuttal provides sufficient clarification and additional evidence for me to raise my score to 3.
Thank you for your careful reassessment and for the time and effort you have devoted to reviewing our paper. Your constructive feedback helped us address important weaknesses and improve the clarity and rigor of the manuscript.
We appreciate your understanding and your recognition of the clarifications and additional evidence provided during the rebuttal.
Response to K1Za (1/2)—TPOT and Preemption
Thank you for raising these questions about the serving measurements.
Q1. TPOT, throughput, and KV residency
FullKV’s high TPOT is consistent with its throughput and GPU KV usage because the request-averaged metric includes time spent waiting after the first token, even after preempted requests have released their KV pages.
For request i, Di is
gen_timein seconds from the first to the last output token, and Ni isoutput_len. This interval includes post-first-token waiting and re-prefill. Initial queueing and prefill precede it. Table 7 averages 1000Di/(Ni−1) across requests.We could not locate the original FullKV (nano-vLLM) JSON log, so we illustrate the TPOT calculation using the FullKV (vLLM) JSON log for DeepSeek-R1-Distill-Llama-8B on AMC23. Across 1280 requests, we obtain:
= \frac{1272976.3484}{1280} \approx 994.5128\\,\mathrm{ms/token}.$$ This is within 1 ms/token of the **994 ms/token** reported in Table 7 for the same FullKV (vLLM) setting. | Metric | FullKV (vLLM) | | --- | ---: | | Total output tokens | 8897584 | | Workload elapsed time, $T$ | 7199.52 s | | Throughput | 1235.8568 tok/s | | Token-interval-weighted TPOT, $1000\sum_i D_i/\sum_i(N_i-1)$ | 496.92 ms/token | | Mean requests between first and last token, $\sum_i D_i/T$ | 614.03 | | Mean running / waiting requests from scheduler logs | 72.28 / 543.22 | **Effect of waiting on TPOT.** If a preempted request waits $W_i$ seconds for readmission after emitting its first token, those seconds increase $D_i$ without generating new tokens, adding $1000W_i/(N_i-1)$ ms/token to its TPOT. Re-prefill further extends $D_i$ as the KV cache is rebuilt. Other requests can reuse the released pages and keep contributing to aggregate throughput. In this FullKV (vLLM) run, multiplying the post-first-token output rate of 1235.68 intervals/s by the interval-weighted TPOT of 0.49692 s/token gives an average of 614.03 requests between their first and last tokens. Using the sampled running count of 72.28 as a proxy, dividing that count and the gap of 541.75 by the output interval rate gives about **58.5 ms of running request-time** and **438.4 ms of non-running request-time** per output interval, respectively. The running-time estimate is on the same scale as the reviewer's 51 ms/token. The gap in request counts is close to the sampled waiting count of 543.22, suggesting substantial post-first-token queueing. This split is approximate because the scheduler counts can also include requests before their first token. For a pause of the same duration, shorter outputs incur a larger TPOT increase. Request averaging gives these outputs more relative weight than interval weighting, helping explain the difference between 994.51 and 496.92 ms/token in this run. **Q2. Preemption, re-prefill, and end-to-end latency** Fig. 8 profiles selected per-event capacity-control and compaction operations. **Our reported throughput uses the full workload elapsed time, including preemption and re-prefill.** We measured the following in a separate GrowPage rerun using Qwen3-8B on AMC23. | Metric | Value | | --- | ---: | | Total requests | 1280 | | End-to-end elapsed time | 5150.60 s | | End-to-end throughput | 2244.45 tok/s | | Preemption events | 1406 | | Cumulative host-side preemption time | 0.062842 s | | Cumulative re-prefill model-call wall time | 136.59 s | | Re-prefill model-call wall time / workload wall time | 2.65% | | Requests preempted at least once | 294 (22.97%) | | Mean preemptions per affected request | 4.78 | | Mean end-to-end latency difference (preempted vs. never preempted) | +56.8% | The +56.8% is an unadjusted difference between the two request groups. Preempted requests generally generated longer outputs, which also contribute to the observed latency gap.