acceptodds
Under review as a conference paper at ICLR 2027

GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving

Abstract

Long-output reasoning has made the key–value (KV) cache a critical memory bottleneck for efficient LLM serving. Existing KV compression methods usually rely on a predefined per-request budget and adjust only which KV states are retained, leaving the total capacity fixed throughout decoding. However, reasoning workloads exhibit substantial demand variation: different requests require different KV capacities, and the attention demand of an individual request evolves during generation. We introduce GrowPage, an on-demand KV budgeting framework that treats KV capacity as a runtime resource. GrowPage maintains lightweight dual-timescale query summaries to capture recent and long-term attention behaviors, and uses their relative attention working sets to estimate demand evolution. At each capacity boundary, GrowPage either compresses KV states within the current allocation or acquires an additional physical page when broader demand emerges. By integrating with PagedAttention's page-level memory abstraction, GrowPage preserves continuous batching and prefix caching. Experiments on reasoning benchmarks across multiple models show that GrowPage achieves a superior performance–throughput trade-off over existing approaches.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Serving, KV Cache Compression, Dynamic Memory Management, PagedAttention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_K1ZaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

GrowPage is an online per-request KV-capacity controller for long-output reasoning serving. It maintains short- and long-timescale exponential moving averages of pre-RoPE queries. At a cache-capacity boundary, the two summaries induce attention working sets over historical keys; the difference between their mean working-set ratios is used as a demand-trend signal. If demand appears to broaden, the request grows by one PagedAttention block; otherwise, historical KVs are selected and compacted to create one block of free space while retaining the current physical allocation. The method is implemented on nano-vLLM with Zipage-style asynchronous compression and evaluated on two 8B reasoning models, five benchmarks, and mixed workloads.

Critical strengths
  1. Practical integration with a paged serving runtime. The paper goes beyond a retention score and incorporates its capacity controller into page allocation and asynchronous compaction. It also considers preemption and request recovery.

  2. Informative controls for the demand signal. The future-demand correlation and counterfactual branches support the proposed signal. The random and inverse policies further indicate that the timing of capacity decisions matters.

Critical weaknesses
  1. The novelty claim omits a closely related adaptive-budget method. GVote (arXiv:2509.03136) also predicts future-query demand to avoid a manually fixed KV budget. I regard this as very close to the motivation of GrowPage, yet it is neither discussed nor compared. Without a clear distinction and controlled comparison, I am not convinced by the claim that existing methods keep total per-request capacity fixed.

  2. I could not reconcile the serving results. My calculation from Tables 1, 5, and 7 gives an average DeepSeek-R1/AMC23 output of about 6,848 tokens. Under the stated single-A100 setting, roughly 54 GiB remains for KV storage; at 128 KiB per token, FullKV can keep only about 60 such requests resident. The reported aggregate throughput would therefore require about 51 ms/token per resident request, but Table 7 reports 1,027 ms/token. I may be missing an undocumented scheduling convention, but the paper does not provide enough information to close this roughly 20x gap. The overhead analysis is also incomplete because preempted requests require re-prefill, while Figure 8 reports only selected per-event costs. I therefore do not find the headline throughput gains verifiable from the submitted results.

Questions for authors
  1. What initial and maximum per-request capacities were used for GrowPage? How does the comparison with Zipage change when average KV use and resident concurrency are matched?

  2. How is TPOT defined, and what time intervals does it include? Could the authors explain how the DeepSeek-R1/AMC23 FullKV throughput is consistent with the reported TPOT and memory budget?

  3. Does the reported overhead include the cumulative cost of preemption and re-prefill? How do these operations affect end-to-end latency under concurrent serving?

  4. How does GrowPage differ technically from GVote? A comparison under the same runtime and resource constraints would materially affect the novelty assessment.

4 comments
AuthorsOpenReview ↗

Response to K1Za (1/2)—TPOT and Preemption

Thank you for raising these questions about the serving measurements.

Q1. TPOT, throughput, and KV residency

FullKV’s high TPOT is consistent with its throughput and GPU KV usage because the request-averaged metric includes time spent waiting after the first token, even after preempted requests have released their KV pages.

For request ii, DiD_i is gen_time in seconds from the first to the last output token, and NiN_i is output_len. This interval includes post-first-token waiting and re-prefill. Initial queueing and prefill precede it. Table 7 averages 1000Di/(Ni−1)1000D_i/(N_i-1) across requests.

We could not locate the original FullKV (nano-vLLM) JSON log, so we illustrate the TPOT calculation using the FullKV (vLLM) JSON log for DeepSeek-R1-Distill-Llama-8B on AMC23. Across 1280 requests, we obtain:

= \frac{1272976.3484}{1280} \approx 994.5128\\,\mathrm{ms/token}.$$ This is within 1 ms/token of the **994 ms/token** reported in Table 7 for the same FullKV (vLLM) setting. | Metric | FullKV (vLLM) | | --- | ---: | | Total output tokens | 8897584 | | Workload elapsed time, $T$ | 7199.52 s | | Throughput | 1235.8568 tok/s | | Token-interval-weighted TPOT, $1000\sum_i D_i/\sum_i(N_i-1)$ | 496.92 ms/token | | Mean requests between first and last token, $\sum_i D_i/T$ | 614.03 | | Mean running / waiting requests from scheduler logs | 72.28 / 543.22 | **Effect of waiting on TPOT.** If a preempted request waits $W_i$ seconds for readmission after emitting its first token, those seconds increase $D_i$ without generating new tokens, adding $1000W_i/(N_i-1)$ ms/token to its TPOT. Re-prefill further extends $D_i$ as the KV cache is rebuilt. Other requests can reuse the released pages and keep contributing to aggregate throughput. In this FullKV (vLLM) run, multiplying the post-first-token output rate of 1235.68 intervals/s by the interval-weighted TPOT of 0.49692 s/token gives an average of 614.03 requests between their first and last tokens. Using the sampled running count of 72.28 as a proxy, dividing that count and the gap of 541.75 by the output interval rate gives about **58.5 ms of running request-time** and **438.4 ms of non-running request-time** per output interval, respectively. The running-time estimate is on the same scale as the reviewer's 51 ms/token. The gap in request counts is close to the sampled waiting count of 543.22, suggesting substantial post-first-token queueing. This split is approximate because the scheduler counts can also include requests before their first token. For a pause of the same duration, shorter outputs incur a larger TPOT increase. Request averaging gives these outputs more relative weight than interval weighting, helping explain the difference between 994.51 and 496.92 ms/token in this run. **Q2. Preemption, re-prefill, and end-to-end latency** Fig. 8 profiles selected per-event capacity-control and compaction operations. **Our reported throughput uses the full workload elapsed time, including preemption and re-prefill.** We measured the following in a separate GrowPage rerun using Qwen3-8B on AMC23. | Metric | Value | | --- | ---: | | Total requests | 1280 | | End-to-end elapsed time | 5150.60 s | | End-to-end throughput | 2244.45 tok/s | | Preemption events | 1406 | | Cumulative host-side preemption time | 0.062842 s | | Cumulative re-prefill model-call wall time | 136.59 s | | Re-prefill model-call wall time / workload wall time | 2.65% | | Requests preempted at least once | 294 (22.97%) | | Mean preemptions per affected request | 4.78 | | Mean end-to-end latency difference (preempted vs. never preempted) | +56.8% | The +56.8% is an unadjusted difference between the two request groups. Preempted requests generally generated longer outputs, which also contribute to the observed latency gap.
AuthorsOpenReview ↗

Response to K1Za (2/2)—KV Capacity, Zipage, and GVote

Q3. KV capacity settings and comparison with Zipage

We selected GrowPage's initial per-request KV capacities with reference to budgets evaluated in prior work, including Zipage and R-KV.

WorkloadInitial KV capacity (tokens)
MATH5002048
AMC232048
AIME244096
LiveCodeBench v14096
GSM8K1024
Mixed workload (Fig. 5)1024

The implementation supports an optional per-request capacity cap. We left it unset in the reported experiments, so growth is limited by available pages in the shared GPU KV pool. We will clarify these settings in the revised paper.

We added a 3072-token Zipage point to the Qwen3-8B mixed-workload experiment in Fig. 5 to compare the methods at similar mean resident concurrency.

MethodPer-request KV budget (tokens)Mean KV-resident requestsPass@1 (%)TPS (tok/s)
Zipage-30723072 (fixed)12781.42191
GrowPage3442 (mean)12184.92213

GrowPage improves Pass@1 by 3.5 percentage points at comparable throughput (+1.0%). In Fig. 5, it also approaches Zipage-8192's Pass@1 (84.9% vs. 85.1%) while delivering about 1.9× the throughput.

Q4. Technical distinction and nano-vLLM comparison with GVote

Thank you for pointing out GVote (Tang et al., arXiv:2509.03136). We overlooked this closely related work and will cite it and compare the two approaches in the revised manuscript.

Novelty. GrowPage’s core contribution is to treat KV cache capacity as a resource shared by concurrent requests and adjust each request’s page allocation online during decoding.

  1. GVote compresses prompt KV, while GrowPage compresses accumulated KV history. GrowPage also compresses KV from generated tokens, which can account for most of the cache when reasoning outputs far exceed their prompts.
  2. GVote sets prompt KV retention at prefill, while GrowPage decides whether to grow during decoding. Observation 2 shows that attention demand can shift substantially within a request, making it difficult to size a budget for the entire decode from prefill information alone. When a request fills its current allocation, GrowPage uses observed decoding queries to decide whether to request another page or compact within its current capacity.
  3. GVote requires adaptations for standard prefix caching and PagedAttention, while GrowPage uses both.
    • Prefix caching. GVote selects KV entries based on the full prompt. Requests sharing a token prefix can therefore produce different compacted pages, making those pages difficult to reuse through standard prefix matching.
    • PagedAttention. GVote can retain a different number of tokens per KV head, while Nano-vLLM allocates physical pages shared by all heads of a request. Reclaiming physical pages through per-head compression requires adapting the page layout and allocator.

Same-runtime comparison on AMC23. To address the reviewer’s request, we reimplemented GVote from the paper in Nano-vLLM because a public reference implementation was unavailable. We refer to this version as GVote-Prefill (paged adaptation). Gvote_prefill JSON log

  • Prefix caching. We disabled prefix caching for the GVote adaptation because the compacted pages depend on the full prompt.
  • Paged KV layout. For each request, we kept GVote’s selections for each KV head in each layer and used the largest retained count across layers and heads as the common cache length. We filled shorter selections with additional valid KV entries ranked by the actual prefill query’s attention scores, packed the results into whole pages, and released the surplus pages.
  • Position handling. We preserved the original RoPE positions and continued advancing logical positions during decoding, while using the compacted physical length for cache addressing.
Method (Qwen3-8B, AMC23)Pass@1 (%) ↑Throughput (tokens/s) ↑Elapsed time (h) ↓
GVote-Prefill (paged adaptation)91.611622.71
GrowPage91.422611.42

GrowPage achieves 1.95× the throughput and about 48% less elapsed time, with a Pass@1 difference of only 0.2 percentage points. On AMC23, prompts are short relative to the generated reasoning traces, so prompt compression can reduce only a limited share of total KV usage.

The published GVote method uses a head-wise variable-length layout. In our paged adaptation, the additional KV entries used to equalize head lengths participate in attention, so the adaptation may affect both accuracy and serving efficiency.

We believe that algorithmic contribution, compatibility with existing inference engines, and measured end-to-end serving gains are all important criteria for evaluating KV cache compression.

Thank you for the detailed responses and additional experiments. The TPOT definition and inclusion of preemption costs address my main concerns about the serving measurements, and the technical distinction from GVote is clearer. However, the Zipage comparison does not fully match average KV usage, and the GVote adaptation limits attribution of the throughput gains. Overall, the rebuttal provides sufficient clarification and additional evidence for me to raise my score to 3.

Thank you for your careful reassessment and for the time and effort you have devoted to reviewing our paper. Your constructive feedback helped us address important weaknesses and improve the clarity and rigor of the manuscript.

We appreciate your understanding and your recognition of the clarifications and additional evidence provided during the rebuttal.

Reviewer_TpYBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies dynamic KV cache budgeting for efficient LLM reasoning serving. Unlike conventional KV cache compression methods that operate under a fixed budget, GrowPage adjusts each request's KV capacity during decoding as its attention demand evolves. The authors propose an online demand estimator based on dual-timescale query summaries to decide when more capacity is needed, and integrate the resulting grow-only allocation policy with page-level KV memory management in a practical serving system. Experiments on several reasoning models and benchmarks show favorable accuracy–throughput trade-offs relative to fixed-budget compression methods.

Critical strengths
  1. A meaningful and relatively underexplored problem formulation. The paper targets a question that existing KV compression work largely sidesteps: not which tokens to keep under a fixed budget, but how much KV capacity each request should receive as reasoning proceeds. This is well aligned with modern serving systems, where page-level KV management naturally supports flexible allocation and heterogeneous demands. The formulation offers a complementary direction to existing compression research and a clear opportunity for algorithm–system co-design.

  2. A practical approach with promising serving benefits. GrowPage uses a lightweight online demand estimator and integrates dynamic KV allocation into a PagedAttention-based serving system. Experiments show favorable accuracy–throughput trade-offs across multiple reasoning workloads, and the additional analyses of demand estimation and alternative capacity-control policies lend useful support to the design.

Critical weaknesses
  1. The initial KV capacity is underspecified and its sensitivity is not analyzed. Because the allocation policy is grow-only, the initial budget directly affects both memory occupancy and how often expansion is triggered. The paper does not clearly state how this value is set, nor does it study sensitivity across initial allocations. This limits assessment of the method's robustness and reproducibility.

  2. The benefit of intra-request dynamic budgeting is not isolated. The paper motivates GrowPage through both cross-request heterogeneity and within-request demand variation. While the existing ablations support the effectiveness of demand-guided capacity control, they do not clearly distinguish the benefits of online intra-request adaptation from those of assigning different capacity budgets across requests. A request-level adaptive fixed-budget baseline would help clarify the additional value of adjusting KV capacity during decoding.

Questions for authors
  1. Please specify how the initial KV capacity is determined, and report sensitivity results across a range of initial capacities (reasoning accuracy, average KV memory usage, serving throughput).

  2. Could the authors compare GrowPage against a request-level adaptive fixed-budget baseline, potentially using an oracle per-request budget as a diagnostic reference, to better isolate the benefits of intra-request dynamic capacity adjustment?

4 comments
AuthorsOpenReview ↗

Response to TpYB (1/2)—Initial KV Capacity and Sensitivity

Q1. Initial KV Capacity and Sensitivity

We selected the initial per-request KV capacities with reference to budgets evaluated in prior work, including Zipage and R-KV.

WorkloadInitial KV capacity (tokens)
MATH5002048
AMC232048
AIME244096
LiveCodeBench v14096
GSM8K1024
Mixed workload (Fig. 5)1024

We left the optional per-request capacity cap unset in the reported experiments, so subsequent growth is constrained by available pages in the shared GPU KV pool. We will clarify these settings in the revision.

We also added an initial-capacity sensitivity study on Qwen3-8B and AMC23.

Initial capacity (tokens)Avg. KV budget (tokens)Total time (s)TPS (tok/s)TPOT (ms/tok)Pass@1 (%)Mean running seqs
51228285028232810789.9120
102429165035235712490.8118
204831645102226112291.4113

Initial capacities of 1024 and 2048 achieve similar accuracy and throughput on this workload. Relative to 2048, starting at 1024 reduces the average KV budget by 7.8% and increases throughput by 4.2%, with a 0.6 percentage-point decrease in Pass@1. Reducing the initial capacity further to 512 lowers Pass@1 by another 0.9 percentage points and yields slightly lower throughput than 1024, despite its smaller average KV budget and higher mean running concurrency.

A plausible explanation is that a smaller initial capacity reaches decision boundaries earlier. Compression at these early boundaries may discard context needed later, affecting subsequent reasoning and generation length. Throughput also depends on decoding, compression, and scheduling costs, so a smaller initial capacity does not necessarily yield higher TPS.

AuthorsOpenReview ↗

Response to TpYB (2/2)—Intra-request Dynamic Budgeting

Q2. Intra-request Adaptation versus Oracle Fixed Budgets

We added a question-specific fixed-budget reference on Qwen3-8B/AMC23. It uses GrowPage's token scoring, selection, and scheduler. A budget is selected offline for each question and held fixed throughout decoding for all 32 generations of that question.

1. Budget construction and accuracy

Following the eight-generation calibration protocol used for Fig. 2, we selected empirical budgets from {512, 1024, 2048, 4096, 8192} tokens.

Fixed budget (tokens)QuestionsGeneration requests
5126192
102412384
20488256
40967224
81927224
Total401280

The mean assigned budget is 2944 tokens. Here, "oracle" denotes a hindsight-informed, empirical assignment. Sufficiency is evaluated on eight calibration generations per question–budget pair, while the serving evaluation uses 32 generations per question. Additional trajectories can require larger budgets, even with the same seed configuration. Thus, calibration success does not guarantee agreement with FullKV's accuracy over all 32 draws.

Calibration using 32 generations could raise some selected budgets. We retained the eight-generation assignment to preserve the calibration setting of Fig. 2 and evaluated its accuracy separately in the 32-generation serving run.

2. Serving results and analysis

MethodMean KV budget (tokens)Total time (s)TPS (tok/s)TPOT (ms/tok)Pass@1 (%)Mean running seqs
GrowPage31645102226112291.4113
Oracle fixed budget29445625214114591.8111

The oracle reference achieves a smaller mean budget and 0.4 percentage points higher Pass@1. GrowPage achieves 5.6% higher TPS, 9.3% shorter total time, and 15.9% lower TPOT, using a 7.5% larger mean budget.

From the rounded TPS and elapsed time, the oracle run generated approximately 12.04M output tokens, versus 11.54M for GrowPage, about 4.4% more. Its longer runtime therefore reflects both greater output volume and lower aggregate throughput. We further interpret these results through two possible mechanisms.

  • Very small fixed budgets. Six questions, or 192 generation requests, receive a 512-token budget. Early compression risks removing context needed later, as discussed in Q1, and this small capacity limit persists throughout decoding. Repeated eviction can lead to regenerating intermediate reasoning or revisiting previous steps, lengthening the output even when the final answer is correct. Selecting a small budget based on answer correctness can therefore still yield an expensive generation.

  • Large fixed budgets. Seven questions, or 224 generation requests across the workload (17.5%), receive an 8192-token budget. Once these requests accumulate long caches, they can increase KV access costs and pressure on the shared page pool. This can increase scheduler waiting and preemption, affecting other requests as well. The oracle's 2944-token mean is an arithmetic average of assigned budgets. Actual physical occupancy also depends on generation progress and residence time, so this mean can conceal a costly high-budget subset.

3. Advantages of adapting capacity during decoding

First, selecting an appropriate fixed budget before generation requires predicting the memory needs of a future reasoning trajectory. Our oracle uses repeated offline experiments and observed correctness to make this assignment. A practical serving system must make its initial allocation with much less information. The difference between calibration with eight generations and evaluation with 32 also illustrates the uncertainty of extrapolating a fixed budget from limited sampled trajectories.

Second, even an informed fixed assignment can retain the efficiency risks above. A small cap can constrain later reasoning, while a large cap permits expensive caches to accumulate. GrowPage revisits the decision at capacity boundaries, choosing between page growth and compression according to the observed demand trend. It can respond when demand broadens and maintain capacity when demand concentrates, reducing the risk of imposing one unsuitable capacity limit throughout a generation.

Observation 2 in the paper provides the complementary motivation that KV demand changes across stages within the same request. Together with this diagnostic comparison, it supports using adaptation during decoding as the main capacity control mechanism, with the initial budget serving as a starting point that can be adjusted as reasoning progresses.

Thank you for the clarifications! The rebuttal addresses my concerns well, and I will keep my original score, as it already reflects my positive assessment.

Thank you again for your positive assessment of our work. We are encouraged by your feedback and appreciate the time and care you devoted to reviewing our paper.

Reviewer_5dDDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GrowPage, a dynamic KV-cache management approach for efficient LLM reasoning serving. Its main observation is that KV-cache requirements vary across requests and may also change during the generation of a single request; therefore, allocating the same fixed KV-cache capacity to every request is inefficient.

GrowPage compares recent attention patterns with longer-term attention patterns to estimate whether a request is beginning to rely on a broader range of historical context. It then uses this signal to decide whether to allocate an additional KV-cache page. If no clear growth in demand is detected, the method compresses the existing cache and keeps the current capacity unchanged. GrowPage is integrated with PagedAttention, continuous batching, and prefix caching. Experiments on two 8B models show that it can improve serving throughput while retaining accuracy close to that of a full KV cache.

Overall, the problem addressed by the paper is not new. Its main contribution is not the observation that KV-cache demand is heterogeneous, but rather a lightweight online mechanism that connects changes in attention patterns to request-level page allocation, together with a reasonably complete serving-system implementation.

Critical strengths
  1. Practical system design. GrowPage treats the KV-cache capacity of each request as a runtime variable and expands it at page granularity, which allows the method to fit naturally into existing PagedAttention-based serving systems.

  2. Lightweight and interpretable decision signal. By comparing short-term and long-term attention patterns, the method attempts to detect growth in the active working set without requiring an additional trained predictor. The reported runtime overhead is also small.

  3. Competitive empirical results & Relatively complete implementation. GrowPage substantially improves throughput while maintaining relatively high task accuracy. The correlation analysis, random-policy control, and ablation studies provide useful evidence that the proposed allocation signal is meaningful. The paper considers practical serving components such as continuous batching, prefix caching, asynchronous compaction, and CUDA Graphs, rather than evaluating the method only through algorithm-level simulation.

Critical weaknesses
  1. Limited novelty of the problem formulation and several claimed contributions. Request-level variation in KV-cache demand, temporal changes in attention, and attention-based token retention have already been extensively studied. Relative to prior work, the main incremental idea appears to be using changes in attention patterns to decide whether to allocate one additional page, which makes the scope of the algorithmic novelty relatively narrow.

  2. Insufficient comparison with strong dynamic baselines. The paper mainly compares GrowPage with static budgets, existing compression methods, and a random growth policy. It does not sufficiently evaluate simpler online heuristics based on generation length, compression loss, or the attention mass of evicted tokens. As a result, it remains unclear whether the proposed dual-timescale signal is necessary.

  3. Limited experimental coverage. The evaluation focuses on two dense 8B models and a single-GPU setting. Results on larger models, mixture-of-experts models, multi-GPU deployments, and realistic online workloads with tail-latency measurements are missing.

  4. The source of the system-level gains is not fully isolated. Some throughput improvements may result from both the dynamic budgeting algorithm and its system integration. More controlled comparisons under equal memory, equal concurrency, or equal accuracy would help clarify the contribution of the allocation policy itself.

Questions for authors
  1. Beyond using short-term and long-term attention changes to trigger page growth, what is the fundamental algorithmic novelty of GrowPage relative to prior attention-based KV-cache compression methods such as Zipage?

  2. Have the authors compared GrowPage with simpler dynamic policies based on generation length, compression loss, or the attention mass assigned to evicted tokens? How much benefit comes specifically from the dual-timescale attention signal?

  3. Does GrowPage retain its accuracy and throughput advantages on larger models, multi-GPU deployments, and realistic online workloads?

2 comments
AuthorsOpenReview ↗

Response to 5dDD (1/2)—Algorithmic Novelty and Dynamic Policy Comparisons

Q1. Algorithmic Novelty beyond Zipage

GrowPage’s algorithmic novelty spans dynamic KV capacity control and the complementary token scoring and selection mechanisms used during compression. The same short- and long-term query summaries guide both capacity decisions and token scoring, while local quotas followed by global selection balance historical coverage with token importance.

  1. Differences in request-level KV capacity decisions. Zipage compresses KV within a preset per-request capacity limit. When a request fills its current allocation, GrowPage compares the fractions of historical KV entries that the short- and long-term summaries need to cover the same target attention mass. If the short-term fraction exceeds the long-term fraction by more than the growth threshold and a page is available, GrowPage adds one page. Otherwise, it compacts the cache to free one page’s worth of slots within the current allocation.

  2. Differences in token scoring. Zipage’s window-based scorer uses the latest 16 queries in its reported setup, and R-KV also computes its attention-importance scores from a recent query window. GrowPage reuses the two exponential moving average (EMA) query summaries from capacity control. At each layer and KV head, we take the larger of the two attention scores for each candidate KV entry. This lets an entry rank highly on the strength of its long-term score even when it receives little recent attention. G-KV and Zipage’s global-scoring variant store historical scores for individual KV entries. GrowPage represents query history with fixed-size summaries whose storage is independent of sequence length.

  3. Differences in token selection and retention. Zipage selects the highest-scoring entries across the sequence using global top-k. GrowPage first retains a quota of the highest-scoring entries in each historical block, then fills the remaining budget with the highest-scoring unselected entries across all blocks. Allocating the local quota first keeps some KV entries even in blocks that score poorly at the current boundary. This preserves context that later queries can revisit if their attention shifts. In Table 4 (Appendix D.2), increasing the local quota from 16 to 64 improves Qwen3-8B Pass@1 by 1.1 percentage points on AMC23 and 1.4 percentage points on AIME24.

Q2. Comparison with Simpler Dynamic Policies

We evaluated three alternative policies on Qwen3-8B and AMC23. The variants share GrowPage’s KV token scoring, selection, and scheduler. The only change is the rule that decides whether to add one page or compact within the current allocation at a capacity boundary.

  • Length-only (L = 1024). The target KV budget increases by one page for every 1024 generated tokens. At a capacity boundary, the policy requests one additional page if the current budget is below this target.
  • Evicted Attention Mass (EAM). The shared selection rule identifies the KV entries that would be evicted. Using the latest decode query, EAM sums the pre-eviction attention weights assigned to those entries and requests growth when this mass exceeds a threshold.
  • Attention-Output Distortion (AOD). Using the same query and proposed eviction set, AOD computes the attention output before eviction (o) and after eviction with renormalized attention weights (o'). It uses ||o - o'||_2 / (||o||_2 + ||o'||_2 + eps) as a proxy for compression loss and requests growth when this distortion exceeds a threshold.

EAM and AOD average their scores across layers and query heads before applying their respective calibrated thresholds. The initial EAM threshold of 0.000378 triggered almost no page growth in the full AMC23 inference run. After several rounds of threshold adjustment, EAM’s growth behavior still differed substantially from that of the other policies. This calibration difficulty makes EAM less practical as a dynamic KV controller in our setting, since switching workloads may require further threshold tuning. GrowPage uses a single threshold across all datasets in our main experiments. Given the tuning effort already required on AMC23, we ended the EAM threshold sweep after these attempts and report only its decision cost in the table.

PolicyAvg. KV budget (tokens)Total time (s)Throughput (tok/s)TPOT (ms/tok)Pass@1 (%)Mean running seqsAvg. decision cost (ms/event)
GrowPage31645102226112291.41135.84
Length-only32925598201213388.51060.07
EAM——————21.79
AOD34665842192814590.310135.94

The final column reports the mean time per capacity-boundary decision. With the same compaction and scheduling mechanisms, GrowPage improves Pass@1 by 2.9 and 1.1 percentage points and throughput by 12.4% and 17.3% over Length-only and AOD, respectively, while using a smaller average KV budget in both comparisons.

AuthorsOpenReview ↗

Response to 5dDD (2/2)—Model Scaling and Deployment Coverage

Q3. Model Scaling and Deployment Coverage

We added experiments on Qwen3-14B, where GrowPage achieves higher accuracy and throughput than Zipage with a smaller average KV budget. We evaluated this larger dense model on AMC23 using a single A100 80GB GPU.

MethodAvg. KV budget (tokens)Total time (h)Throughput (tok/s)TPOT (ms/tok)Pass@1 (%)Mean running seqs
FullKV (nano-vLLM)—3.6568665095.945
Zipage40962.06124316292.868
GrowPage35371.76147013593.779

Compared with Zipage, GrowPage uses a 13.6% smaller average KV budget, improves throughput by 18.3% and Pass@1 by 0.9 percentage points, and reduces TPOT by 16.7%. Compared with FullKV, GrowPage achieves 2.14× throughput, with Pass@1 decreasing from 95.9% to 93.7%. These results extend our evidence from 8B models to a larger 14B model in the same family.

For deployment coverage, our available compute is limited to a single A100 80GB GPU, which constrains our current evaluation to single-GPU serving. Access to production request traces, including realistic arrival timestamps, is also limited. We will explicitly clarify the scope of our single-GPU benchmark and mixed-workload experiments in the revision, and identify multi-GPU deployment and evaluation on production online workloads, including tail latency, as future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.