acceptodds
Under review as a conference paper at ICLR 2027

RACE: Calibrated KV-Cache Reservations from Prefill States

Abstract

LLM serving systems make request-admission decisions before the decoding demands of incoming requests are fully known. In prefill–decode disaggregated systems, forecasting output length can help decoding workers plan key–value (KV) cache reservations and determine how many requests to admit. The challenge is that repeated generations from the same prompt can differ in length, making reservations based on a single predicted length vulnerable to unexpectedly long responses. We introduce Rollout-Aware Calibrated Endpoints (RACE), which predicts a token budget by accounting for both typical response length and response-length variability. RACE learns these quantities with two predictors trained on prompt-processing hidden states and multiple sampled responses per training prompt. It assigns larger safety margins to prompts with more variable response lengths, then calibrates these margins on an independent prompt set. On Qwen3-8B and DeepScaleR, RACE reduces the average token budget by 9.94% compared with GBM-Quantile-PAC, a calibrated quantile-regression baseline using the same features. Its budgets accommodate 97.56% of test outputs under the configured output-length limit. These smaller reservations reduce the capacity set aside per request, leaving more room for concurrent work under reservation-based scheduling.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM serving, KV-cache, output-length prediction

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_oWQVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RACE predicts output-token reservations from prefill hidden-state features. Two regressors estimate typical response length and within-prompt variation using repeated rollouts. A separate calibration set chooses an order-statistic multiplier for an upper endpoint, targeting population coverage with a specified confidence level. Experiments measure reservation size and coverage across four model/task settings, ablate request-specific margin assignment, and replay a reservation-based scheduler on a fixed request trace.

Critical strengths
  1. The statistical target and calibration protocol are explicit. Appendix C distinguishes population coverage from confidence over calibration samples, accounts for capped observations conservatively, handles tied/discrete scores, and specifies a full-budget fallback. Using one prespecified response per independent calibration prompt avoids treating correlated within-prompt samples as independent calibration units.
  2. The experiments meaningfully probe the margin predictor. The primary comparison uses matched prefill features and paired prompt-level bootstrap intervals. The fixed-length-predictor ablation separately recalibrates constant, length-proportional, shuffled, and learned margins and varies rollout count. This gives useful evidence that assigning variation to the correct request helps, beyond simply changing the global margin size (Sections 5–6; Figures 3–4).
Critical weaknesses
  1. The simulator does not enforce the physical resource constraint whose benefit is being claimed. Section 4 and Appendix E allow requests to continue when actual KV demand exceeds capacity; overruns are recorded but incur no eviction, recomputation, offload, or admission-recovery cost. Each active request also advances one token per step regardless of batch or context. At the 40K setting, Table 17 reports a 9,469-token increase in peak demand versus GBM-PAC, despite a small increase in excursion-time fraction. Short excursions can still make a physical schedule infeasible. The simulation is transparently described, but does not yet establish that the reported throughput/JCT gains survive a hard-cap serving implementation (pp. 5, 26–27).
  2. The strongest-baseline and equal-coverage evidence is mixed. Table 1's native-feature Mixture-PAC reserves 21,552 tokens at 97.85% coverage in the primary setting, versus RACE's 21,721 at 97.56%; this comparator has richer features, so it does not invalidate the feature-matched gain, but it limits claims of best practical performance. Table 17 likewise reports RACE at −1.45% throughput and +2.01% mean JCT relative to Mixture-PAC. Appendix Table 4 also marks coverage noninferiority as failed for ShareGPT and GSM8K. Meeting the nominal 95% target on the observed sample is distinct from establishing comparable coverage to GBM-PAC. These disclosed results deserve greater emphasis when interpreting the 9.94–22.26% savings (pp. 7, 18, 26).

Others:

  1. The added value of repeated-rollout scale supervision over a simple length-proportional margin is small and is not tested at matched coverage.
  2. The rule μ(S) + q·s(S), with a learned scale and a held-out order statistic, is the normalized (locally weighted) split-conformal construction (e.g., Papadopoulos et al., 2008; Lei et al., 2018; neither is cited) combined with a training-conditional/PAC rank (Vovk, 2012; Park et al., 2019). Appendix C is the corresponding standard order-statistic argument. The novelty therefore rests on supervising s with within-prompt spread from repeated rollouts.
  3. Variability from training and calibration is not quantified.

Minors:

  • L. 278: ShareGPT is cited as Kong et al. (2026), a systems paper (LLM-CoOpt). Please cite the dataset source instead; App. B.3 refers to the "Aeala V4.3 cleaned split".
  • Sec. 4 and App. E do not say which setting's 512 requests form the scheduling trace. The magnitudes suggest Qwen3-8B/DeepScaleR, but ShareGPT-fresh also has 512 test prompts. They also do not say which of the two recorded responses per prompt is used.
  • L. 24 (abstract): the 97.56% coverage is reported without the 95% target and 95% confidence that it should be read against.
  • L. 102–103: "how many output tokens a serving system should reserve KV-cache capacity for before decoding begins" is awkward. Suggested: "the number of output tokens for which a serving system should reserve KV-cache capacity before decoding begins".
  • L. 197: "The two LightGBM regressors" is the first mention of LightGBM. Please introduce and cite it (Ke et al., 2017), and likewise XGBoost (Chen & Guestrin, 2016) for AFT-PAC (L. 907).
  • L. 223: "the ordered scores increase with k" should read "are nondecreasing in k". In Eq. 7, the lower clip at 0 is redundant, since μ ≥ 0, q ≥ 0, and s ≥ 32. The notation U_α also hides its dependence on δ.
  • Notation is overloaded: S is both the prefill state and part of S_max (L. 245); A is both the active set (L. 245) and the scores A_i (App. C); L is both output length and the depth set (App. B.1); G is both the projection matrix (Eq. 10) and G_i(t) (L. 261); k is both the calibration rank and, on the x-axes of Figs. 6–7, the number of responses per prompt (Tables 6–8 use n_resp); Q_p denotes quantiles while Q1–Q4 denote quartiles.
  • The same margin rules have different names in different places: Shuffled / Length-proportional / Constant (Fig. 4, Tables 6–8); permuted / center proportional / constant (Figs. 6–7); Permuted learned scale / Center proportional / Constant / additive PAC (Table 9b). Colors and markers for the same rule also differ between Fig. 4 and Figs. 6–7. Fig. 7 says "scale-label budget", while the text says "label budget".
  • Native Mixture-PAC's extra inputs are called "vocabulary-level statistics" (L. 286) in one place and "richer boundary statistics" (L. 932–933) in another; please define them once. Table 9a's "Boundary scalars" (40 dims) and "RACE (full feature set)" (1,064 dims) are never defined.
  • Fig. 3b reports ρ = 0.477 between s(S) and length-prediction error, and Table 9b reports 0.398 for what reads as the same quantity. Please state the difference: a per-prompt mean over 496 pairs vs. 1,005 individual responses.
  • Several different RACE numbers with three responses per prompt appear on different cohorts and test sets: 21,721 (Tables 1, 5, 9), 21,542 (Table 6), and 21,229 (Tables 7, 15; Fig. 2). This is why the saving over GBM-PAC is 9.94% in one place and 11.58% in another. Captions should state which predictor and test set they use.
Questions for authors
  1. What reservation/throughput/JCT benefits remain when KV capacity is enforced, with a specified overrun policy and its cost? Please include prefill-feature extraction/prediction overhead or a measured bound on it.
  2. Can you compare RACE and Mixture-PAC with the same richer feature set, and show matched observed-coverage trade-offs or a prespecified noninferiority comparison in each setting? How should the failed noninferiority results in Table 4 qualify the cross-setting conclusion?
  3. How sensitive are coverage and reservation efficiency to deployment-time prompt or sampling-policy drift, and what is the recalibration procedure? This question concerns deployment robustness; it does not allege an error in the iid theorem.
  4. Can you compare margin rules at matched observed coverage? For example, plot budget–coverage curves like Fig. 2c for the learned, length-proportional, and constant margins. Please also include a scale model trained on absolute residuals |y − μ(S)| from single responses at the same total generation budget. It would also help to report Spearman(s(S), μ(S)) and the partial correlation between s(S) and |L0 − L1| given μ(S). If the advantage over these simpler rules at matched coverage is only 1–2%, the case for repeated-rollout supervision weakens considerably.
  5. How stable are the main comparisons across random fit/validation/calibration splits and training seeds? Please report the mean ± s.d. of savings and coverage against GBM-PAC, Mixture-PAC, and length-proportional margins. Relatedly, with two or three responses the range target is 1/0.6 times the 80–20 spread target (App. B.2), up to the floor and the log(1 + ·) offset, and calibration should absorb a constant factor. Why, then, does the "range scale" variant shift the budget by 1.8% and coverage by 0.8 pp (Table 5)? Unless I am missing an implementation detail, this points to run-to-run variance comparable to the reported gaps.
5 comments
AuthorsOpenReview ↗

Response to Reviewer oWQV

Thank you for pointing out the calibration protocol and margin-allocation experiments. Your questions led us to run more experiments on serving, matched features, workload shift, and stability. We answer them below.

Q1. Do the gains persist with physical KV limits and real serving costs?

We tested RACE in a PD-disaggregated vLLM service with three RTX A6000 GPUs, each with 48 GB of memory. The setup had one prefill instance and two decode instances, using NixlConnector for KV transfer. Each instance ran Qwen3-8B in BF16 with tensor parallelism set to one. Each decoder had 65,536 usable KV token slots. At the DeepScaleR high-load point:

MethodThroughput (tokens/s) ↑Mean JCT (s) ↓p95 JCT (s) ↓
Tuned native vLLM1733,2167,904
GBM-PAC1922,9677,361
Mixture-PAC2072,7517,038
RACE2192,4696,382

RACE boosts throughput by 14.06% and lowers mean and p95 JCT by 16.78% and 13.30% compared to GBM-PAC. Against Mixture-PAC, it increases throughput by 5.80% and reduces mean and p95 JCT by 10.25% and 9.32%. These results show RACE’s scheduling benefits in the tested PD service.

On a separate RTX 3090, we measured mean added prefill times of 1.832, 2.318, and 2.974 ms for batch sizes 1, 4, and 8. These measurements include feature extraction and budget prediction during the prefill window, which ends when sampling starts.

We also separate budget overruns from physical allocation conflicts. In a hard-cap replay, physical conflicts are managed by preemption and recomputation. The 95/95 RACE runs had no preemptions. In a 90% target sensitivity test, one out of 32 requests missed its budget, but no recovery events happened. So, an underestimated request can continue as long as physical pages are available, and budget misses are different from recovery events.

Q2. How does RACE compare with Mixture-PAC using the same richer features?

Thank you for suggesting this control. We retrained RACE using the same 1,064-dimensional input as Mixture-PAC on Qwen3-8B/DeepScaleR. Both use a 95% target coverage and 95% calibration confidence. The main and extra evaluations have 512 prompts with two responses each and 500 prompts with four responses each.

Test setMethodMean reserved tokens ↓Coverage (%)
PrimaryRACE, rich features21,955.2198.05
PrimaryMixture-PAC, rich features21,558.7297.85
AdditionalRACE, rich features21,407.5596.70
AdditionalMixture-PAC, rich features21,488.2796.40

RACE gets higher observed coverage on both sets, with a bigger primary budget and a smaller extra budget. The extra budget-saving interval includes zero. These paired results show the trade-off between reservation and coverage, and the PD comparison highlights RACE’s practical serving advantage over Mixture-PAC.

In the four-setting comparison, the coverage-noninferiority criterion is met in both DeepScaleR settings. The ShareGPT and GSM8K coverage-difference intervals have lower endpoints of −2.25 and −1.44 percentage points, which are below the −1-point threshold. To sum up: reservations are smaller in all four settings, observed coverage stays above 95%, and coverage noninferiority is shown in two settings.

Q3. How sensitive are coverage and reservation efficiency to deployment-time prompt or sampling-policy drift, and what is the recalibration procedure?

We tested prompt-distribution drift from DeepScaleR to ShareGPT, keeping both predictors fixed and comparing the original calibration to recalibration in the target domain.

For recalibration, we collect one response from each of 277 target conversation families. Using their predicted and observed response lengths, we adjust the reservation margins at the same 95% coverage target and 95% calibration confidence. We evaluate using 502 separate conversation families.

CalibrationMean reserved output tokens ↓Coverage (%)
Original DeepScaleR calibration16,758.4199.30
Target-domain recalibration7,347.8397.71

The original calibration is conservative for the target workload. Recalibration cuts the mean reservation by 56.15%, while observed coverage stays above 95%. These results show that RACE can lower excess reservation after a workload shift by recalibrating its safety margins, without retraining either predictor.

AuthorsOpenReview ↗

Response to Reviewer oWQV-2

Q4. How do learned margins compare at matched observed coverage and generation cost?

Thank you for suggesting these controls. We added comparisons for reservation efficiency, supervision cost, and information beyond predicted response length.

Reservation budgets at or above 95% observed coverage. We keep the length predictor fixed and check all 277 calibration ranks for each margin rule. The table shows the smallest mean budget among the points that reach at least 95% observed coverage.

Margin rulePrimary budget ↓Coverage (%)Additional budget ↓Coverage (%)
RACE19,155.7595.2120,429.9795.70
Constant21,256.6795.4121,659.8595.10
Length-proportional19,385.2896.1919,428.0895.05
Residual-based18,660.6495.1219,838.6695.20

Budgets are reserved output tokens. The main and extra evaluations use 512 prompts with two responses each and 500 prompts with four responses each. These are summaries of fixed-predictor trade-offs, with actual coverage shown for each point. RACE reserves smaller-than-constant margins in both groups, while residual-based margins require smaller budgets than RACE at these points.

Comparison at matched generation cost. We compare repeated-response spread supervision with single-response residual supervision, using separate training prompts for the length and variation predictors. One study matches the number of generated responses; the other uses common generated-token thresholds and accounts for overshoot.

Repeated-response supervision lowers reservations in several setups. However, these savings come with coverage trade-offs: the lower ends of the paired 95% intervals fall below the set −1-percentage-point coverage-noninferiority threshold.

Information beyond predicted length. The requested correlations are as follows:

CorrelationPrimary cohortAdditional cohort
Predicted variation vs. predicted length0.9240.930
Predicted variation vs. observed response-length variation0.4110.548
Partial correlation controlling for predicted length0.0930.134

Entries are Spearman or partial Spearman correlations. Observed variation uses the absolute length difference for 496 main prompts with two naturally ended responses and the mean pairwise difference for 477 extra prompts with four naturally ended responses. The positive partial correlations show a modest link beyond predicted length, while much of the variation signal overlaps with predicted length.

Q5. How stable are the results, and why does range supervision change the budget?

We ran five full fit, validation, and calibration re-splits and five training seeds on the original split. Entries are mean ± sample SD; coverage differences are RACE minus the comparator in percentage points.

VariationComparatorPrimary saving (%)Primary ΔcoverageAdditional saving (%)Additional Δcoverage
Re-splitsGBM-PAC9.987 ± 13.7681.543 ± 0.85010.539 ± 14.277−0.280 ± 1.383
Re-splitsMixture-PAC0.610 ± 3.3860.254 ± 0.5201.076 ± 3.0190.060 ± 1.022
Re-splitsLength-proportional−1.938 ± 1.8250.215 ± 0.243−0.791 ± 1.578−0.130 ± 0.375
Training seedsGBM-PAC7.189 ± 3.0221.035 ± 0.4128.908 ± 2.984−0.380 ± 0.520
Training seedsMixture-PAC0.092 ± 2.7200.410 ± 0.7731.628 ± 2.5860.190 ± 0.408
Training seedsLength-proportional2.623 ± 2.067−0.859 ± 0.3614.175 ± 1.753−1.030 ± 0.509

Average savings over GBM-PAC stay positive, but re-splitting causes a lot of variation and changes the ranking against length-proportional margins. These results measure fitting and calibration sensitivity apart from paired test-sample uncertainty. Mixture refits use the CPU version.

Your rescaling argument is correct. Multiplying fixed scale predictions by a positive constant and recalibrating keeps the endpoint, with a maximum numerical difference of 7.28 × 10⁻¹² tokens in our check. Retraining changes the fitted function through target flooring, the log transform, and validation selection. The original spread and range runs also used different seeds:

Scale targetSeed 1042 budgetSeed 642 budget
80–20 spread21,720.9722,481.04
Full range22,044.0322,117.20

So, the original difference combines target choice and fitting variability. The crossed experiment separates these factors and confirms that fixed-prediction calibration remains the same.

AuthorsOpenReview ↗

Response to Reviewer oWQV-3

Thank you for the detailed editorial checks. We will:

  1. Cite the Aeala ShareGPT mirror.
  2. Identify the simulation trace as the primary 512 Qwen3-8B/DeepScaleR prompts, using recorded response 0.
  3. State the 95% target and 95% calibration confidence alongside the abstract's 97.56% observed coverage.
  4. Adopt your suggested reservation wording at lines 102–103.
  5. Introduce and cite LightGBM (Ke et al., 2017), XGBoost (Chen and Guestrin, 2016), and the AFT implementation.
  6. Use “nondecreasing,” remove redundant lower clipping, and write Uα,δU_{\alpha,\delta}.
  7. Separate overloaded symbols for states, active requests, scores, depths, projections, generated tokens, calibration rank, rollout count, and quartile groups.
  8. Standardize margin names, colors, markers, and “scale-label budget.”
  9. Define the rich input as four 256-dimensional projections plus 40 scalars: five metadata, five final-logit, and 30 layer-level features. The five additions to RACE's standard input are final-token entropy and log-probabilities for EOS, newline, <think>, and </think>.
  10. Distinguish Figure 3b's 496 prompt-level paired errors (ρ = 0.477) from Table 9b's 1,005 individual natural responses (ρ = 0.398).
  11. Label every cohort and predictor: 21,720.97 is the original predictor on 512 × 2 outputs; 21,229.17 is the same predictor on 500 × 4; 21,542.40 refits only the variation predictor on the common 1,051/183 fitting/validation prompts and evaluates on 500 × 4. The 9.94% and 11.58% savings correspond to the first two evaluations.

We appreciate your thoughtful feedback and welcome further discussion. If these clarifications and additional results address your concerns, we would be grateful if you would reconsider your assessment and update your rating accordingly.

Reviewer_nAwkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RACE, a method for optimizing KV-cache reservations in large language model (LLM) serving. The paper focuses on the problem of wasted KV-cache space caused by the uncertainty of request output lengths before decoding begins. The authors point out that predicting only the average output length cannot account for the variation in response lengths generated from the same prompt under stochastic sampling. To address this issue, RACE extracts features from hidden states produced during the prefill stage and trains two predictors to estimate response length and its variability, respectively. The predicted variability is then used to adjust a request-specific safety margin, while the safety margin coefficient is selected using a separate calibration dataset to satisfy a target coverage rate. The experimental evaluation demonstrates the effectiveness of RACE across multiple large language models and datasets. The paper reports that RACE reduces token reservations by approximately 9.94%–22.26% compared with the GBM-Quantile-PAC baseline, while maintaining an observed coverage rate above 95%. In addition, scheduling simulations show that smaller cache reservations can reduce request completion time and improve system throughput.

Critical strengths
  1. Accounting for request-specific response-length variability One strength of the paper is that it considers differences in response-length variability across requests. Two prompts may have similar typical response lengths but substantially different variation under stochastic decoding. A uniform safety margin would not capture this difference. RACE addresses this issue by predicting response-length variability separately and using it to adjust the reservation for each request. This is a reasonable approach to reducing unnecessary KV-cache reservations, particularly for requests whose response lengths are relatively stable.
  2. Combining variability prediction with coverage calibration RACE not only aims to improve prediction accuracy but also uses a calibration mechanism to control coverage, allowing the system to balance resource utilization and reliability. Experimental results show that RACE can reduce KV-cache reservations while maintaining a high coverage rate.
Critical weaknesses
  1. Possible selection bias from excluding capped responses during training In Section 3.2, the paper excludes responses that reach the data-collection limit when constructing training targets for both the response-length and length-variation predictors. A prompt must also have at least two naturally terminated responses to contribute a training sample. In the main experiment, the collection limit and service output limit are both 32,768 tokens. However, Section 3.3 includes capped responses during calibration by assigning them the service output limit as their target. This exclusion is not random. Prompts with longer or more variable responses are more likely to produce capped outputs. Removing these responses may bias the training targets toward shorter and less variable outputs, while some high-risk prompts may be excluded entirely. Consequently, the learned variability predictor may underestimate the uncertainty of requests that need larger safety margins. Although global calibration can maintain overall coverage, it cannot necessarily correct inaccurate risk rankings across requests. The paper does not directly examine how this training choice affects request-level variability prediction or reservation efficiency.

  2. Limited evidence from an actual serving system The paper primarily relies on simulations to evaluate scheduling benefits. Although the results show that RACE can reduce request completion time and improve throughput, the gains may be affected by the complexities of real-world serving systems. The lack of evaluation in a real deployment therefore weakens the paper's claims.

Questions for authors
  1. Effect of excluding capped responses from training Section 3.2 excludes capped responses when constructing the training targets for both the response-length and variability predictors. Since prompts with longer or more variable responses are more likely to reach the collection limit, this filtering may introduce selection bias and lead to underestimated variability for such requests. Figure 3 provides evidence that the variability predictor is useful, but its analysis is restricted to naturally terminated responses, leaving its performance on capped responses unclear. Could the authors evaluate how this filtering affects variability prediction and reservation efficiency, particularly for prompts that produce capped outputs? A comparison with training approaches that include capped service lengths or treat capped responses as censored observations would help assess the impact of this design choice.

  2. Online prediction overhead RACE requires extracting hidden states from multiple prefill layers, performing feature projection and computation, and running two prediction models. These operations introduce additional overhead during online serving, but their latency is not clearly quantified. Could the authors report the overhead of feature extraction and prediction separately, and evaluate whether the reported improvements in request completion time and throughput remain significant after accounting for these costs?

  3. Comparison with prompt-length-grouped safety margins The introduction discusses H-MAS, which groups requests by input prompt length and estimates safety margins using historical prediction errors within each group. The paper argues that this approach cannot distinguish requests with different response-length variability. However, the experiments compare RACE with constant, length-proportional, and shuffled margins, without directly evaluating a prompt-length-grouped baseline. Could the authors include a comparison using the grouping strategy of H-MAS, while keeping the response-length predictor, calibration dataset, and coverage target the same? This would help clarify how much of RACE's improvement comes from learning request-specific variability rather than using a simpler grouping-based margin.

1 comment
AuthorsOpenReview ↗

Response to Reviewer nAwk

Thank you for highlighting the need for request-specific variability modeling and calibration. We have added the capped-response comparisons, prediction-cost measurements, and the prompt-length-grouped baseline you requested.

Q1. How does excluding capped responses affect variation prediction and reservations?

Thank you for raising this training decision. We compared targets based on naturally terminated lengths (N) and those using capped service lengths (S). The first letter indicates the length target, and the second indicates the variation target. The first four variants use the same eligible training prompts, while the last one includes all fitting and validation prompts.

Training targetsPrimary budget ↓Coverage (%)Additional budget ↓Coverage (%)
NN: original RACE21,720.9797.5621,229.1796.40
NS: service-length variation21,737.3598.3421,214.1196.10
SN: service-length center21,805.4797.7521,307.9596.20
SS: same eligible prompts21,283.5997.8520,759.7295.80
SS: all fitting/validation prompts21,849.7197.5621,366.3596.30

The primary set contains 512 prompts and 1,024 responses. The additional set has 500 prompts and 2,000 responses. All variants include capped responses during calibration and evaluation, using a 95% coverage target and 95% calibration confidence.

With the same eligible training prompts, SS reduces primary-set reservations by 2.01% and increases coverage from 97.56% to 97.85%. For the additional set, the smaller budget changes coverage from 96.40% to 95.80%. Including capped service lengths provides an alternative training target that clearly affects reservation size and coverage.

We also examined prompts with capped test outputs: 16 in the primary group and 23 in the additional group. Changing only the variation target from N to S raises primary-group coverage from 78.13% to 87.50%, while additional-group coverage remains at 86.96%. In these groups, the scale–service-spread correlations are −0.395/−0.098 for NN and −0.496/−0.271 for NS. These subgroup results show improved primary coverage but continued challenges in ranking variation among capped requests. We will include these results with Figure 3.

Q2. What is the prediction cost, and do serving benefits remain after integration?

Thank you for requesting a breakdown. We measured feature computation, transfer and assembly, and prediction separately on an RTX 3090:

Batch sizeFeature computation (ms)Transfer and assembly (ms)Predictors and budget computation (ms)Added prefill time (ms)
10.9990.0390.6291.832
41.0870.0450.9092.318
81.1240.0491.0362.974

The entries show average batch-level latencies. The last column measures the prefill increment separately over 100 paired repetitions to capture the integrated execution overhead.

We also tested Qwen3-8B in BF16 with TP=1 on a setup using three RTX A6000 GPUs, each with 48 GB. There was one prefill instance, two decode instances, and KV transfer used NixlConnector. Each decode instance has 65,536 KV token slots. At the tested DeepScaleR high-load point:

MethodThroughput (tokens/s) ↑Mean JCT (s) ↓p95 JCT (s) ↓
GBM-PAC1922,9677,361
RACE2192,4696,382

RACE provides 14.06% higher throughput, 16.78% lower mean JCT, and 13.30% lower p95 JCT. These results show that prediction integration overhead is only a few milliseconds and that there are real serving gains in the tested PD deployment.

Q3. How does RACE compare with prompt-length-grouped margins?

We developed a grouping rule based on H-MAS, using fitting-set prompt-length quartiles and within-group out-of-fold prediction errors. RACE and the grouped rule use the same length predictor, calibration prompts, coverage target, and calibration process.

Test setRACE budget ↓Grouped budget ↓RACE coverage (%)Grouped coverage (%)
Primary21,720.9724,677.0897.5698.14
Additional21,229.1724,891.8196.4098.75

RACE uses 11.98% and 14.71% fewer tokens while covering 97.56% and 96.40% of responses. The grouped rule reserves more space and achieves higher observed coverage. The paired coverage-difference intervals are [−1.56, +0.39] and [−3.65, −1.20] percentage points, and both lower bounds are below the set −1-point noninferiority threshold. The table shows the trade-off between capacity savings and coverage.

Thank you again for your positive assessment of our work and your constructive feedback, which has helped us strengthen the paper.

Reviewer_FtuCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RACE predicts KV-cache reservation budgets from the target LLM's prefill hidden states. Two predictors learn typical output length and length variability using supervision derived from repeated offline generations for each training prompt. RACE assigns request-specific safety margins and calibrates the budgets on an independent prompt set. Across four model-dataset settings involving three LLMs and three datasets, it reduces mean reserved output tokens by 9.94%-22.26% relative to GBM-PAC, with observed service-length coverage above 95% in every setting. Scheduling simulations on a fixed 512-request trace report a 21.44% reduction in mean request completion time and a 4.01% increase in token throughput relative to GBM-PAC, averaged across the evaluated capacities, loads, and arrival seeds.

Critical strengths
  1. Modeling differences in response-length variability across prompts directly addresses the trade-off between wasted reservations and underestimated demand under stochastic decoding.
  2. The method reuses the target model's prefill hidden states and uses two LightGBM predictors to produce request-specific budgets before decoding begins.
  3. Under the stated assumptions, independent calibration provides a population service-length coverage guarantee at the chosen calibration confidence, with a fallback to the full service limit when the calibration sample is insufficient.
Critical weaknesses
  1. A baseline comparison and adequate discussion of the closely related MAPS [1] are missing. MAPS also calibrates output-length upper bounds to guide KV-memory admission and scheduling, but the current manuscript neither cites nor discusses it. RACE has distinct variability modeling and calibration guarantees, and already includes a constant-margin ablation; nevertheless, MAPS should be included as a baseline to compare budget quality, prediction overhead, and scheduling performance under matched models, workloads, and coverage targets, with differences in resources and calibration guarantees made explicit. If a direct comparison is infeasible, the authors should explain the specific obstacles, why reasonable adaptations cannot resolve them, and whether controlled comparisons of the prediction and calibration components are possible.
  2. The simulation does not incorporate the performance cost of handling capacity violations. Section 4 and Appendix E allow decoding to continue when aggregate KV demand exceeds nominal capacity. Although capacity-excursion fractions and peak demand are reported, the performance metrics omit the costs of handling such conflicts through preemption, swapping, or recomputation, leaving it unclear whether the reported throughput and completion-time gains would persist.
  3. The paper does not report end-to-end scheduling results from a real inference engine. The reported scheduling gains come from discrete-event simulations. It therefore remains unclear how much of the simulated benefit survives feature extraction, budget prediction, KV transfer, and scheduling overheads.
  4. Integration with on-demand paged allocation and its incremental benefit need clarification. The paper uses predicted budgets for reservation-based admission, but does not sufficiently explain how this policy interacts with the underlying paged allocator or evaluate the additional benefit of introducing it into a system that already allocates KV memory on demand.

[1] Zhang T, Chen Y, Zhao Y, et al. MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving. arXiv preprint arXiv:2609.15359, 2026.

Questions for authors
  1. Please include MAPS as a baseline in a fair comparison to establish RACE's incremental benefit. If this is infeasible, please explain the specific reasons, why reasonable adaptations would not suffice, and which component-level comparisons could establish the benefits of the methodological differences.
  2. Do the throughput and mean request completion-time gains persist when the simulation includes the costs of handling aggregate KV demand above capacity through preemption, swapping, or recomputation?
  3. Are there results from integrating RACE into a real inference engine that quantify prediction-path overhead and report end-to-end request latency, time to first token, time per output token, and throughput?
  4. How does prediction-based admission interact with on-demand paged allocation, and what additional benefit does introducing RACE's predicted budgets provide under the same engine, resources, and workload?
3 comments
AuthorsOpenReview ↗

Response to Reviewer FtuC

Q1. Missing comparison with MAPS.

A1: Thank you for pointing out MAPS. We see it as closely related to RACE and agree that comparing them helps evaluate the benefits of our approach.

We compared MAPS and RACE by focusing on the calibration part. We used the one-sided additive calibration rule with a shared length predictor and a fixed calibration set, which we call MAPS*. This version does not include MAPS’s auxiliary predictor, its continuously updated calibration, or the full scheduling pipeline.

Both methods use the same frozen length predictions, 277 calibration responses, evaluation outputs, and a 32,768-token output limit. We apply ordinary split conformal calibration with a 95% marginal coverage target in both cases. We assess budget quality using Qwen3-8B on 512 test prompts from the DeepScaleR-Preview-Dataset, with two sampled responses per prompt. For serving, both methods use the same GPU, KV capacity, request trace, concurrency limit, and serving setup.

MethodMean token budget ↓Coverage (%)Added critical-path latency (ms) ↓Throughput (tokens/s) ↑Mean JCT (s) ↓p95 JCT (s) ↓
MAPS*21,256.6795.412.3453,1327,091
RACE, matched CP calibration20,139.7596.481.7482,7686,654

RACE reduces the mean reserved output-token budget by 5.25% and increases observed coverage by 1.07%. Throughput improves by 6.67%. Mean and p95 completion times decrease by 11.62% and 6.16%, respectively. Added critical-path latency decreases from 2.3 ms to 1.7 ms.

These results show that RACE’s request-specific safety margins work better than a shared additive correction in this setting. We will add the comparison and its implementation details to the revised manuscript and keep them in the camera-ready version if accepted.

Q2. Do the claimed throughput improvements and completion time reductions actually hold in real-world serving scenarios?

A2: Thank you for your question, which helps us improve our practical evaluation. We tested RACE in an online, PD-disaggregated vLLM service using three NVIDIA RTX A6000 GPUs (each with 48GB memory). One GPU handles prefill and two handle decoding, with KV-cache transfer managed by NixlConnector. Each instance runs Qwen3-8B in BF16 with tensor parallelism set to one. The usable KV-cache capacity is 65,536 tokens per decode instance.

The following table shows DeepScaleR results at a set high-load point. Compared to GBM-PAC, RACE improves throughput by 14.06% and reduces mean and p95 completion times by 16.78% and 13.30%. Against tuned native vLLM, the improvements are 26.59%, 23.23%, and 19.26%. RACE also beats Mixture-PAC, with 5.80% higher throughput, 10.25% lower mean completion time, and 9.32% lower p95 completion time.

These results show that RACE’s throughput and completion-time benefits hold in the tested online PD deployment, supporting its practical value under high load.

AuthorsOpenReview ↗

Response to Reviewer FtuC-2

Q3. Are prediction overhead, TTFT, TPOT and end-to-end latency all measured?

A3: Thank you for pointing out prediction overhead and user-perceived latency. We measured the added prediction-path latency, TTFT, TPOT, and request completion time, including waiting time in the serving pipeline.

In separate paired tests on the RTX 3090, RACE adds an average of 1.832 ms, 2.318 ms, and 2.974 ms to prefill at batch sizes of one, four, and eight. These measurements include the feature-extraction and prediction path.

The A6000 latency measurements are:

KV capacity per decoder (tokens)Mean TTFT (s)Mean TPOT (s/token)Mean first-to-second-token wait (s)
65,5360.57230.15041380.62
40,0000.68830.34012927.08

The first token is produced during prefill, and later decoding may wait for admission. We report the first-to-second-token interval along with TTFT to show this waiting time. TPOT is calculated for each request as the time between its first and last tokens, divided by the number of output tokens after the first, and then averaged.

In the main high-load PD test at 65,536 tokens per decoder, RACE has a mean completion time of 2469 s and a p95 of 6382 s, compared to 2967 s and 7361 s for GBM-PAC. These are reductions of 16.78% and 13.30%.

Overall, these measurements show millisecond-scale added prefill latency in the timing test and shorter request completion times in the PD deployment.

Q4. How does prediction-based admission interact with on-demand paging and what further benefit does RACE provide?

A4: Thank you for asking about on-demand paging. RACE works with vLLM’s paged allocator. Its predicted budget decides when a request is admitted for decoding, while physical KV pages are allocated as needed as tokens are generated. The predicted reservation does not mean all physical pages are allocated in advance.

To measure the extra benefit, we compared RACE with tuned native vLLM in the same three-A6000 PD setup, using Qwen3-8B, the same workload, and a KV capacity of 65,536 tokens per decode instance. Both setups use vLLM’s on-demand page allocation.

MethodThroughput (tokens/s) ↑Mean JCT (s) ↓p95 JCT (s) ↓
Tuned native vLLM17332167904
RACE21924696382

RACE boosts throughput by 26.59% and lowers mean and p95 completion times by 23.23% and 19.26%. These results show that prediction-based admission provides additional benefits alongside on-demand paging in the tested PD setup.

We appreciate your thoughtful feedback and welcome further discussion. If these clarifications and additional results address your concerns, we would be grateful if you would reconsider your assessment and update your rating accordingly.

Dear Reviewer,

Thank you again for taking the time to read our response. Your questions helped us improve the paper and guided the additional experiments we shared in our responses. As the discussion draws to a close, please let us know if there is anything you would still like us to clarify. We’d be happy to address that point directly while there’s still time.

If the new results have addressed concerns that influenced your initial rating, we hope you’ll consider raising it. We appreciate your consideration.

Reviewer_qENWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the problem of allocating KV-cache space in a prefill-decode disaggregated LLM serving system. The proposed method, RACE, predicts a per-request output token length by separately estimating the typical response length and the variability using hidden states from the prefill & data from multiple rollouts. The effectiveness of this approach was studied through scheduling simulations. The experimental results on Qwen3-8B/DeepScaleR showed that RACE reduces the reserved space by ~10% compared to a calibrated quantile-regression baseline (GBM-Quantile-PAC).

Critical strengths

(+) The paper clearly motivates its decomposed design, which separates typical length from length variability, and points out that two prompts with the same median length can have very different variance. This provides a new insight.

(+) The experimental result suggests that the proposed approach has the potential to better estimate the length of an output, thus saving valuable KV cache space.

Critical weaknesses

(-) The evaluation only relies on simulated studies using synthetic workload rather than a real system implementation and real-world traces. As a result, the study does not accurately account for potential performance overhead in practical systems. It is not clear whether the measured memory KV cache savings will lead to meaningful performance and efficiency improvement. For example, the evaluation does not consider practical systems overhead such as the costs of training, access to prefill hidden state, etc. Also, there is no discussion or study of how RACE will handle distribution/workload shifts and what the cost will be.

(-) The reported improvement of about 10% is relatively small and the improvement is even smaller compared to some other design points, such as Mixture.

(-) The proposed approach has a non-trivial chance of underestimating the actual output length. For example the paper targets coverage between 90% and 97.5% (underestimation in 2.5% to 10%). However, the paper does not model or accurately discuss what happens when the KV cache usage is underestimated. Recovering from out-of-memory in 5-10% of cases seems quite expensive.

Questions for authors
  • What will be the performance and efficiency improvement of RACE on real systems, considering all the practical overhead and cost (such as training, access to pre-filled states, the cost of adapting to distribution shift, and the cost of recovering from underestimation)?

  • How does RACE handle distribution shifts and underestimation?

  • What are the main algorithmic novelties beyond the separation of median and variance compared to other designs? How much of the improvements are coming from using more data from repeated rollouts versus the separation of median and variance? How novel is this separation if you compare broad related work in systems resource allocation beyond reserving KV cache for decoding?

3 comments
AuthorsOpenReview ↗

Response to Reviewer qENW

Thank you for pointing out the need to predict response-length variation and for noting the missing systems and robustness evidence. We have now included online-serving measurements, a cost breakdown, and targeted comparisons to address these points.

Q1. Do the reservation savings translate into practical serving benefits?

We tested RACE in a PD-disaggregated vLLM service using three NVIDIA RTX A6000 GPUs, each with 48 GB of memory. One instance manages prefill and two handle decoding, with KV transfer through NixlConnector. Each runs Qwen3-8B in BF16 with tensor parallelism set to one. Each decode instance has a usable KV capacity of 65,536 tokens. The results below use the same DeepScaleR workload at the high-load operating point we evaluated.

MethodThroughput (tokens/s) ↑Mean JCT (s) ↓p95 JCT (s) ↓
Tuned native vLLM1733,2167,904
GBM-PAC1922,9677,361
Mixture-PAC2072,7517,038
RACE2192,4696,382

RACE increases throughput by 14.06% compared to GBM-PAC and reduces mean and p95 completion times by 16.78% and 13.30%. It also improves all three metrics over Mixture-PAC and native vLLM. These results extend the original simulation evidence to the tested online PD deployment.

Q2. What happens when RACE underestimates a response length?

Thank you for raising this concern. RACE’s predicted budget guides request admission, but it is not a stopping condition or a strict memory limit for each request. If a response exceeds its predicted budget, it does not automatically cause an out-of-memory failure. The budget manages admission, and physical KV pages are allocated as decoding happens. A request can use more than its reservation if there are free pages. In earlier single-engine hard-cap replays, allocation conflicts were handled by preemption and recomputation rather than letting memory demand exceed the pool.

In the separate hard-cap replay, the 95/95 RACE runs had no preemptions. With a 90% coverage target, there was one budget miss out of 32 requests, but this did not require preemption or recomputation. This shows that budget misses and physical conflicts are different. It does not measure the recovery cost if RACE underestimation causes a physical conflict. We will clearly distinguish these two rates and avoid treating a 5% budget-miss target as a 5% out-of-memory rate. These results show that prediction-based admission works well and that budget overruns are not the same as memory conflicts.

Q3. How does RACE adapt to workload shifts, and what does adaptation cost?

Thank you for pointing out the need for workload adaptation. We added a DeepScaleR-to-ShareGPT transfer experiment, keeping both predictors unchanged and recalibrating the reservation margins using one response from each of 277 target conversation families. We evaluated the results on 502 separate conversation families.

Recalibration lowers the mean reservation from 16,758.41 to 7,347.83 tokens, a 56.15% drop. Observed coverage goes from 99.30% to 97.71%, which is still above the 95% target. These results show that recalibration significantly reduces unnecessary reservations after the workload change while keeping high response coverage.

The calibration set uses 277 responses, totaling 618,553 generated output tokens. RACE recalibrates the extra reserved space for requests without retraining either predictor. The calibration step takes about 0.85 ms.

Q4. What is novel, and what explains its gains?

RACE learns request-specific safety margins from response-length variation within prompts using prefill states. Repeated responses provide the variation targets, and independent calibration sets the overall margin size using established coverage-calibration principles.

Using the same training data and length predictor, RACE reserves 13.19% fewer tokens and covers 0.95 percentage points more outputs than shuffled margins. Both versions use the same predicted variation values, but only differ in how they assign them to requests. This highlights RACE’s main advantage: matching safety margins to individual requests improves both reservation size and coverage.

We appreciate your thoughtful feedback and welcome further discussion. If these clarifications and additional results address your concerns, we would be grateful if you would reconsider your assessment and update your rating accordingly.

Dear Reviewer qENW,

As the discussion period draws to a close, we wanted to ask whether you have any remaining questions about our response or the additional experiments. We’d be happy to clarify anything that remains unclear. If you feel our response has addressed your concerns, we’d be grateful if you would consider revising your rating.

Thank you again for your time and thoughtful feedback.

Dear Reviewer,

Thank you again for taking the time to read our response. Your questions helped us improve the paper and guided the additional experiments we shared in our responses. As the discussion draws to a close, please let us know if there is anything you would still like us to clarify. We’d be happy to address that point directly while there’s still time.

If the new results have addressed concerns that influenced your initial rating, we hope you’ll consider raising it. We appreciate your consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewer

Dear Reviewer,

Thank you again for your time. As the discussion period is nearing its end, we would appreciate any further feedback on our responses and additional experimental results. If any concerns remain or further clarification would be helpful, please let us know. We would be happy to address them and continue the discussion.

Thank you for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.