- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RACE predicts output-token reservations from prefill hidden-state features. Two regressors estimate typical response length and within-prompt variation using repeated rollouts. A separate calibration set chooses an order-statistic multiplier for an upper endpoint, targeting population coverage with a specified confidence level. Experiments measure reservation size and coverage across four model/task settings, ablate request-specific margin assignment, and replay a reservation-based scheduler on a fixed request trace.
Critical strengths
- The statistical target and calibration protocol are explicit. Appendix C distinguishes population coverage from confidence over calibration samples, accounts for capped observations conservatively, handles tied/discrete scores, and specifies a full-budget fallback. Using one prespecified response per independent calibration prompt avoids treating correlated within-prompt samples as independent calibration units.
- The experiments meaningfully probe the margin predictor. The primary comparison uses matched prefill features and paired prompt-level bootstrap intervals. The fixed-length-predictor ablation separately recalibrates constant, length-proportional, shuffled, and learned margins and varies rollout count. This gives useful evidence that assigning variation to the correct request helps, beyond simply changing the global margin size (Sections 5–6; Figures 3–4).
Critical weaknesses
- The simulator does not enforce the physical resource constraint whose benefit is being claimed. Section 4 and Appendix E allow requests to continue when actual KV demand exceeds capacity; overruns are recorded but incur no eviction, recomputation, offload, or admission-recovery cost. Each active request also advances one token per step regardless of batch or context. At the 40K setting, Table 17 reports a 9,469-token increase in peak demand versus GBM-PAC, despite a small increase in excursion-time fraction. Short excursions can still make a physical schedule infeasible. The simulation is transparently described, but does not yet establish that the reported throughput/JCT gains survive a hard-cap serving implementation (pp. 5, 26–27).
- The strongest-baseline and equal-coverage evidence is mixed. Table 1's native-feature Mixture-PAC reserves 21,552 tokens at 97.85% coverage in the primary setting, versus RACE's 21,721 at 97.56%; this comparator has richer features, so it does not invalidate the feature-matched gain, but it limits claims of best practical performance. Table 17 likewise reports RACE at −1.45% throughput and +2.01% mean JCT relative to Mixture-PAC. Appendix Table 4 also marks coverage noninferiority as failed for ShareGPT and GSM8K. Meeting the nominal 95% target on the observed sample is distinct from establishing comparable coverage to GBM-PAC. These disclosed results deserve greater emphasis when interpreting the 9.94–22.26% savings (pp. 7, 18, 26).
Others:
- The added value of repeated-rollout scale supervision over a simple length-proportional margin is small and is not tested at matched coverage.
- The rule μ(S) + q·s(S), with a learned scale and a held-out order statistic, is the normalized (locally weighted) split-conformal construction (e.g., Papadopoulos et al., 2008; Lei et al., 2018; neither is cited) combined with a training-conditional/PAC rank (Vovk, 2012; Park et al., 2019). Appendix C is the corresponding standard order-statistic argument. The novelty therefore rests on supervising s with within-prompt spread from repeated rollouts.
- Variability from training and calibration is not quantified.
Minors:
- L. 278: ShareGPT is cited as Kong et al. (2026), a systems paper (LLM-CoOpt). Please cite the dataset source instead; App. B.3 refers to the "Aeala V4.3 cleaned split".
- Sec. 4 and App. E do not say which setting's 512 requests form the scheduling trace. The magnitudes suggest Qwen3-8B/DeepScaleR, but ShareGPT-fresh also has 512 test prompts. They also do not say which of the two recorded responses per prompt is used.
- L. 24 (abstract): the 97.56% coverage is reported without the 95% target and 95% confidence that it should be read against.
- L. 102–103: "how many output tokens a serving system should reserve KV-cache capacity for before decoding begins" is awkward. Suggested: "the number of output tokens for which a serving system should reserve KV-cache capacity before decoding begins".
- L. 197: "The two LightGBM regressors" is the first mention of LightGBM. Please introduce and cite it (Ke et al., 2017), and likewise XGBoost (Chen & Guestrin, 2016) for AFT-PAC (L. 907).
- L. 223: "the ordered scores increase with k" should read "are nondecreasing in k". In Eq. 7, the lower clip at 0 is redundant, since μ ≥ 0, q ≥ 0, and s ≥ 32. The notation U_α also hides its dependence on δ.
- Notation is overloaded: S is both the prefill state and part of S_max (L. 245); A is both the active set (L. 245) and the scores A_i (App. C); L is both output length and the depth set (App. B.1); G is both the projection matrix (Eq. 10) and G_i(t) (L. 261); k is both the calibration rank and, on the x-axes of Figs. 6–7, the number of responses per prompt (Tables 6–8 use n_resp); Q_p denotes quantiles while Q1–Q4 denote quartiles.
- The same margin rules have different names in different places: Shuffled / Length-proportional / Constant (Fig. 4, Tables 6–8); permuted / center proportional / constant (Figs. 6–7); Permuted learned scale / Center proportional / Constant / additive PAC (Table 9b). Colors and markers for the same rule also differ between Fig. 4 and Figs. 6–7. Fig. 7 says "scale-label budget", while the text says "label budget".
- Native Mixture-PAC's extra inputs are called "vocabulary-level statistics" (L. 286) in one place and "richer boundary statistics" (L. 932–933) in another; please define them once. Table 9a's "Boundary scalars" (40 dims) and "RACE (full feature set)" (1,064 dims) are never defined.
- Fig. 3b reports ρ = 0.477 between s(S) and length-prediction error, and Table 9b reports 0.398 for what reads as the same quantity. Please state the difference: a per-prompt mean over 496 pairs vs. 1,005 individual responses.
- Several different RACE numbers with three responses per prompt appear on different cohorts and test sets: 21,721 (Tables 1, 5, 9), 21,542 (Table 6), and 21,229 (Tables 7, 15; Fig. 2). This is why the saving over GBM-PAC is 9.94% in one place and 11.58% in another. Captions should state which predictor and test set they use.
Questions for authors
- What reservation/throughput/JCT benefits remain when KV capacity is enforced, with a specified overrun policy and its cost? Please include prefill-feature extraction/prediction overhead or a measured bound on it.
- Can you compare RACE and Mixture-PAC with the same richer feature set, and show matched observed-coverage trade-offs or a prespecified noninferiority comparison in each setting? How should the failed noninferiority results in Table 4 qualify the cross-setting conclusion?
- How sensitive are coverage and reservation efficiency to deployment-time prompt or sampling-policy drift, and what is the recalibration procedure? This question concerns deployment robustness; it does not allege an error in the iid theorem.
- Can you compare margin rules at matched observed coverage? For example, plot budget–coverage curves like Fig. 2c for the learned, length-proportional, and constant margins. Please also include a scale model trained on absolute residuals |y − μ(S)| from single responses at the same total generation budget. It would also help to report Spearman(s(S), μ(S)) and the partial correlation between s(S) and |L0 − L1| given μ(S). If the advantage over these simpler rules at matched coverage is only 1–2%, the case for repeated-rollout supervision weakens considerably.
- How stable are the main comparisons across random fit/validation/calibration splits and training seeds? Please report the mean ± s.d. of savings and coverage against GBM-PAC, Mixture-PAC, and length-proportional margins. Relatedly, with two or three responses the range target is 1/0.6 times the 80–20 spread target (App. B.2), up to the floor and the log(1 + ·) offset, and calibration should absorb a constant factor. Why, then, does the "range scale" variant shift the budget by 1.8% and coverage by 0.8 pp (Table 5)? Unless I am missing an implementation detail, this points to run-to-run variance comparable to the reported gaps.
5 comments
Response to Reviewer oWQV-2
Q4. How do learned margins compare at matched observed coverage and generation cost?
Thank you for suggesting these controls. We added comparisons for reservation efficiency, supervision cost, and information beyond predicted response length.
Reservation budgets at or above 95% observed coverage. We keep the length predictor fixed and check all 277 calibration ranks for each margin rule. The table shows the smallest mean budget among the points that reach at least 95% observed coverage.
| Margin rule | Primary budget ↓ | Coverage (%) | Additional budget ↓ | Coverage (%) |
|---|---|---|---|---|
| RACE | 19,155.75 | 95.21 | 20,429.97 | 95.70 |
| Constant | 21,256.67 | 95.41 | 21,659.85 | 95.10 |
| Length-proportional | 19,385.28 | 96.19 | 19,428.08 | 95.05 |
| Residual-based | 18,660.64 | 95.12 | 19,838.66 | 95.20 |
Budgets are reserved output tokens. The main and extra evaluations use 512 prompts with two responses each and 500 prompts with four responses each. These are summaries of fixed-predictor trade-offs, with actual coverage shown for each point. RACE reserves smaller-than-constant margins in both groups, while residual-based margins require smaller budgets than RACE at these points.
Comparison at matched generation cost. We compare repeated-response spread supervision with single-response residual supervision, using separate training prompts for the length and variation predictors. One study matches the number of generated responses; the other uses common generated-token thresholds and accounts for overshoot.
Repeated-response supervision lowers reservations in several setups. However, these savings come with coverage trade-offs: the lower ends of the paired 95% intervals fall below the set −1-percentage-point coverage-noninferiority threshold.
Information beyond predicted length. The requested correlations are as follows:
| Correlation | Primary cohort | Additional cohort |
|---|---|---|
| Predicted variation vs. predicted length | 0.924 | 0.930 |
| Predicted variation vs. observed response-length variation | 0.411 | 0.548 |
| Partial correlation controlling for predicted length | 0.093 | 0.134 |
Entries are Spearman or partial Spearman correlations. Observed variation uses the absolute length difference for 496 main prompts with two naturally ended responses and the mean pairwise difference for 477 extra prompts with four naturally ended responses. The positive partial correlations show a modest link beyond predicted length, while much of the variation signal overlaps with predicted length.
Q5. How stable are the results, and why does range supervision change the budget?
We ran five full fit, validation, and calibration re-splits and five training seeds on the original split. Entries are mean ± sample SD; coverage differences are RACE minus the comparator in percentage points.
| Variation | Comparator | Primary saving (%) | Primary Δcoverage | Additional saving (%) | Additional Δcoverage |
|---|---|---|---|---|---|
| Re-splits | GBM-PAC | 9.987 ± 13.768 | 1.543 ± 0.850 | 10.539 ± 14.277 | −0.280 ± 1.383 |
| Re-splits | Mixture-PAC | 0.610 ± 3.386 | 0.254 ± 0.520 | 1.076 ± 3.019 | 0.060 ± 1.022 |
| Re-splits | Length-proportional | −1.938 ± 1.825 | 0.215 ± 0.243 | −0.791 ± 1.578 | −0.130 ± 0.375 |
| Training seeds | GBM-PAC | 7.189 ± 3.022 | 1.035 ± 0.412 | 8.908 ± 2.984 | −0.380 ± 0.520 |
| Training seeds | Mixture-PAC | 0.092 ± 2.720 | 0.410 ± 0.773 | 1.628 ± 2.586 | 0.190 ± 0.408 |
| Training seeds | Length-proportional | 2.623 ± 2.067 | −0.859 ± 0.361 | 4.175 ± 1.753 | −1.030 ± 0.509 |
Average savings over GBM-PAC stay positive, but re-splitting causes a lot of variation and changes the ranking against length-proportional margins. These results measure fitting and calibration sensitivity apart from paired test-sample uncertainty. Mixture refits use the CPU version.
Your rescaling argument is correct. Multiplying fixed scale predictions by a positive constant and recalibrating keeps the endpoint, with a maximum numerical difference of 7.28 × 10⁻¹² tokens in our check. Retraining changes the fitted function through target flooring, the log transform, and validation selection. The original spread and range runs also used different seeds:
| Scale target | Seed 1042 budget | Seed 642 budget |
|---|---|---|
| 80–20 spread | 21,720.97 | 22,481.04 |
| Full range | 22,044.03 | 22,117.20 |
So, the original difference combines target choice and fitting variability. The crossed experiment separates these factors and confirms that fixed-prediction calibration remains the same.
Response to Reviewer oWQV-3
Thank you for the detailed editorial checks. We will:
- Cite the Aeala ShareGPT mirror.
- Identify the simulation trace as the primary 512 Qwen3-8B/DeepScaleR prompts, using recorded response 0.
- State the 95% target and 95% calibration confidence alongside the abstract's 97.56% observed coverage.
- Adopt your suggested reservation wording at lines 102–103.
- Introduce and cite LightGBM (Ke et al., 2017), XGBoost (Chen and Guestrin, 2016), and the AFT implementation.
- Use “nondecreasing,” remove redundant lower clipping, and write .
- Separate overloaded symbols for states, active requests, scores, depths, projections, generated tokens, calibration rank, rollout count, and quartile groups.
- Standardize margin names, colors, markers, and “scale-label budget.”
- Define the rich input as four 256-dimensional projections plus 40 scalars: five metadata, five final-logit, and 30 layer-level features. The five additions to RACE's standard input are final-token entropy and log-probabilities for EOS, newline,
<think>, and</think>. - Distinguish Figure 3b's 496 prompt-level paired errors (ρ = 0.477) from Table 9b's 1,005 individual natural responses (ρ = 0.398).
- Label every cohort and predictor: 21,720.97 is the original predictor on 512 × 2 outputs; 21,229.17 is the same predictor on 500 × 4; 21,542.40 refits only the variation predictor on the common 1,051/183 fitting/validation prompts and evaluates on 500 × 4. The 9.94% and 11.58% savings correspond to the first two evaluations.
We appreciate your thoughtful feedback and welcome further discussion. If these clarifications and additional results address your concerns, we would be grateful if you would reconsider your assessment and update your rating accordingly.
Response to Reviewer oWQV
Thank you for pointing out the calibration protocol and margin-allocation experiments. Your questions led us to run more experiments on serving, matched features, workload shift, and stability. We answer them below.
Q1. Do the gains persist with physical KV limits and real serving costs?
We tested RACE in a PD-disaggregated vLLM service with three RTX A6000 GPUs, each with 48 GB of memory. The setup had one prefill instance and two decode instances, using NixlConnector for KV transfer. Each instance ran Qwen3-8B in BF16 with tensor parallelism set to one. Each decoder had 65,536 usable KV token slots. At the DeepScaleR high-load point:
RACE boosts throughput by 14.06% and lowers mean and p95 JCT by 16.78% and 13.30% compared to GBM-PAC. Against Mixture-PAC, it increases throughput by 5.80% and reduces mean and p95 JCT by 10.25% and 9.32%. These results show RACE’s scheduling benefits in the tested PD service.
On a separate RTX 3090, we measured mean added prefill times of 1.832, 2.318, and 2.974 ms for batch sizes 1, 4, and 8. These measurements include feature extraction and budget prediction during the prefill window, which ends when sampling starts.
We also separate budget overruns from physical allocation conflicts. In a hard-cap replay, physical conflicts are managed by preemption and recomputation. The 95/95 RACE runs had no preemptions. In a 90% target sensitivity test, one out of 32 requests missed its budget, but no recovery events happened. So, an underestimated request can continue as long as physical pages are available, and budget misses are different from recovery events.
Q2. How does RACE compare with Mixture-PAC using the same richer features?
Thank you for suggesting this control. We retrained RACE using the same 1,064-dimensional input as Mixture-PAC on Qwen3-8B/DeepScaleR. Both use a 95% target coverage and 95% calibration confidence. The main and extra evaluations have 512 prompts with two responses each and 500 prompts with four responses each.
RACE gets higher observed coverage on both sets, with a bigger primary budget and a smaller extra budget. The extra budget-saving interval includes zero. These paired results show the trade-off between reservation and coverage, and the PD comparison highlights RACE’s practical serving advantage over Mixture-PAC.
In the four-setting comparison, the coverage-noninferiority criterion is met in both DeepScaleR settings. The ShareGPT and GSM8K coverage-difference intervals have lower endpoints of −2.25 and −1.44 percentage points, which are below the −1-point threshold. To sum up: reservations are smaller in all four settings, observed coverage stays above 95%, and coverage noninferiority is shown in two settings.
Q3. How sensitive are coverage and reservation efficiency to deployment-time prompt or sampling-policy drift, and what is the recalibration procedure?
We tested prompt-distribution drift from DeepScaleR to ShareGPT, keeping both predictors fixed and comparing the original calibration to recalibration in the target domain.
For recalibration, we collect one response from each of 277 target conversation families. Using their predicted and observed response lengths, we adjust the reservation margins at the same 95% coverage target and 95% calibration confidence. We evaluate using 502 separate conversation families.
The original calibration is conservative for the target workload. Recalibration cuts the mean reservation by 56.15%, while observed coverage stays above 95%. These results show that RACE can lower excess reservation after a workload shift by recalibrating its safety margins, without retraining either predictor.