acceptodds
Under review as a conference paper at ICLR 2027

SpecGrove: Reusable Tree Capacity for Concurrent Diffusion Speculation

Abstract

Concurrent block-diffusion speculative decoding must share limited verification capacity across requests with heterogeneous returns from tree expansion. We introduce SpecGrove, which turns a single DFlash proposal and DDTree expansion per request into reusable, nested capacity tiers. SpecGrove scores tiers by covered draft-prefix mass and jointly selects heterogeneous tree sizes using measured packed-pass cost. An exact-row Bellman frontier optimizes the resulting finite-tier surrogate, and selected trees share one request-isolated target pass. We establish optimality for this finite-tier objective and characterize conditions under which packed verification preserves the joint autoregressive output law. Across Qwen3-4B and Qwen3-8B on four reasoning and code-generation benchmarks, SpecGrove improves complete wall-clock throughput by 13.0–16.8% over matched DFlash and 4.7–39.0% over capacity-feasible fixed-size DDTree at concurrency eight. Under Poisson arrivals, it reduces tail latency relative to matched allocation controls.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, large language models, inference acceleration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dFUcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a service framework called SpecGrove to address the issue of multiple requests competing for limited validation capacity in concurrent block diffusion speculative decoding. Its main contributions include three aspects: firstly, by utilizing the single DFlash proposal and the prefix closure truncation of the best priority DDTree extension, a reusable nested capacity hierarchy was constructed to avoid duplicate tree building overhead; Secondly, based on the coverage prefix draft quality, request service weight, and actual packaging execution cost curve, a finite level agent optimization objective was constructed, and precise Bellman frontier dynamic programming was used to solve capacity allocation; Thirdly, empirical evaluations were conducted on throughput, Poisson arrival delay, and BF16 generation quality based on Qwen3-4B and Qwen3-8B models and four benchmark test sets on a single NVIDIA H20 graphics card.

Critical strengths

Propose a reusable nested capacity hierarchy based on DDTree prefix truncation: The paper points out that by performing prefix closure truncation on a single DFlash proposal and a canonical best priority DDTree extension, multiple nested capacity hierarchies can be naturally derived, with each level's exact row consumption containing its own root anchor, allowing a single proposal to support multiple feasible tree sizes without the need for repeated tree construction. Establishing precise Bellman frontiers and finite level proxy optimization objectives: The paper theoretically derives the expected target progress determined by the coverage prefix probability, and designs the precise Bellman frontiers algorithm by weighting the proxy ratio (considering the quality of draft coverage, generation length, and waiting age) combined with the measured packaging execution cost curve. The algorithm solves the optimal allocation while considering non convex cost jumps. Comprehensive matching benchmark and online service evaluation were conducted: The author evaluated on four inference and code generation benchmarks based on Qwen3-4B and Qwen3-8B models, and the results showed that at a concurrency of 8, SpecGrove's full clock throughput increased by 13.0% -16.8% compared to matched DFlash, 4.7% -39.0% compared to fixed size DDTree, and effectively reduced tail latency under Poisson arrival load

Critical weaknesses

1、The paper will make precise Bellman frontier solutions and optimality certificates as its core contributions. However, Table 15 in Appendix C.9 shows that under the same adaptive DDTree hierarchy, the difference in throughput between SpecGrove and simple equal share allocation and greedy allocation is only negative 0.9% to positive 0.0%. Among the 8 test units of the 8B model, 7 of them even lag behind these two simple strategies. The author also admits that the performance improvement comes from dynamic hierarchical scaling rather than the Bellman algorithm itself, which directly weakens the necessity of constructing complex dynamic programming and theoretical derivation throughout the text

2、The paper specifically introduces control rules such as service weight, capacity fitting, and high concurrency level gating. However, the ablation experiment in Table 15 shows that removing service weights, active gating, capacity fitting, or changing to age only admission one by one has an impact on the throughput of both 4B and 8B models within plus or minus 0.5% and in different directions. This proves that the aforementioned additional control mechanisms have failed to bring substantial performance gains, only unnecessarily increasing system hyperparameters and engineering complexity.

Questions for authors

Given that the equal division and greedy strategies in Table 15 have almost identical or even better performance than SpecGrove, please explain why the extremely low complexity equal division or greedy strategies are not directly used,

2 comments

Critical Weakness 1 and Question 1: Why retain DP instead of equal or greedy allocation?

Thank you. One tree expansion supplies reusable tiers; DP selects their joint combination. SpecGrove retains DP because the tested finite action space can be solved globally at low measured cost.

For fixed eligible tiers, utilities, weights, and a positive row-cost curve, DP compares the best utility at every reachable exact row total and returns the global optimum of that finite objective in exact arithmetic. Greedy follows one local upgrade path, while equal allocation restricts combinations through its sharing rule. Reusable tiers provide the main observed throughput benefit; DP gives their allocation an exact, inexpensive rule.

To isolate the allocation rule, we added replay and timing comparisons with reusable tiers, utilities, and cost curves held fixed, together with exhaustive verification and complete-call throughput controls.

Table 1. Rebuttal-added allocation-objective replay and DP cost measurements. DP closes the equal/greedy objective gaps at a median call cost below 0.2 ms.

Model  Waves  Equal gap (%)   Greedy gap (%)  Greedy same  DP median
               mean/max        mean/max      objective      (ms)
-----------------------------------------------------------------
4B      188   1.370/10.055     0.0668/0.815       85.1%       0.188
8B      195   2.167/8.008      0.0038/0.647       99.0%       0.189

Gap is the difference between DP and heuristic objective values divided by the DP value. All allocators use identical frozen utilities and row-cost curves. Waves are correlated execution records; timings cover the native DP call.

Greedy reaches DP’s objective in 85.1%/99.0% of recorded 4B/8B waves, helping explain the original throughput near-ties. In the remaining waves, a locally ranked upgrade path can miss a better joint combination. Maximum observed objective gaps are 0.815%/0.647% for greedy and 10.055%/8.008% for equal allocation.

DP closes these gaps without requiring the utilities and measured cost curve to satisfy conditions that would make a greedy path globally optimal.

Table 2. A newly verified eight-request wave comparing DP and greedy allocations. DP obtains a higher finite-objective value by selecting a different joint tier combination.

Allocator  Node-tier combination               Rows    Objective
----------------------------------------------------------------
DP         Eight 23-node trees                  192     1.572115
Greedy     Seven 23-node, one 11-node tree       180     1.559303

All 6561 joint tier choices were enumerated using identical utilities, weights, cost curve, and capacity limit.

In this example, greedy selects 180 rows because its next density-ranked upgrade jumps to 202 rows across the calibrated cost breakpoint at 193. DP selects eight 23-node trees at 192 rows and obtains 1.572115 rather than 1.559303, a 0.815% finite-objective improvement. Exhaustive enumeration confirms the optimum for these inputs.

The frontier retains the best utility at every reachable exact row total and then compares utility/cost ratios across those totals. This joint comparison avoids the missed combination in the example.

We also report complete-call throughput, including planning time. Native DP calls have p95 costs of 0.206/0.211 ms on 4B/8B.

Table 3. New complete-call throughput controls for DP relative to equal and greedy allocation. The wall-clock effect varies with the tested configuration.

Model       DP/equal gain range       DP/greedy gain range
---------------------------------------------------------
4B             -1.74% to +0.12%            -0.50% to +4.77%
8B             -1.63% to +1.94%            -0.35% to +0.21%

These controls use R=384, C=8/16, GSM8K/MATH-500, 128 prompts/task, temperature 1, and cap 256.

The original aggregate near-ties remain in revised-paper Table 18. The new results explain why DP is retained: it provides the global optimum of the fixed finite objective at low measured overhead, while its actual wall-clock effect is measured separately. Equal and greedy are useful low-cost controls; neither supplies the same guarantee for arbitrary eligible tiers and measured nonconvex cost curves. Revised Sections 4.2 and 5.7 and Appendix C.9, paper Tables 19–20, report the guarantee and these records.

Critical Weakness 2: The role and complexity of service-policy controls

Thank you for raising the role and complexity of these controls. We view them as configurable hyperparameters of the serving policy.

The weight coefficient adjusts how strongly the allocator prioritizes requests that lag in generated length or have waited longer; a coefficient of zero gives uniform weights. Capacity fitting and active gating determine eligible tiers under capacity and load conditions, while admission selects waiting requests for a wave. These controls let a deployment specify service priorities and candidate capacity choices within the same allocation framework.

To extend the original ablation, we added a 64-cell sensitivity study across 4B/8B, C=8/16, and GSM8K/MATH-500. It tests weight coefficients 0/1/2, removal of active gating or tier fitting, and age-only admission, alongside default DP, equal, and greedy policies.

Table 4. Default SG throughput in the newly added service-policy hyperparameter study. Default gating gives positive gains in all four tested 4B configurations, while the other effects depend on configuration.

Alternative             4B: SG/alternative       8B: SG/alternative
-----------------------------------------------------------------
Weight coefficient 0       -0.15% to +2.13%          -1.47% to +1.44%
Weight coefficient 2       -0.81% to +2.60%          -3.24% to +0.82%
No active gating           +0.31% to +2.80%          -0.55% to +0.29%
No tier fitting            -0.53% to +0.27%          -0.59% to +0.29%
Age-only admission         -0.27% to +0.44%          -0.01% to +0.13%

The default weight coefficient is 1. Each range covers all four task/concurrency configurations per model. Entries are percentage throughput changes of default SG relative to the listed alternative; negative values mean the alternative is faster. Each arm changes only the listed setting.

The results distinguish the effects of individual settings. On 4B, default active gating improves observed throughput by 0.31%–2.80% over disabling it in all four configurations. Default-versus-age-only contrasts remain close to zero, while weight and tier-fitting effects vary across configurations. The original small aggregate changes therefore describe throughput sensitivity on those workloads; the new grid measures how that sensitivity varies with model and workload.

These settings can be tuned on a disjoint calibration workload: choose the priority coefficient according to the service objective, then compare eligible-tier and admission settings under the intended load. Uniform weights provide a simpler objective when differentiated request priorities are unnecessary. Other settings can remain at their defaults or be selected through calibration. The sensitivity study supplies workload-specific tuning evidence, with the default configuration as a reproducible reference.

Their engineering role is confined to inputs already used by the system: weights change the utility objective, fitting and gating change eligible actions, and admission changes the requests presented to a wave. Once those inputs are fixed, the same Bellman recurrence solves the resulting finite objective. Changing policy settings reuses the proposal, tier, and verification machinery. Revised Section 5.7 and Appendix C.9, paper Table 22, report the settings and sensitivity results.

Thank you again. We would be happy to discuss any further questions

Reviewer_JqvaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses row allocation among concurrent requests in diffusion speculative decoding. SpecGrove reuses nested cutoffs from a single DDTree expansion as capacity tiers, estimates utility via draft-prefix mass, and uses a measured row-cost curve and exact-row Bellman DP to select tree sizes. The authors provide theoretical optimality proofs and an output-law preservation theorem under ideal assumptions. Experiments on Qwen3-4B/8B show throughput gains over DFlash and fixed 45-node DDTree at high concurrency.

Critical strengths
  1. On the novelty side, I believe that reusing nested cutoffs as capacity tiers is an elegant and easily integrable idea. The row accounting rigorously includes per-request anchors, and the exact-row DP effectively handles non-convex measured cost curves.

  2. The paper uses a combination of wall-clock time, covers mixed workloads, Poisson arrivals, tail latency, and fairness, and includes extensive ablations.

  3. The paper transparently discloses negative results (e.g., single-request slowdowns, BF16 numerical mismatches, mixed 4B low-concurrency results), with detailed appendix diagnostics. This would help other works to gain important insights.

Critical weaknesses

However, my initial assessment has the following weaknesses. Through the rebuttal phase, the authors have addressed most of them and I've raised my score for this work. For reference and integrity, I've edited the original concern and still included them in the following:

  1. There are some disconnection between the theoretical guarantees and practical BF16. Theorem 5 assumes exact same-prefix target conditionals, but the authors report BF16 sequence matches of only 1248/1536 (4B) and 1155/1536 (8B), explicitly setting strict_sequence_law_certified=false. The utility proxy Qi(v)Q_i(v) also lacks empirical calibration against true target progress. Consequently, the theoretical framework does not verify the actual system's lossless correctness, which is critical given the 4B MBPP quality drop (-2.03 pp, CI excluding zero). This part needs further clarification.

  2. More empirical evidence is needed to support the main contribution of this work. E.g., Table 15 shows that full SpecGrove’s gain over equal/greedy allocation is nearly zero (−0.9%/+0.0%-0.9\%/+0.0\% on 4B). The main benefit stems from "having multiple selectable tiers," not the exact-row Bellman DP. Furthermore, the cost curve models only total rows, ignoring full wall time and context lengths; thus, "exact allocation" is only exact for a coarse surrogate, not the actual throughput objective.

  3. The practical generalizability is a bit ambiguous and the baseline is insufficient. The fixed-tree baseline lacks capacity tuning, as the main table only compares against an infeasible fixed 45-node DDTree at R=193, C=8, rather than the optimal fixed tier for each (R,C). Combined with the prototype-level scale (single H20, no CUDA graphs, 4B/8B models), it remains highly ambiguous whether the reported gains would translate to production serving environments.

Questions for authors
  1. Do the BF16 numerical mismatches originate from masks, positions, SDPA reduction order, or KV layout? The authors should report posterior divergence (KL/TV) rather than just sequence matches, and provide a numerically fixed version to verify whether the 4B MBPP quality drop disappears.

  2. The authors are encouraged to compare against the optimal capacity-tuned fixed tier for each (R,C) on the full matrix (e.g., choosing the best from {11, 23, 45}), to isolate the true benefit of request-wise allocation over well-tuned static configurations.

Through the rebuttal response from the authors, I believe these questions have been addressed.

4 comments

Critical Weakness 1 and Question 1: Numerical correctness, utility calibration, and MBPP quality

Thank you. We added precision-controlled posterior comparisons, repeated sequence checks, execution-path diagnostics, utility calibration, and a matched MBPP precision control. These experiments separately examine the numerical premises of the correctness theorem, the allocation estimator, and sampled task quality.

Numerical origins and precision controls

The new posterior experiment holds prefixes, candidate trees, and tokens fixed while changing target-verification precision. Increasing precision reduces mean posterior TV by over three orders of magnitude and eliminates Top-1 disagreements in the measured probes.

We compare cached AR, unpacked trees, single-request packed trees, and multi-request packed trees under both default and math SDPA. The audited paths pass byte-identical prefix-KV cloning, position-semantic, attention-support, and request-isolation checks.

Table 1. New precision-controlled posterior comparisons on frozen prefixes and trees. FP32 target verification reduces mean TV by over three orders of magnitude and eliminates Top-1 mismatches in these probes.

Model  Target    Mean TV     Max TV      Mean KL    Top-1 differences
------------------------------------------------------------------
4B     BF16     7.33e-3     1.99e-1     1.03e-3              10/1312
4B     FP32     1.74e-6     3.50e-5     5.21e-11              0/1312
8B     BF16     5.36e-3     1.09e-1     6.57e-4               9/1312
8B     FP32     1.73e-6     3.37e-5     5.55e-11              0/1312

KL is directed from AR to the candidate path. Each model/precision pair contains 1312 correlated probes from four fixed MBPP prompts, covering default and math SDPA. This intervention promotes target verification only; prefixes and trees remain frozen.

The new sequence check repeats the original execution conditions and recovers 1536/1536 greedy matches for each model in FP32, while retaining tier construction and allocation.

Table 2. New FP32 sequence checks alongside the originally reported BF16 comparisons. FP32 increases observed agreement to 1536/1536 for both models.

Model    BF16 matches     FP32 matches    Temperature    Output cap
          (original)          (new)
----------------------------------------------------------------
4B         1248/1536        1536/1536           0               64
8B         1155/1536        1536/1536           0               64

Counts refer to repeated condition executions per model, rather than independent prompt samples.

We also added a layer-wise analysis of archived execution paths. Across four selected 4B GSM8K cases, unpacked and single-request packed trees agree in all 24 intermediate-layer comparisons. AR/tree differences already occur before multi-request packing, and 8/12 AR/unpacked comparisons still differ under math SDPA.

Table 3. Newly reported layer-wise analysis of archived BF16 execution paths. Differences already occur without multi-request packing, while unpacked and single-request packed trees agree in all tested comparisons.

Compared execution paths                    Default SDPA    Math SDPA
-------------------------------------------------------------------
AR / unpacked tree                                 12/12         8/12
Unpacked / single-request packed                     0/12         0/12
Single-request / multi-request packed               12/12        12/12

Entries count intermediate-layer numerical disagreements. Each backend covers four selected GSM8K cases at 1/12/46 verification rows; the multi-request path uses eight slots.

These controls locate a precision-sensitive execution-path effect spanning execution shapes and attention backends. They do not uniquely identify an individual CUDA operator; that requires operator-level tracing.

The correctness argument concerns the conditional distributions used to generate tokens. Capacity allocation occurs before target sampling, and every emitted token is a fresh target draw. Under Theorem 5’s same-prefix conditionals, causal decisions, independent draws, and eventual EOS/cap completion, the chain rule preserves the joint stopped-output law. The revised paper presents this exact-kernel result alongside the measured BF16 deviations and the recorded strict_sequence_law_certified=false status. FP32 substantially improves numerical agreement with the same-prefix premise while retaining the capacity mechanism. Revised Section 5.8 and Appendix C.10, paper Table 24, report these controls.

Calibration of the utility proxy

We added a calibration measuring both progress prediction and the resulting tier decision. On 16 archived prefixes at three tiers per model, draft-based selection incurs mean/max target-objective loss of 0.66%/0.86% on 4B and 0.00%/0.00% at saved precision on 8B.

Table 4. New draft-utility calibration and target-objective selection-loss analysis. The 4B tier labels agree less often, but their measured objective loss on the same frozen trees is small.

Model  Prefixes x tiers  MAE (tokens)  Rank agreement  Tier agreement  Loss mean/max
--------------------------------------------------------------------------------
4B          16 x 3            2.238          81.7%            3/16      0.66%/0.86%
8B          16 x 3            2.243          66.1%           15/16      0.00%/0.00%

Rank agreement compares prefixes within a fixed tier. Selection loss compares target expected progress divided by the original row cost on the same frozen trees, using untruncated single-request progress. The differing 8B tier labels are tied in target-objective value at the saved numerical precision.

Tier-label agreement and decision loss measure different aspects of the estimator: one records whether the selected tier changes; the other measures the consequence for expected progress per measured row cost.

Revised Sections 3.1 and 4.1 explicitly connect the deployed score to retained draft-prefix mass and account for EOS/cap stopping through an additional truncation term in the utility-error bound. Theorem 4 relates uniform utility and cost errors to selection loss for the concurrent finite objective.

The target verifier generates output tokens; the estimator determines where verification capacity is allocated. This separation allows the same reusable tiers and exact frontier to use an improved progress estimator without changing row accounting or sampling. Appendix C.9, paper Table 21, reports the calibration and decision-loss records.

Matched MBPP precision control

We added a repeated-run MBPP precision control using the original 128 prompts, five generation repetitions per prompt, C=4, R=193, temperature 1, output cap 1024, and the original extraction, tests, and scoring limits. Crossing AR/SG with BF16/FP32 produces 2560 generated and scored answers; none reaches the output cap.

Table 5. New repeated-run MBPP precision control under the original task protocol. FP32 gives a smaller SG–AR point-estimate gap than BF16.

Precision    AR passed    SG passed    SG-AR (pp)      95% CI (pp)
----------------------------------------------------------------
BF16          424/640      412/640          -1.88    [-4.38, +0.63]
FP32          425/640      418/640          -1.09    [-2.97, +0.63]

Here, pp denotes percentage points. Intervals use 10000 paired bootstrap resamples over prompt clusters, averaging the five run-level differences within each prompt. All 2560 scores were checked against their generated answers.

The theoretical quantity is expected task score. Under Theorem 5’s assumptions, SG and AR share the stopped-output law and therefore the expectation of any fixed MBPP pass indicator. For conditional TV errors bounded uniformly over relevant histories and packed states, Theorem 6 bounds the expected-score difference by the stopped-output TV distance, which is at most the smaller of 1 and the sum of the per-step conditional-error bounds. Revised Section 4.3 states this connection. The posterior statistics in Table 1 are measurements on recorded states, rather than uniform bounds over all possible histories.

The two precision controls address complementary questions. The sequence comparison uses temperature 0 and cap 64; the task-quality experiment uses temperature 1 and cap 1024. Native CUDA multinomial sampling follows each execution path’s generator consumption, so the experiment does not impose token-wise common-random-number coupling. Distribution-equivalent samplers can consequently produce different finite answer sets.

The FP32 quality arm promotes both target and draft; the frozen-tree posterior control promotes target verification alone. The estimated BF16-to-FP32 change in SG–AR is +0.78 pp, with paired 95% interval [−1.09, +2.66] pp. Thus, the precision intervention restores observed greedy agreement and sharply reduces posterior error, while the task-quality contrast remains uncertain at this sample size. The original −2.03 pp MBPP result remains reported; the new control does not establish that this original deficit has disappeared. Revised Section 5.8 and Appendix C.8, paper Table 17, report the quality records and original baseline comparisons.

Critical Weakness 2: The contribution of exact allocation and the accuracy of the cost model

Thank you. One tree expansion supplies reusable tiers; DP selects their joint combination. SpecGrove retains DP because the tested finite action space can be solved globally at low measured cost.

For fixed eligible tiers, utilities, weights, and a positive row-cost curve, DP retains the best utility at every reachable exact row total and returns the global optimum of that finite objective in exact arithmetic. Greedy follows one local upgrade path, while equal allocation restricts combinations through its sharing rule. Reusable tiers provide the main observed throughput benefit; DP supplies an exact allocation rule for their finite objective.

The original throughput near-ties remain in revised-paper Table 18, corresponding to original Table 15. For this rebuttal, we added allocation-objective replay, exhaustive verification, DP timing, and complete-call throughput controls using identical frozen utilities and cost curves.

Table 6. New allocation-objective replay and DP overhead measurements on recorded waves. Exact-row DP closes the observed objective gaps left by equal and greedy allocation.

Model  Waves  Equal gap (%)   Greedy gap (%)  Greedy same  DP median
               mean/max        mean/max      objective      (ms)
-----------------------------------------------------------------
4B      188   1.370/10.055     0.0668/0.815       85.1%       0.188
8B      195   2.167/8.008      0.0038/0.647       99.0%       0.189

Gap is the difference between the DP and heuristic objective values divided by the DP value. Waves are correlated execution records; timings cover the native allocation call.

Greedy attains the same objective in 85.1%/99.0% of recorded 4B/8B waves, explaining the close aggregate throughput. In the remaining waves, its single upgrade path can omit a better joint combination.

The exact-row recurrence includes each request’s anchor in its row cost. Starting from utility zero at row total zero and infeasibility elsewhere, it considers every eligible tier for each request, retaining the maximum accumulated weighted utility at each exact total. It then compares utility divided by measured cost across every reachable total up to the budget. This covers joint combinations that a single marginal-upgrade path need not visit.

Table 7. A newly verified real-wave example of a missed joint allocation. DP selects a combination with a higher surrogate objective than greedy.

Allocator  Node-tier combination               Rows    Objective
----------------------------------------------------------------
DP         Eight 23-node trees                  192     1.572115
Greedy     Seven 23-node, one 11-node tree       180     1.559303

Greedy selects 180 rows; its next density-ranked upgrade jumps to 202 rows across the calibrated cost breakpoint at 193. DP selects eight 23-node trees at 192 rows, achieving a 0.815% objective improvement. Enumeration of all 6561 joint choices confirms the optimum on identical utilities, weights, and cost curve.

The recurrence uses O(nRk) arithmetic, O(R) reward memory, and O(nR) back-pointer memory, with at most three tiers here. The evaluated tuple-copying Python path has a conservative O(n²Rk) time bound. Median native calls take 0.188/0.189 ms, and p95 costs are 0.206/0.211 ms for 4B/8B.

The new complete-call controls include planning time. DP/equal gains range from −1.74% to +0.12% on 4B and −1.63% to +1.94% on 8B; DP/greedy gains range from −0.50% to +4.77% and −0.35% to +0.21%, respectively. These controls use R=384, C=8/16, GSM8K/MATH-500, 128 prompts/task, temperature 1, and cap 256. Appendix C.9, paper Table 20, reports the throughput controls.

The rationale for DP is therefore exact finite-objective allocation at low measured cost, rather than a uniform wall-clock advantage over heuristics. It also separates search error from predictor calibration: the optimizer solves its specified finite objective, while the relationship between that objective and execution time is evaluated independently. Revised Sections 4.2 and 5.7 and Appendix C.9, paper Tables 19–20, report these results.

For the cost model, we added context-transfer profiling and a complete-wave cost extension. Throughput always uses complete invocation time, including planning.

Table 8. New cost profiling across context lengths and target verification’s share of full wall time. Longer contexts increase prediction error, while non-target stages account for a substantial fraction of execution time.

Model  MAPE@64  MAPE@256  MAPE@1024  Target share C=4  Target share C=8
-------------------------------------------------------------------
4B       1.52%    13.34%     31.47%            60.20%            53.92%
8B       0.94%    11.73%     34.17%            60.76%            53.52%

The row-cost curve is calibrated at 64-token contexts and evaluated at lengths 64/256/1024. Wall-time shares sum actual measurements over four tasks at R=193.

For dense SGLang, the new wave-cost predictor adds median calibrated non-target overhead at each active-request count to target-forward cost. This overhead covers proposal, tree preparation, sampling, and KV commit. Disjoint calibration inputs fit the profile and select the cost variant. Both dense models select wave cost at C=4 and target-only cost at C=8.

At a fixed active count, the overhead term changes the utility/cost trade-off while preserving the exact-row recurrence. The current predictor uses rows and active count; Table 8 quantifies transfer across context lengths. A positive row curve conditioned on a wave’s fixed context state can use the same frontier, whereas dependence on the selected request combination would require an expanded state.

The dense SGLang results evaluate the complete integrated policy, with 0.23%–5.15% four-task geometric-mean gains over tuned DDTree (Table 10). The cost extension is evaluated as part of that policy. Revised Sections 4.2 and 5.7 and Appendix C.9, paper Table 23, connect finite-objective optimality to the measured costs and throughput.

Critical Weakness 3 and Question 2: Capacity-tuned baselines and practical generalizability

Thank you. We added the full capacity-tuned matrix: two models, 12 budget/concurrency settings, and four tasks, giving 96 matched comparisons. SG improves all 12 four-task model–budget aggregates at concurrency at least eight over the independently calibrated fixed tier, by 0.27%–5.85%.

For every model/budget/concurrency configuration, independent calibration selects a feasible tier from {11, 23, 45}. We separately report the strongest fixed tier observed on each evaluation task as a hindsight oracle. At R=193/C=8, eight 23-node trees require 192 rows and provide a feasible fixed reference; the 45-node shape requires 368 rows. Gain denominators use the stated executable reference.

Table 9. New capacity-tuned fixed-tier and hindsight-oracle comparisons. All 12 high-concurrency aggregates favor SG over the calibrated fixed reference.

 R    C    4B/tuned    8B/tuned    4B/oracle    8B/oracle
------------------------------------------------------
96    1      +6.27%      +0.29%       +6.27%       +0.29%
96    4      +2.65%      +2.08%       +2.65%       +2.08%
96    8      +5.08%      +5.85%       +5.08%       +5.85%
193   1      -2.58%      -3.72%       -2.58%       -3.72%
193   4      -0.42%      -3.63%       -0.42%       -3.64%
193   8      +3.36%      +0.27%       +3.36%       +0.27%
193  16      +4.24%      +3.94%       +4.24%       +3.94%
384   1      +1.39%      -0.72%       +1.39%       -0.72%
384   4      -0.46%      -4.09%       -0.46%       -4.09%
384   8      +3.03%      +1.61%       +1.68%       +0.71%
384  16      +2.51%      +3.25%       -0.37%       +1.49%
384  32      +2.44%      +1.74%       +2.44%       +1.74%

Entries are geometric-mean gains over four matched task-wise throughput ratios, using 128 prompts/task, temperature 1, and cap 256. Tuned tiers use independent calibration; oracle tiers are selected on evaluation data. These are point estimates from one run per configuration.

The original 4.7%–39.0% comparison is at R=384/C=8, where all eight fixed-45 trees fit in 368 rows. The new study strengthens that comparison by tuning static capacity throughout the matrix. It finds positive calibrated-reference aggregates at concurrency at least eight and mixed outcomes at lower concurrency.

The oracle columns quantify the effect of selecting the strongest fixed tier on evaluation data. For example, 4B at R=384/C=16 gives +2.51% against the calibrated tier and −0.37% against the oracle.

We also added SGLang evaluations on 4B/8B and an MoE target, together with a native 27B dense-model study. Within SGLang, SG and DDTree share tree verification and accepted-path commit, enabling a comparison of allocation on the same tree-execution path.

Table 10. Serving-engine and model-scale evaluations added for this rebuttal. SG improves the four-task geometric mean over the stated DDTree reference in every listed model–concurrency comparison.

Target                  Backend    Gain at C=4    Gain at C=8
------------------------------------------------------------
Qwen3-4B                SGLang           +0.23%         +2.94%
Qwen3-8B                SGLang           +5.15%         +1.27%
Qwen3-Coder-30B-A3B      SGLang           +3.37%         +2.98%
Qwen3.8-27B             Native           +1.45%         +4.42%

Gains are geometric means of four matched task-wise throughput ratios. SGLang uses independently tuned DDTree at R=193. Native 27B uses fixed 23/11-node references at R=193/384 for C=4/8. All comparisons use 128 prompts/task, temperature 1, and cap 256.

Thank you again. We would be happy to discuss any further questions

Reviewer_oZrTOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpecGrove to select heterogeneous tree sizes using measured packed-pass cost. It also establishes optimality for this finite-tier objective and characterizes conditions under which packed verification preserves the joint autoregressive output law. SpecGrove improves complete wall-clock throughput by 13.0–16.8% over matched DFlash and 4.7–39.0% over capacity-feasible fixed-size DDTree at concurrency 8.

Critical strengths

This paper has a deep theoretical analysis, which is interesting. The evaluation results are presented in a detailed and easy-to-understand way. The proposed method is useful and practical for concurrency levels of 8-16.

Critical weaknesses
  1. Not evaluated on top of serving systems like SGLang, vLLM, or TRT-LLM. (For low and medium concurrency, even report sglang/vllm/trtl-llm would be very meaningful.)

  2. Model scale. Larger models, like 122B Qwen3.5 or 320B GLM5.3 Flash, would be super interesting. If not allowed by resources, Qwen3.8-27B is also meaningful.

Questions for authors

How is the method when serving larger models or larger MoE models?

4 comments

Critical Weakness 1: Evaluation on mainstream serving systems

Thank you for this suggestion. We added 72 SGLang measurements covering three targets (Qwen3-4B, Qwen3-8B, and Qwen3-Coder-30B-A3B), two concurrency levels (C=4/8), four benchmarks (GSM8K, MATH-500, HumanEval, and MBPP), and three methods (DFlash, capacity-tuned DDTree, and SG).

SGLang provides model loading, native DFlash proposals, scheduling, and KV pools. DDTree and SG share tree verification and accepted-path commit, allowing their capacity policies to be compared on the same tree-execution path. The DFlash comparison additionally includes a difference in verification path. Fixed DDTree tiers are selected using independent calibration inputs, providing a tuned static reference for each configuration.

Table 1. Throughput gains in the SGLang experiments added for this rebuttal. All six model/concurrency comparisons have positive four-task geometric-mean gains over DFlash and independently tuned DDTree.

Target                   C    SG/DFlash    SG/tuned DDTree
-------------------------------------------------------
Qwen3-4B                 4      +29.73%            +0.23%
                         8      +17.79%            +2.94%
Qwen3-8B                 4      +24.49%            +5.15%
                         8      +17.48%            +1.27%
Qwen3-Coder-30B-A3B       4      +25.99%            +3.37%
                         8      +23.54%            +2.98%

Each entry is the geometric mean of four matched task-wise throughput ratios. These are point estimates from one timing round. The 4B/C=4 gain over tuned DDTree is modest (+0.23%). Individual tasks can show regressions, including −2.39% on 4B/C=8 GSM8K.

All methods use the same physical A100 80GB, SGLang 0.5.23, synchronous TP=1, BF16 weights/KV, and FP32 target output projection. Within each target comparison, the methods use matched target/draft pairings and inputs. Each benchmark contains 128 prompts, with row budget R=193, temperature 1, and output cap 256. Fixed tiers and dense SG profiles are selected through independent calibration; dense profile fitting and profile selection use separate inputs.

Measurements use balanced interleaving. Throughput is calculated from actual emitted-token totals and complete invocation time, including cleanup, and therefore includes online planning and execution overhead. Accepted records show no CPU-quota throttling or overlap with recorded Triton compilation; rejected attempts are archived separately. Each target also passes 32/32 finite greedy preflight comparisons.

The C=4 measurements extend the evaluation to moderate concurrency, while the C=8 measurements examine greater concurrent demand. Across these tested settings, the positive aggregate gains provide evidence that the capacity mechanism remains useful within SGLang’s execution and KV infrastructure. Sharing tree verification between SG and DDTree helps distinguish the effect of their capacity policies from differences between verification implementations.

This added study evaluates synchronous SGLang throughput at C=4/8. Queueing and tail latency are evaluated separately in the original Poisson-arrival experiments in revised Section 5.6; native C=1 results remain in Appendix C.7. Revised Section 5.4, paper Table 4, and Appendix C.1 report the added SGLang results and execution protocol.

Critical Weakness 2 and Question 1: Larger dense and MoE models

Thank you for suggesting these additional model scales. We added two studies: a native Qwen3.8-27B experiment with its DFlash2 draft, following the suggested 27B scale, and a SGLang experiment using Qwen3-Coder-30B-A3B-Instruct with its corresponding DFlash draft. These studies evaluate a larger dense target and an MoE target separately.

Qwen3.8-27B has 48 linear-attention and 16 full-attention layers. We evaluate all four benchmarks at C=4/R=193 and C=8/R=384. SG has positive throughput gains in all eight measured task/concurrency comparisons, with four-task geometric-mean gains of 1.45% and 4.42%, respectively.

Table 2. Throughput in the 27B dense-model experiment added for this rebuttal. SG has positive gains over the stated fixed-tier DDTree references in all eight measured configurations.

C     R    Task          SG tok/s    Fixed tok/s      Gain
---------------------------------------------------------
4   193    GSM8K             96.2           95.5    +0.75%
4   193    HumanEval        102.5           99.8    +2.68%
4   193    MATH-500          98.8           98.3    +0.48%
4   193    MBPP              65.4           64.2    +1.91%
8   384    GSM8K            127.4          123.7    +3.06%
8   384    HumanEval        137.0          134.6    +1.78%
8   384    MATH-500         122.4          115.0    +6.36%
8   384    MBPP              86.3           81.0    +6.55%

The native fixed references use 23-node trees at C=4 and 11-node trees at C=8. Each benchmark uses 128 prompts, temperature 1, and output cap 256. Throughput values are rounded for display, while gains are calculated from unrounded measurements. These references follow the native 27B protocol and are reported separately from independently tuned SGLang DDTree.

The MoE target, Qwen3-Coder-30B-A3B-Instruct, has approximately 30.5B total and 3.3B active parameters. Within SGLang, SG gives four-task geometric-mean gains of 3.37%/2.98% over independently tuned DDTree and 25.99%/23.54% over DFlash at C=4/8. These comparisons use R=193, 128 prompts per benchmark, temperature 1, and output cap 256. Aggregate improvements coexist with task-level variation, including −2.53% against tuned DDTree on C=8 MATH-500.

SpecGrove’s allocation interface uses each request’s eligible tree tiers, anchor-inclusive row costs, estimated progress, and a measured execution-cost curve. Adapting a target requires the corresponding model-execution support and model-specific calibration. The reusable-tier construction and exact-row allocation principle are retained. For the dense target, the implementation supports its hybrid-attention architecture; for the MoE target, the measured execution costs include expert-routed computation.

The two studies examine different aspects of model scale. The 27B dense experiment increases active model size relative to the original 4B/8B targets. The MoE experiment evaluates expert routing with a larger total parameter count, while reporting active parameters separately. Each gain compares methods within the same model and backend; the dense and MoE cohorts are reported separately.

Both studies use a single A100. The added results provide evidence for the capacity mechanism at the tested 27B dense and 30B-class MoE scales. Revised Section 5.4 and paper Table 4 summarize these results, and Appendix C.1 provides the execution protocols and checkpoint records.

Thank you again. We would be happy to discuss any further questions

Thanks for the response. I will keep my score since I already lean toward acceptance.

Thank you very much for your positive assessment and encouraging feedback. We sincerely appreciate your time and consideration throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.