- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper jointly optimizes speculative draft trees and the policy that decides when to stop drafting and how many candidates to verify. It alternates offline tree search with policy updates and uses sampled execution feedback to correct its performance estimates. Overall, the paper is a valid, incremental contribution to the community, but I still have several concerns on the novelty and other aspects.
Critical strengths
- Optimizing actual decoding time rather than acceptance length alone is a good direction.
- I appreciate the effort to separate the gains from changing the tree and adapting the controller. The ablations make the contribution easier to assess.
- Testing several model families and hardware platforms is useful, especially for a method that depends on runtime costs.
Critical weaknesses
-
I am not fully convinced by the novelty of the paper. For example, adaptive tree construction and draft stopping are already well-studied in the prior works. Jointly optimizing them is reasonable and expected, but the paper needs to make clearer what this adds beyond combining these existing ideas (A+B). The gains over Learning to Draft are sometimes only a few percent, which makes the added complexity harder to justify.
-
The serving relevance remains unclear in the manuscript. Dynamic trees are difficult to support efficiently at high concurrency, and the evaluation is mainly batch size 1. How would the controller's variable stopping points and verification budgets work with continuous batching and CUDA graphs in SGLang or vLLM? Without an implementation or measurements in such a setting, I find it hard to judge whether the gains would survive production deployment.
-
The offline optimization pipeline also needs a clearer cost breakdown. Each deployment configuration requires traces, latency fitting, policy updates, and integer optimization. How much wall-clock time and GPU compute does this take, and what needs to be repeated when the workload or hardware changes? Reporting normalized feedback budgets alone does not tell me whether the relatively modest serving gains repay the setup cost.
Questions for authors
Please see weaknesses section above.
Though the authors have provided a rebuttal, these issues seem to persist and I recommend the authors to reposition especially the novelty of this paper.
6 comments
oCuF · Response 2
W1
W1“The gains over Learning to Draft are sometimes only a few percent … the added complexity harder to justify.”
R2-Key Point: Yes—Main Table 1 disables CUDA Graphs for algorithmic parity; same-method runtime optimizations add \small\colorbox{#244F78}{\color{white}\textbf{+2.54\\% to +9.07\\%}} in Table H1, our solved static template embeds over 20% posterior decision headroom in a controlled fixed-template analysis (Table N2), and all solver complexity stays strictly offline.
Our detailed evidence on hardware potential, decision headroom, and online simplicity follows below:
- \rule{0pt}{2.52em}\color{#315B75}\textsf{R2-1: Algorithmic Gains \\& Same-Method Engineering Potential (Table H1):}
With CUDA Graphs disabled in Main Table 1 (), OGSPO improves 4-task mean throughput over LTD by +5.13% (: ) and +8.84% (: ) on H800 (and up to +19.3% over native EAGLE-3 across 12 settings), while controlled ablations across 6 paired groups (Fig. 3a) show policy adaptation on the new structure nearly doubles the median structural gain (, vs. for equal extra training on the old tree). Because OGSPO's shared nested template fixes candidate layouts offline, it supports pre-compiled indexing, cache reuse, and CUDA Graphs. Table H1 isolates these same-method engineering gains (holding token outputs and decisions identical before vs. after optimization): capturing the MLP actor in CUDA Graphs cuts per-call latency from to (+2.54%), combined runtime optimization on the 4-stage nested policy adds +9.07% across 400 requests, and on the 48-node system adds +3.40%. - \rule{0pt}{2.52em}\color{#315B75}\textsf{R2-2: Over 20\\% Latent Decision Headroom Within Solved Static Templates:}
In a controlled decision-accuracy analysis on 419 sampled HumanEval states (, 4-exit nested template after the third structural update), 100% posterior action selection within the finite menu exceeds the best fixed exit/pruning combination in that tree by \small\colorbox{#EAF0F7}{+24.08\\% in statewise local rate} () and +26.87% in global progress-to-time ratio (). As stop/continue decision accuracy scales to 80%, 90%, and 95%, the same static template yields +13.66%, +18.70%, and +21.34% local rate gains (Table N2).
While offline optimization is rigorous, its preparation costs include data collection as well as fitting (R4, Table S1), whereas online inference requires only a lightweight controller—either the 512K-parameter MLP (/call with CUDA Graph capture) in our route or a compact regression tree in our route—acting on a pre-solved static nested structure.
Table H1: Same-Method Runtime Improvements on H800 (Holding Decisions & Outputs Identical)
| \bbox[#E9E5EF]{\small\textbf{Optimization}} | \bbox[#E9E5EF]{\small\textbf{Evaluation Scope}} | \bbox[#E9E5EF]{\small\textbf{Before (tokens/s)}} | \bbox[#E9E5EF]{\small\textbf{After (tokens/s)}} | \bbox[#E9E5EF]{\small\textbf{Gain}} |
|---|---|---|---|---|
| Controller CUDA Graph capture () | Single-request controller () | 142.44 | 146.06 | +2.54% |
| Combined runtime optimization () | Five held-out tasks (400 requests) | 100.03 | 109.10 | +9.07% |
| Verification packing () | ShareGPT development requests (cap 512) | 232.98 | 240.90 | +3.40% |
Table N2: Local Decision Headroom Within Solved Static Nested Templates (, H800, )
| \bbox[#E9E5EF]{\small\textbf{Template \\& Diagnostic Scope}} | \bbox[#E9E5EF]{\small\textbf{Baseline Reference TPS}} | \bbox[#E9E5EF]{\small\textbf{@ 80\\% Acc.}} | \bbox[#E9E5EF]{\small\textbf{@ 90\\% Acc.}} | \bbox[#E9E5EF]{\small\textbf{@ 95\\% Acc.}} | \bbox[#E9E5EF]{\small\textbf{@ 100\\% Posterior Ceiling}} |
|---|---|---|---|---|---|
| 4-Exit Template (419 States, Stop Acc.) | 118.96 (Best fixed exit) | 135.20 (+13.66%) | 141.21 (+18.70%) | 144.35 (+21.34%) | 147.60 (+24.08% local / +26.87% ratio) |
| 4-Exit Template (419 States, Stop+Prune Acc.) | 118.96 (Best fixed exit) | 129.94 (+9.23%) | 138.21 (+16.18%) | 142.76 (+20.01%) | 147.60 (+24.08%) |
oCuF · Response 3&4
W2)
W2“the controller's variable stopping points and verification budgets work with continuous batching and CUDA graphs in SGLang or vLLM?”
R3-Key Point: Because OGSPO fixes its nested stage topology offline, it naturally supports CUDA Graph capture and verification packing (+3.40% at ), while improving held-out ShareGPT throughput by \small\colorbox{#244F78}{\color{white}\textbf{+46.11\\%}} and +25.52% even under a basic shared-stop batch executor (Table B1).
Serving details and results follow:
Adapting to a batched executor recalibrates , solves a batch-specific tree, and trains the controller; existing acceptance records remain .
In our measured implementation, the batch continues drafting or enters verification together, while each request maintains separate candidate tokens, accepted lengths, masks, KV caches, and retained budgets. The batch-adapted controller stops at the first decision point in 74.60% of cycles. Independent stopping can instead group stopped requests in a verification queue while accounting for waiting and packing costs in .- \rule{0pt}{2.52em}\color{#315B75}\textsf{CUDA Graphs \\& Serving-Engine Integration:}
Fixing the tree topology offline makes its stage layouts amenable to pre-compilation, , and CUDA Graph capture ( actor capture at ). Integrating with continuous-batching engines (SGLang/vLLM) additionally requires the corresponding queue/KV-cache adapter and measurement of its scheduling costs.
Table B1: Batched Complete-Call Throughput on H800 at (Llama-3.1-8B-Instruct/EAGLE-3, )
| \bbox[#E9E5EF]{\small\textbf{Evaluation / EAGLE-3 baseline}} | \bbox[#E9E5EF]{\small\textbf{EAGLE-3 → OGSPO (tokens/s)}} | \bbox[#E9E5EF]{\small\textbf{Gain}} |
|---|---|---|
| ShareGPT, 768-token generation limit; Training-selected depth-3 tree | 152.52 → 222.83 | +46.11% |
| ShareGPT, 1,024-token generation limit; Training-selected depth-8 tree | 103.95 → 130.48 | +25.52% |
W3)
W3“How much wall-clock time and GPU compute does this take … what needs to be repeated when the workload or hardware changes?”
R4-Key Point: Data collection is a shared investment; policy fitting on existing records (23.53 s at ; 0.56–3.42 s at ). The frozen pair transfers across evaluated datasets without per-dataset fitting; hardware or batching shifts require timing calibration and tree–policy updates while reusing acceptance records (Table S1).
Costs and reuse rules follow:
W3):
- Workload Shifts (0 s Retuning): A frozen pair transfers directly across the evaluated datasets without per-dataset fitting, serving the four main tasks and summarization extensions at . At , the ShareGPT-trained pair gains +22.25% on summarization and +9.70% on MBPP against the ShareGPT-selected EAGLE-3 depth-8 baseline. Performance depends on workload and baseline.
- Batch / Hardware Shifts (Table S1): Hardware or batch-execution changes require new timing calibration and tree–policy updates; acceptance records remain reusable for the same model pair and decoding setting.
Table S1: Logged Preparation and Update Costs ( and Routes, H800 + CPU)
| \bbox[#E9E5EF]{\small\textbf{Stage}} | \bbox[#E9E5EF]{\small\textbf{Time}} | \bbox[#E9E5EF]{\small\textbf{When needed}} |
|---|---|---|
| Initial batched timing-data collection () | 2.65 h cumulative worker time | Batch executor timing coverage |
| Shared acceptance/teacher-data collection () | 18.34 min | Reusable acceptance records |
| Latency-model fitting () | 18.55 s | Fit to measured execution costs |
| Batched tree search () | 11.06 min | Update the structure for changed costs or policy |
| Decision-position evaluation () | 5.38 min | Update decision positions |
| Controller fitting & cross-validation () | 23.53 s | Fit policy on existing batch records |
| route: shared init / feedback / policy refit | 1.90 h / 135.1–262.9 s / 0.56–3.42 s | MLP init / 405-run residual job / panel refit |
The timing-collection figure is cumulative worker time, including GPU execution.
oCuF · Response 1
We thank Reviewer oCuF for recognizing our focus on decoding time over acceptance length, our tree/controller ablations, and our cross-platform evaluations.
Response 1: What Does OGSPO Fundamentally Add Beyond "Adaptive Tree Construction" and Draft Stopping? (
W1— Part 1)R1-Key Point: OGSPO is not runtime heuristic "adaptive tree construction": it solves a static, statistically throughput-optimal shared nested template offline and uses delayed-decision nested optimization (U0/U2) plus counterfactual residual-corrected policy learning (U1) to unlock the posterior theoretical ceiling that isolated pre-drafting portfolios fail to release.
Details follow:
Honestly, we disagree with categorizing OGSPO as "adaptive tree construction" or viewing it as a combination of existing heuristics (
W1).Using a calibrated latency predictor and integer programming to solve for statistically optimal static trees or portfolios is our starting point (§3.2, App. C.4), explicitly calibrating the posterior theoretical throughput ceiling and statistically optimal tree portfolios in a finite candidate domain.
However, our preliminary study (§3.2, Fig. 1 & App. D) reveals a severe capability bottleneck when static portfolios are used in isolation: the best held-out selector recovered only 2.0% of the posterior portfolio gain (vs. 50%–76% with verification-informed selection).
Anchoring our formulation on an explicit posterior optimality ceiling fundamentally separates OGSPO from methods that tweak local expansion heuristics or restricted template families. OGSPO bridges the gap to this ceiling via two coupled mechanisms: