- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents Solvo, a sampling-centric framework for tree speculative decoding. It allocates tree slots before sampling token identities, combines deterministic candidates with residual sampling, and introduces cost-aware tree growth. A causal verification analysis establishes preservation of the target distribution under the stated construction. Experiments across five target models and six tasks, together with component ablations and tuned-baseline comparisons, evaluate decoding throughput.
Critical strengths
- The separation of allocation from sampled token identity is technically valuable. It addresses a concrete source of selection bias, and the causal verification argument connects the tree-construction rule to distributional correctness.
- The shared-backend comparisons and cumulative ablations strengthen the empirical validation. The stochastic and greedy evaluations, including the additional TALON tuning comparison, give useful evidence that the construction improves an implemented decoder.
Critical weaknesses
- The main limitation is the demonstrated deployment scope. The throughput evidence uses one GPU platform, batch size one, and no concurrent requests; it does not yet establish how the cost-aware stopping rule behaves under serving contention or larger batches.
- The margin over the tuned TALON baseline is relatively small in greedy decoding. This makes sensitivity to the model-specific threshold and latency calibration important when judging how portable the improvement is.
Questions for authors
- How much throughput is lost when the threshold and stopping-cost calibration are fixed on a separate workload and transferred without retuning?
- Does the advantage over tuned TALON remain under concurrent requests or a larger serving batch?
Thank you for your review! We are glad to answer your questions.
Q1/W2. How much throughput is lost when the threshold and stopping-cost calibration are fixed on a separate workload and transferred without retuning?
In all our experiments, the stopping cost is calibrated on only 4 Alpaca prompts and then frozen across all six datasets (480 prompts). For θ, the value selected on MT-Bench-101, an independent dataset not among the six in our paper, matches the paper setting (see our response to Reviewer wLg1). This also addresses the single-GPU-platform concern in W1: we further test transfer from the RTX PRO 6000 to an RTX 3090.
Using Llama3.1-8B-Instruct, we transfer the threshold (θ = 0.01) and the stopping-cost calibration (Ld = 0.717 ms, Lb = 22.947 ms) obtained on the RTX PRO 6000 directly to an RTX 3090, and compare with recalibrating on the RTX 3090 (Ld = 1.298 ms, Lb = 30.427 ms), keeping θ = 0.01. All six datasets (480 prompts) are used.
The difference is negligible.
For θ, we use the recalibrated Ld/Lb and re-sweep θ on all six datasets (480 prompts) on the RTX 3090, alongside the RTX PRO 6000 results:
The optimal θ remains 0.01, and the trend over θ is essentially the same on both GPUs.
In summary:
Q2. Does the advantage over tuned TALON remain under concurrent requests or a larger serving batch?
Yes, it does.
We ran continuous serving with SGLang using Llama3.1-8B on MT-Bench. "Batch" denotes the running batch size. TPS:
TALON uses μ = 0.03, the single-request optimum from our sweep in Table 9. Solvo reuses the parameters from the paper without any change.
Since TALON's optimal μ may shift with batch size, we additionally swept μ at batch = 8:
At T=1, TALON's optimal μ is still 0.03. At T=0 the optimum is 0.05, only marginally above 0.03. Even with this optimum, TALON reaches 1265.20, still below Solvo's 1297.55.
W1. How does the cost-aware stopping rule behave under serving contention or larger batches?
Thank you for this insightful question.
Under continuous serving, the stopping decision can take the other requests in the batch into account. For example, if most requests have already been stopped by the per-request rule while a few continue drafting, the additional draft forward may no longer pay off for the batch as a whole.
A natural extension for Solvo is therefore to continue drafting only if another layer improves the speedup of the whole batch. This yields further gains: