- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CalTree separates sibling ranking from allocating a speculative tree’s node budget across branches and depths. A 29-parameter logistic model predicts conditional acceptance using retrieval and request features, and path probabilities guide tree construction. Experiments span five target models, with modest SGLang throughput gains over native SSSD.
Critical strengths
-
The paper clearly motivates the distinction between sibling ranking and tree allocation.
-
The lightweight model uses retrieval, repetition, and long-range-match features without target weights or hidden states, fitting the existing framework.
Critical weaknesses
-
The advantage over simpler scoring is unclear. Tuned discounts recover 24–53% of CalTree-LR’s replay gains but are omitted from serving comparisons. CalTree-LR also raises tree-building time from 64–112 to 119–213 μs. End-to-end comparisons are needed.
-
Output comparability is not fully established: only 83–90% and 89–94% of speculative outputs exactly match AR. Existing checks do not resolve full-workload quality or timing effects, which matter given the 1–3% throughput gains, though they do not imply sampling bias.
-
Fixed-budget gains may not persist under tuned serving. SSSD’s adaptive draft size is disabled, budgets are set by concurrency, and some high-concurrency settings fail to beat AR.
Questions for authors
-
How does CalTree compare with tuned discounts in SGLang, including all overhead?
-
Can full-workload quality checks clarify output differences and timing effects?
-
Do gains persist with tuned or adaptive budgets, and when is AR preferable?
3 comments
Response to Reviewer gADs (part 2 of 2)
Q3, tuned or adaptive budgets, and when AR is preferable. The gains persist under both, on both targets (Appendices L and O).
-
Tuned budgets (Tables 17 and 23). Sweeping D = 4 to 32 at batch sizes 1 and 8, CalTree-LR is faster than SSSD at every budget: by 1.0 to 3.0% on Qwen2.5-7B and 1.0 to 3.4% on Llama-3.1-8B, where every run is faster than every run of SSSD.
-
Adaptive draft size (Tables 18 and 24). With SSSD's adaptive draft size enabled (tuned discounts then use the factors for the current budget), CalTree-LR is faster than SSSD at batch sizes 1 to 32, by 1.0 to 2.1% on Qwen2.5-7B and 1.1 to 2.8% on Llama-3.1-8B, and changes throughput by +0.2 to +0.8% and −0.2 to +1.9% relative to the tuned discounts.
-
Choosing the budget and the scores add up. SSSD's default budgets are too large at batch sizes 8 and 16:
Target, batch size SSSD, default D SSSD, fastest D CalTree-LR at that D Qwen2.5-7B, 8 0.99x AR (D = 16) 1.22x (D = 4) 1.24x, 1.1% above tuned discounts Llama-3.1-8B, 8 1.05x (D = 16) 1.20x (D = 8) 1.23x, 1.1% above tuned discounts Qwen2.5-7B, 16 0.94x (D = 8) 1.07x (D = 4) 1.08x, 1.4% above SSSD Llama-3.1-8B, 16 0.97x (D = 8) 1.04x (D = 4) 1.06x, 1.6% above SSSD -
When AR is preferable. At batch size 32 on our single-GPU setup, no budget from 2 to 4 reaches AR (at most 0.90x, Table 25), so AR is the better choice there. With single-token drafts (D = 2) CalTree-LR has almost nothing to allocate and is 0.1 to 1.4% slower than SSSD; at D = 3 it changes throughput by −0.1 to +0.9%.
The revised paper states this in Section 4.6 and the limitations: CalTree improves allocation within a budget and does not replace choosing the budget, and its gain adds to a well-chosen budget.
Follow-up for Reviewer gADs
Thank you again for the review. A short index of the experiments added for your three questions:
- Q1, tuned discounts in SGLang with all overhead (Tables 3, 16, 19, 26, 27). On code editing and HumanEval CalTree-LR is faster than the tuned discounts in every run on both targets (by 1.2 to 3.1% and 0.9 to 1.5%); on chat, where the discounts are tuned, they recover a median 62% of its gain and CalTree-LR is ahead of them at 7 of 8 loads.
- Q2, full-workload quality and timing (Table 15). CalTree-LR's GSM8K and RAG scores equal AR's in every run, CanItEdit hidden-test pass rates are identical, and HumanEval pass rates differ from AR's by at most one problem, alike for every speculative method; the tuned discounts diverge from SSSD as often as CalTree-LR does (87.9% vs 87.1% agreement); CalTree-LR's output token total at batch size 8 is within 0.03% of AR's.
- Q3, tuned and adaptive budgets, and when AR is preferable (Tables 17, 18, 23 to 25). CalTree-LR is faster than SSSD at every budget from 4 to 32 and with adaptive draft sizes, on both targets. A smaller budget gives CalTree-LR 1.23x to 1.24x AR at batch size 8 and 1.06x to 1.08x at 16; at batch size 32 AR is faster.
If any of your questions remains open, we would be glad to address it before the discussion closes, and we hope the new results can be reflected in your assessment.
Response to Reviewer gADs (part 1 of 2)
Thank you for the clear questions; each led to a new experiment.
Q1, CalTree versus tuned discounts in SGLang, all overhead included. The discounts run inside SSSD's C++ tree builder, and every serving table reruns all methods alternating within the same runs. They are a strong control: seven source/depth factors searched for τ on the same replays CalTree learns from, separately at each budget D = 4 to 32 (Appendix D). In throughput:
Q2, full-workload quality and timing.