- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Request Forest, which jointly allocates verification nodes across concurrent requests and candidate branches using DFlash2’s candidates and path scores. It combines shared-budget tree construction with branch-isolated verification, recurrent-state commitment, and load-adaptive switching. The authors prove optimality for a fixed-score, unit-cost surrogate objective and evaluate four hybrid models on four benchmarks, reporting higher throughput and lower intermediate verification-state memory.
Critical strengths
-
The paper addresses a practical bottleneck in high-concurrency speculative decoding. Reusing an existing drafter without retraining and supporting branch-specific recurrent states make the approach relevant to hybrid-model serving.
-
The component ablations provide useful evidence. With the drafter, budget, and state reservation controlled, budget sharing improves throughput by about 7%, and allowing branches adds about 5% on the two ablated models.
Critical weaknesses
-
The additional benefit over existing scheduling methods is not yet convincing. DSpark already globally ranks prefix extensions by cumulative survival probability and selects verification lengths using hardware costs. ECHO also combines cross-request allocation with tree depth and width control. Both are discussed but neither is evaluated. The ablation also lacks independently optimized trees with fixed per-request budgets, so it does not directly establish the benefit of joint forest allocation over separate tree construction.
-
Output preservation needs clearer validation. Equation (7) specifies equality with target-only greedy decoding, but Table 20 reports different task results, including 1226 versus 1222 correct GSM8K answers for target-only and RF. Numerical differences may explain this, but no token-level agreement or divergence analysis is reported. Similar task accuracy does not establish output equivalence, and differences in generated lengths could also affect throughput comparisons.
Questions for authors
-
Can you compare RF with DSpark- or ECHO-style scheduling and with fixed-budget per-request trees under controlled drafter, execution-backend, and verification-resource settings? Please distinguish adaptations of their scheduling policies from comparisons against the full systems.
-
What causes the differences in Table 20? What are the token-level output-agreement rates against target-only decoding, and do differences in output length or truncation materially affect the throughput results?
6 comments
Response to Reviewer Skdv
Q2(b): Do output-length or truncation differences materially affect throughput?
Response:
We agree that identical output limits do not guarantee identical generated lengths. We therefore compare RF and DFlash2 across 27 matched configurations: three benchmarks, batch sizes 128, 144, and 160, and output limits of 128, 256, and 512 tokens. We record end-to-end throughput and the fraction of requests reaching each limit.
RF achieves higher throughput in all 27 configurations, with paired geometric-mean gains of 13.19%, 15.45%, and 15.19% at the three output limits, respectively. The maximum difference in limit-hit rates is only 1.5 percentage points. For example, at B = 128 on HumanEval with a 512-token limit, both methods have a 56.4% limit-hit rate, yet RF improves throughput by 21.95%. At B = 144 on MATH-500 with a 128-token limit, both reach 98.4%, while RF remains 15.54% faster.
These results show that RF’s gains persist across output limits and cannot be explained solely by different limit-hit rates. However, each configuration was measured once, and similar limit-hit rates do not establish identical per-request output lengths. We therefore cannot fully exclude effects from output-length distributions and will clarify this limitation.
Table R3. Throughput and limit-hit rates across maximum output lengths.
| Batch | Benchmark | Method | Limit 128 | Limit 256 | Limit 512 |
|---|---|---|---|---|---|
| 128 | GSM8K | RF | 6684.89 (74.2%) | 6979.36 (12.5%) | 6951.15 (0.2%) |
| 128 | GSM8K | DFlash2 | 5930.43 (74.6%) | 5971.38 (11.3%) | 6193.11 (0.2%) |
| 128 | MATH-500 | RF | 6467.50 (98.6%) | 7384.94 (69.1%) | 7891.26 (24.6%) |
| 128 | MATH-500 | DFlash2 | 5471.22 (98.0%) | 6417.79 (69.1%) | 6592.28 (25.4%) |
| 128 | HumanEval | RF | 6317.42 (97.1%) | 7692.61 (87.5%) | 8487.25 (56.4%) |
| 128 | HumanEval | DFlash2 | 5466.39 (97.3%) | 6409.63 (87.7%) | 6959.61 (56.4%) |
| 144 | GSM8K | RF | 7006.84 (74.0%) | 7341.02 (13.0%) | 7175.55 (0.0%) |
| 144 | GSM8K | DFlash2 | 6352.89 (75.5%) | 6477.58 (12.7%) | 6520.98 (0.2%) |
| 144 | MATH-500 | RF | 6750.79 (98.4%) | 7624.68 (67.5%) | 8191.74 (26.0%) |
| 144 | MATH-500 | DFlash2 | 5842.66 (98.4%) | 6675.17 (67.7%) | 7091.06 (24.8%) |
| 144 | HumanEval | RF | 6526.36 (96.7%) | 8209.06 (90.5%) | 8931.45 (57.6%) |
| 144 | HumanEval | DFlash2 | 5763.08 (96.9%) | 6649.93 (89.1%) | 7389.34 (57.8%) |
| 160 | GSM8K | RF | 7169.11 (74.7%) | 7556.95 (12.8%) | 7164.94 (0.3%) |
| 160 | GSM8K | DFlash2 | 6568.06 (74.7%) | 6813.74 (13.3%) | 6729.82 (0.0%) |
| 160 | MATH-500 | RF | 6962.62 (97.8%) | 7997.49 (68.1%) | 8266.92 (25.3%) |
| 160 | MATH-500 | DFlash2 | 6204.47 (98.8%) | 7102.43 (68.4%) | 7252.67 (24.7%) |
| 160 | HumanEval | RF | 6693.52 (96.7%) | 8059.09 (88.6%) | 9020.75 (57.7%) |
| 160 | HumanEval | DFlash2 | 5973.93 (97.2%) | 6967.36 (88.0%) | 7718.10 (57.7%) |
Each entry reports throughput in tokens/s (percentage of requests reaching the output limit). Output limits are measured in tokens.
▪ We thank the reviewer for the constructive feedback. The scheduling comparisons in Weakness 1 are addressed in Q1, with the fixed-budget independent-tree comparison provided separately below. Weakness 2 is discussed in Q2(a) regarding output differences and the scope of equivalence, and Q2(b) regarding output-length and truncation effects on throughput. Any remaining concerns are addressed separately to avoid repetition.
Weakness: Benefits of joint forest allocation over independent trees
Response:
The original Table 2 shows that Global-Forest (GF) improves throughput over Global-Chain (GC) by approximately 5.0%, demonstrating the benefit of branching under a shared budget. To test whether budget sharing remains beneficial when branching is already enabled, we add Equal-Forest (EF).
EF and GF use identical candidate trees, drafter, scoring rules, total verification budget, and execution backend. EF assigns fixed per-request quotas and selects trees independently; GF jointly allocates the budget across requests and branches.
Table R4. Fixed-budget independent trees versus shared-budget forests (throughput: tokens/s).
| Configuration | Qwen3.8-27B ↑ | Ornith-1.5-35B-A3B ↑ |
|---|---|---|
| Equal-Chain (EC) | 2135.3 | 3860.8 |
| Global-Chain (GC) | 2285.4 | 4130.7 |
| Equal-Forest (EF) | 2270.1 | 4108.9 |
| Global-Forest (GF) | 2398.8 | 4335.7 |
GF outperforms EF by 5.67% on Qwen and 5.52% on Ornith. Together, GC → GF isolates the benefit of branching, while EF → GF isolates the benefit of cross-request budget sharing when branching is already available. These comparisons demonstrate the additional value of joint forest allocation under matched verification resources.
Follow-up on Your Comments
Dear Reviewer Skdv,
Thank you for highlighting the need for stronger controlled comparisons and clearer output-preservation claims. We have added ECHO-style and DSpark-style scheduling comparisons under a common backend, together with the fixed-budget independent-tree baseline. We have also examined throughput and truncation rates across different output limits and clarified the distinction between ideal-arithmetic equivalence and finite-precision behavior in our revisions. Could you kindly let us know whether these experiments and clarifications address your main concerns, or if there are any points you would like us to clarify further? Thank you again for your time and helpful suggestions.
Best regards,
The authors
Response to Reviewer Skdv
Q1: Controlled comparisons with ECHO-style and DSpark-style scheduling
Response:
We thank the reviewer for suggesting these comparisons. ECHO and DSpark were not originally designed for diffusion language models, whereas our work focuses on scheduling for diffusion language models. This difference in scope motivated our original evaluation design, which did not include comparisons with ECHO or DSpark. Nevertheless, we agree that comparing against adaptations of their scheduling policies would help clarify RF’s additional benefits.
In response to your suggestion, We adapt ECHO-style and DSpark-style scheduling to the same Ornith target model, DFlash2 drafter, and verification backend, with matched verification-resource constraints.
ECHO-style uses confidence-gated, depth-first expansion with width extension; DSpark-style selects verification size using prefix-survival estimates and hardware execution costs. RF instead jointly selects path extensions and alternative branches across requests through a global candidate frontier ranked by cumulative path priorities.
Table R1. Throughput under controlled scheduling comparisons on Ornith with DFlash2 (tokens/s).
RF achieves higher throughput in all nine configurations, with a paired geometric-mean gain of 3.02% over ECHO-style across all nine, and 6.58% over DSpark-style across the six configurations at B = 144 and 160.
These results demonstrate RF’s incremental benefit under the tested settings. The comparisons evaluate adaptations of scheduling policies to a common backend, rather than reproductions of the complete original systems.
Q2(a): Why do the task results differ, and are the generated outputs identical?
Response:
Eq. (7) describes ideal-arithmetic equivalence, assuming identical prefixes, state-update semantics, greedy decoding rules, and stopping criteria. Our finite-precision implementation does not guarantee token-for-token identical outputs.
Different computation layouts and execution paths can introduce numerical differences. When token scores are close, these differences can change greedy decisions and subsequent generation, potentially affecting task accuracy. We will revise Eq. (7) and the accompanying text to explicitly delimit the equivalence claim.
As a supplementary quality check, RF’s perplexity differs from target-only autoregressive decoding (AR) by only +0.0108% (Table R2), indicating similar aggregate predictive quality.
Table R2. Perplexity under the evaluation settings.
PPL: perplexity. Change vs. AR: relative PPL change compared with target-only autoregressive decoding.