acceptodds
Withdrawn from ICLR 2027

ReTree: Budget-Efficient Tree Speculative Decoding via Path Guidance and Sibling Recovery

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tree speculative decoding spends a fixed target-model budget on multiple candidate continuations, but exact verification can stop while scored siblings remain unused. We present ReTree, an approximate decoding policy that recovers an existing sibling using historical divergence counts, current target probability support, and stop-token constraints. Recovery reuses the same tree forward pass and introduces no learned correction head. A request-local path prior additionally changes which candidates enter the tree. On Qwen3-4B and Qwen3-8B at a 16-node budget, the measured speed gains over DDTree are 4.2–5.1% at temperature 0 and 8.8–9.1% at temperature 1. Recovery accounts for most of the gain; path guidance increases committed length but does not consistently improve speed over recovery alone. Additional CSD-style and MARS-style comparisons show that other relaxed verifiers can exceed ReTree's speed at some operating points. These comparisons do not establish quality-matched superiority. We characterize the contribution as tree-local reuse of already scored candidates, with an explicit speed–quality trade-off rather than preservation of the target distribution.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative Decoding, Large Language Model Inference, Tree Decoding, Efficient LLM Serving, Decoding Acceleration, Approximate Verification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LcvwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

ReTree achieves an approximate tree inference decoding strategy with zero additional inference overhead by reusing the already-scored sibling nodes when the exact match fails (based on frequency and probability gating).

Critical strengths

Precise and under-explored problem formulation. Rigorous and honest cost decomposition.

Critical weaknesses
  1. The writing is extremely poor, with numerous traces of AI-generated content. Phrases like "A rather than B", "not establish". The defensive writing style are also quite prominent. The overall writing is very unclear, obscure and difficult to understand.
  2. The text has already pointed out multiple times that authors are not performing lossless decoding, but only tested the accuracy rate on some mathematical tasks. Since authors have already tested so many datasets in Table 1, why don't report the accuracy?
  3. There are insufficient comparison methods, such as DFlash2 and DSpark. Only tests were conducted on the Qwen3 series, lacking comparisons among different series of models.
  4. The text does not explain the inference backend. From the algorithm design principle, the modification of nodes in ReTree has a natural "compatibility conflict" with the standard static CUDA Graph.
  5. Similarly, under a large batch size, whether ReTree brings benefits is also questionable.
  6. The author directly hard-code default hyperparameters (λ and ρ) in all main experiments, budget scans (Budget Sweep), and component ablation (Path and Recovery ablation in Table 2) throughout the entire paper. No any ablation experiments were provided to explain why chosen this.
  7. According to the explanation in Appendix B.3, if the generated text by the model cannot be extracted with regular expressions (\boxed{}), this sample will be directly excluded from the denominator. This evaluation method is totally incorrect.
  8. Finally, even without the above issues, the improvement brought by this article is negligible. In Table 3, when allowing an increase of 10% and 15% in the accepted length, the speeds of other methods all exceeded ReTree.
Questions for authors

Please see weakness. Before being accepted as a formal conference paper, this article needs to undergo significant revisions in its writing, methodology design, and experimental results.

Reviewer_iqvpOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

ReTree addresses the underutilization of already-scored sibling candidates in tree-based speculative decoding. It introduces a recovery policy based on historical divergence statistics and target-model probability constraints, alongside request-local path guidance. Experiments on Qwen3-4B and Qwen3-8B show consistent speed improvements over EAGLE-3, DFlash, and DDTree, while budget sweep experiments characterize the trade-off between tree construction cost and end-to-end decoding efficiency.

Critical strengths
  1. The paper addresses a meaningful inference-efficiency problem: already-scored sibling candidates may remain underutilized during tree-based speculative decoding, limiting the effective use of target-model computation.
  2. ReTree introduces a practical recovery policy combining historical divergence statistics and target-model probability constraints, alongside request-local path guidance, to increase committed length without additional target-model forward passes.
  3. Experiments show consistent speed improvements across multiple tasks, model sizes, decoding temperatures, and tree budgets. Ablations identify recovery as the main source of speed gains, while budget sweeps and latency decomposition characterize the trade-off between construction overhead and end-to-end efficiency.
Critical weaknesses
  1. The quality evaluation is limited to mathematical tasks, with no quality results reported for other datasets, such as code and chat. This makes it difficult to assess the impact of relaxed verification on output quality across different tasks.
  2. ReTree's relaxed verification criterion for newly recovered siblings is similar to those of CSD and MARS, as these methods also use historical correction statistics or target-model logits to guide token acceptance. The paper should more clearly distinguish the design differences and explain the specific advantages of ReTree's tree-local sibling recovery.
  3. Path guidance increases committed length when used alone but does not improve average speedup, and at larger tree budgets, full ReTree can be slower than Recovery alone. This suggests that Path is a budget-dependent auxiliary mechanism whose contribution to end-to-end efficiency remains limited.
  4. The sensitivity to recovery and path-guidance hyperparameters remains unclear, particularly for parameters such as the recovery thresholds such as ρ\rho and λ\lambda, and path-guidance coefficients such as β\beta and η\eta.
Questions for authors
  1. Could the authors report task-quality results on additional evaluated datasets, particularly code and chat, to clarify the impact of relaxed verification on output quality across different tasks?
  2. Could the authors clarify how ReTree's relaxed verification criterion for newly recovered siblings differs from those of CSD and MARS, and what specific advantages tree-local sibling recovery offers?
  3. Could the authors report sensitivity experiments for the recovery thresholds (ρ\rho and λ\lambda) and path-guidance coefficients (β\beta and η\eta), showing their effects on decoding efficiency and output quality?
Reviewer_2eoHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReTree, which combines approximate sibling recovery with request-local n-gram path guidance for tree speculative decoding. When exact following fails, recovery may commit an already-scored child using historical token-pair counts, target probability support, and stop-token checks. It adds no target forward pass during decoding. The method trades exact distribution preservation for additional committed tokens, making output quality central to its evaluation. On Qwen3-4B/8B at 16 nodes, reported speed gains over DDTree are 4.2–5.1% at temperature 0 and 8.8–9.1% at temperature 1.

Critical strengths
  1. Recovery reuses existing computation. A recovered child's descendants have already been scored under the correct ancestor context, allowing continued traversal without new nodes or another target pass. The pseudocode specifies cache handling and checks historical counts before recording the current mismatch.

  2. The timing analysis identifies the source of improvement. Tables 2, A1, and A3 separate recovery, path guidance, and round-level costs. Recovery accounts for most of the gain and costs 0.12–0.17 ms per round in the reported sweep, helping assess its net contribution.

  3. Recovery is tested with another tree constructor. Table A2 evaluates compatibility with DominoTree. This provides some evidence that the mechanism can be used beyond DDTree and increases its potential usefulness across existing tree decoders.

Critical weaknesses
  1. The quality cost remains unclear. Table 4 excludes unparseable answers but omits extraction coverage and effective denominators, so similar conditional scores need not imply similar full-set accuracy. Exact greedy baselines also disagree: DFlash and DDTree score 70.3% and 68.0% on MATH-500. Appendix B.3 leaves the cause unresolved. Code/chat quality and Qwen3-8B quality are unreported. These gaps prevent assessment of whether the approximate substitutions justify the speed gains.

  2. The benefit over existing relaxed verification is unestablished. The gates follow Calibrated Speculative Decoding (CSD). Table 3 includes CSD-style and MARS-style controls, but compares committed-length increases without matching quality and omits per-point gate settings. Some controls are faster, so the comparison does not establish an advantage for ReTree.

  3. Path guidance has limited independent support. At 16 nodes, it raises recovery-only speedup from 5.04× to 5.06×, approximately 0.4%, while reducing speed at 128 and 256 nodes. Without repeated timings, the small gain is insufficient to establish this component's practical value.

Questions for authors
  1. Clarification: What explains the exact greedy baseline differences? Please check matched examples, prompts, checkpoints, numerical settings, stopping rules, and scoring, and identify where generated tokens first diverge.

  2. Additional evaluation: Can you report common-set math accuracy, extraction coverage, and paired code/chat quality, including Qwen3-8B? How does ReTree compare with the relaxed controls at similar quality, with gate settings disclosed?

  3. Additional ablation: Does historical frequency improve the trade-off beyond target-support gating on the same tree? Please also compare recovery-only and full ReTree with repeated timings, since their 16-node speed difference is approximately 0.4%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.