acceptodds
Under review as a conference paper at ICLR 2027

AdaptSpec: Input- and Step-Adaptive Tree-Based Speculative Decoding to Accelerate LLM Inference

Abstract

Speculative decoding accelerates Large Language Model (LLM) inference by using a draft model to propose a speculative length (SL) of tokens at each decoding step, which a target model then verifies in a single forward pass. Tree-based speculative decoding further improves acceptance rates and speedup by drafting a candidate length (CL) of tokens in parallel at each position. However, these methods rely on fixed SL and CL. While recent work adapts SL primarily using step-level signals, prior methods do not jointly optimize SL and CL while conditioning on the request input, despite its significant impact on optimal parameter selection. Our experimental analysis identifies several influential factors, including input categories, that determine the speedup-maximizing configuration. Accordingly, we propose AdaptSpec, which initially adapts SL using machine learning and CL using binary search based on these factors, and subsequently switches to Reinforcement Learning (RL) to jointly optimize both parameters once the RL policy is trained. Experiments across four draft/target model pairs and five datasets show that AdaptSpec achieves up to 98% higher speedup than state-of-the-art methods, translating into fewer GPU-hours per request in LLM serving.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Speculative Decoding, Large Language Model Inference, Inference Acceleration, Reinforcement Learning, Adaptive Inference, Efficient LLM Serving

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KWc9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaptSpec, which employs reinforcement learning (RL) to jointly optimize both SL and CL. The experimental analysis identifies several influential factors, including input categories, that affect the configuration yielding the highest speedup. The authors conduct experiments across four draft–target model pairs and five datasets.

Critical strengths

The paper provides an experimental analysis of several potentially useful signals, such as logit divergence and step-wise acceptance ratios, which could help guide adaptive decisions and improve the accepted length.

Critical weaknesses

1 Insufficient coverage of related work and outdated baselines. The paper lacks citations to several important and closely related studies, and the baselines used for comparison are relatively outdated. SpecInfer, which employs tree-based speculative decoding, is an early work in this area. However, EAGLE-2[1], an important baseline for dynamic tree-based speculative decoding, is only briefly mentioned in the Related Work section and is not included in the experimental comparisons.

Furthermore, numerous existing studies have explored dynamically adapting speculative decoding configurations based on different input signals, some of which are closely related to those considered in this paper. For example, previous methods have investigated entropy [2] and average acceptance rates [3] as signals for adaptive decisions, but these relevant works are not adequately discussed or cited.

In addition, several prior studies have explored reinforcement learning for optimizing speculative decoding and tree construction[4], yet these methods are neither sufficiently discussed nor compared against. The authors should provide a more comprehensive discussion of related work and include stronger, more recent baselines to better demonstrate the novelty and effectiveness of the proposed approach.

[1] EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees

[2] Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models.

[3] Dynamic Depth Decoding: Faster Speculative Decoding for LLMs.

[4] Learning To Draft: Adaptive Speculative Decoding with Reinforcement Learning.

2 Lack of ablation studies on the reinforcement learning design. The paper does not provide sufficient ablation studies to justify its reinforcement learning framework. For example, the reward function design is not thoroughly investigated, and comparisons with alternative reward formulations are missing. These ablations would help clarify the effectiveness of the proposed RL framework and the contributions of its individual design choices.

Questions for authors

Please refer to the questions raised in the Weaknesses section.

Reviewer_ypmsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies tree-based speculative decoding. The authors argue that the best speculative length (SL) and candidate length (CL) depend on the input category, the decoding step, and the acceptance ratio and draft-target KL divergence of recent steps. They propose AdaptSpec. A linear head on the draft model predicts SL. A random-forest speedup estimator guides a binary search over CL. Once enough data is collected, an actor-critic policy takes over and chooses both. On four draft/target pairs and five datasets, the authors report up to 98% higher speedup than vanilla speculative decoding, with 0.4% decision overhead.

Critical strengths

The problem setting is worth studying. The paper treats SL and CL as one per-step decision that can depend on the input. The staged controller, which starts from a supervised predictor and binary search and switches to RL after enough data is collected, is a practical way to avoid a cold start. I credit the setting more than the execution, for the reasons in Weakness 1.

The evaluation covers four model pairs, five datasets and three seeds, and the reported decision overhead includes the request classifier.

Critical weaknesses

The two claims that make this work new are not isolated by experiments. First, "input-adaptive": no ablation removes the request-type feature. The ablations only remove SL prediction, CL search, or RL (Section 5.1). The SL head already sees the draft hidden state, so it is unclear what the category adds. Second, "joint": before RL, SL and CL are chosen one after the other, SL first and then CL given SL. The RL actor outputs both through a two-dimensional Gaussian. The only evidence that RL captures SL-CL interactions is the 12-19% gain over AdaptSpec/RL, and the paper does not show where this gain comes from. This matters to my rating because these two claims are the paper's stated contribution over prior adaptive methods.

The methods that overlap most with this work are not compared. EAGLE-2 and OPT-Tree also adapt the draft tree at each step, and AdaEDL adapts the draft length from draft confidence. The paper cites them in Related Work, but the baselines are Vanilla, SpecInfer, OSD and SpecDec++, and SpecDec++ is evaluated only on MTBench. Without these comparisons, the 46-67% gains over OSD and SpecDec++ do not show that AdaptSpec improves on current adaptive methods.

Questions for authors

Please ablate the request-type feature. How much speedup is lost without it, and how are categories assigned outside MTBench?

What does the RL policy learn about SL-CL interaction that the sequential SL-then-CL pipeline misses? Would a better SL predictor alone give the same 12-19% gain?

Can you compare against EAGLE-2, OPT-Tree, or AdaEDL on at least one model pair, and report SpecDec++ beyond MTBench?

Section 2 describes accepting tokens within a probability threshold (e.g., 0.05). Which acceptance rule is implemented, and how does it guarantee outputs identical to standard decoding (Section 5.2)?

Reviewer_AymYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To move beyond static or coarse-grained SL/CL adaptation under diverse inputs and decoding stages, AdaptSpec operates in a same-family draft--target, multi-GPU split layout. It uses draft hidden states, request type, step index, and previous-3-step average acceptance ratio and KL divergence to drive a lightweight SL predictor; it combines a random forest with binary search for CL selection; and it uses an actor--critic policy that jointly chooses SL/CL, pretrained offline and retrained online every 500 requests. Overall, AdaptSpec is a system integration and engineering optimization built on prior dynamic SL/CL and RL scheduling work, with limited methodological novelty; its low-overhead step-level joint integer decision has some system value, but boundary baselines---request-level adaptation, SL-only adaptation, and a simple ”SL prediction + CL early stopping/selection“ combination---are still needed to demonstrate a non-substitutable advantage.

Critical strengths
  1. The paper incorporates request type, step index, historical acceptance ratio, draft--target KL divergence, and other features into step-level speculative length (SL) prediction, combines a random forest (RF) estimator with binary search for candidate length (CL), and uses an actor--critic policy for joint selection, forming an end-to-end adaptive pipeline. It reports substantial speedups over Vanilla, SpecInfer, and online speculative decoding (OSD) across multiple same-family model pairs and datasets, while all speculative tokens are verified by the same target model, preserving outputs identical to standard decoding, which indicates practical serving value.

  2. The paper reports about 0.4% end-to-end (E2E) decision overhead for AdaptSpec and 2.8% for AdaptSpec/RL, and states that RL reduces decision overhead by 3.22$\times$. Ablation studies on SL, CL, RL, and knowledge distillation (KD) provide initial support that each component contributes to the overall speedup, which positively affects the acceptance judgment.

Critical weaknesses
  1. In Section 4, the paper uses “captures their interactions'' as the core justification for unifying SL/CL with RL, but never provides a formal definition of "interaction" nor distinguishes conditional dependence of optimal CL changes given SL from the interaction gain of a joint policy outperforming separate adaptive components. Observation 3 only shows that W∗(H)W^*(H) varies with HH, while the CL estimator input in Appendix~B.2 already includes HiH_i, implying that an "SL prediction + CL search conditioned on HiH_i" pipeline can already model this conditional dependence. The paper does not explain what joint RL captures beyond this, leaving the motivation for "joint optimization" without testable meaning.
  2. The paper lacks SL ×\times CL factorial experiments, oracle and measured speedup heatmaps, and a strong baseline of "adaptive SL + adaptive CL conditioned on HiH_i without RL". The gain of AdaptSpec over AdaptSpec/RL only shows that RL contributes, not that interaction is captured. Moreover, Section 5.1 states that each component contributes "independently", which appears to conflict with the interaction claim. Moreover, the CL estimator is trained on measured speedups of executed (H,W)(H,W) pairs, creating counterfactual estimation error and distribution shift for unexecuted CLs queried by binary search, which may prevent reliable identification of the joint optimum. Thus, "joint RL captures interactions" is currently asserted and indirectly inferred rather than experimentally isolated and quantified.
Questions for authors
  1. Clarify the mathematical meaning of ``interaction''. For example, whether r(H,W)≠f(H)+g(W)for any f,g,r(H,W)\neq f(H)+g(W)\quad \text{for any } f,g, i.e., whether r(H,W)r(H,W) cannot be decomposed into f(H)+g(W)f(H)+g(W), or whether the joint policy outperforms separate adaptive SL and CL components. Provide oracle and measured speedup heatmaps over an SL ×\times CL grid, and quantify the degree of non-separability with a defined metric. Also, given that the CL estimator in Appendix B.2 already takes HiH_i as input, explain what joint RL adds beyond an `SL prediction + CL search conditioned on HiH_i' pipeline, and whether joint RL is necessary.

  2. Under identical compute/tuning budgets, the results of comparisons: (a) adaptive SL + adaptive CL conditioned on HiH_i without RL; (b) joint RL; (c) SL-only adaptation or fixed configurations. Specify what interaction-modeling capability RL adds beyond feeding HiH_i into the CL estimator. Report the counterfactual estimation bias of the RF trained on executed (H,W)(H,W) labels when querying unexecuted CLs, and its impact on binary search finding the joint optimum. If the interaction gain is not significant, should the claim be downgraded from ``joint optimization captures interaction'' to `conditional CL selection plus RL tuning'?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.