- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies verification and draft training for tree-based speculative decoding, with three main claimed contributions:
Fixed-tree characterization (Sec 3): It characterizes the acceptance ceiling through target-prefix coverage under its fixed-tree exactness definition, and shows that predraw and sequential residual verification attain the same ceiling and exit distribution.
Tree Exit Verification (Sec 4): It proposes sampling an exit node from this distribution, followed by a residual bonus token, to reduce verification overhead without changing expected acceptance.
ExitTrain (Sec 5): It introduces an auxiliary training objective that uses target-derived exit weights to reinforce tokens in the draft tree, aiming to redirect a limited tree budget toward insufficiently covered continuations.
Critical strengths
ExitTrain is an interesting attempt to use tree-level target feedback to guide draft training. The reported experiments suggest that this approach can improve accepted draft length and decoding throughput.
Critical weaknesses
Two of the paper’s three main contributions lack sufficient novelty and significance, and its positioning relative to prior verification methods raises concerns.
- The fixed-tree characterization offers limited theoretical insight, and its scope relative to standard speculative sampling is insufficiently explained. For deterministically selected candidate trees, such as those constructed by EAGLE-3 and DDTree, the basic verification constraint is the target probability mass covered by candidate children at each node. Different tree construction methods do not change this fact; under the paper’s conditional exactness requirement, the acceptance ceiling follows directly from this probability-mass constraint. Nevertheless, the paper devotes substantial space to this characterization and presents it as a major contribution without establishing sufficient additional insight. This observation can also be understood as a degenerate case of the optimal-transport formulation in Towards Optimal Multi-draft Speculative Decoding (Hu et al., 2025). For a deterministic candidate set, the optimal acceptance probability reduces to its target-probability coverage. The paper's fixed-tree ceiling follows naturally by summing this coverage across tree depths. It remains unclear what substantive theoretical insight the proposed characterization adds beyond this elementary case.
Furthermore, the paper repeatedly uses HSD to represent verification mechanisms that exploit target-to-draft probability ratios, although this mechanism already appears in Leviathan et al.’s original speculative decoding work. SpecInfer, Sequoia, and EAGLE-1 also use sampled candidates with corresponding rejection-sampling tree verification. The paper’s exactness definition requires the target distribution to be preserved conditional on the realized candidate tree, which standard sampled-proposal rejection sampling generally does not satisfy. This distinction is not specific to HSD. Discussing this fundamental scope restriction primarily through HSD, without adequately relating it to original speculative sampling and existing tree verifiers, risks overstating the generality and novelty of the results.
- TEV’s efficiency contribution is not established against existing implementations, and the released code conflicts with its execution claims and raises substantial timing concerns. Section 4 claims that globally sampling an exit node eliminates per-node sampling and data-dependent child tests, enabling level-parallel verification. However, The paper does not sufficiently distinguish TEV from SGLang’s existing target-only tree verifier, which already performs similar verification procedure inside a CUDA kernel. Comparing only against DDTree’s predraw implementation does not establish a contribution over existing efficient verification methods. Additionally, Table 3 claims to measure verifier decision and cache-commit latency while excluding the shared target forward pass. However, the released code appears to place some TEV-specific overhead (such as softmax, candidate-probability gathering, and D2H transfer) outside this timing interval, raising concerns about the fairness of the comparison.
These concerns affect two of the three claimed contributions, which together occupy the majority of the paper’s technical content and framing. A version centered on ExitTrain, with Sec 3 reduced to background and TEV either dropped or properly benchmarked against existing verifiers, would be a stronger submission. Therefore, I recommend a clear rejection for this manuscript.
Questions for authors
-
Standard lossless speculative decoding requires the output distribution, marginalized over both draft sampling and verification randomness, to match the target distribution. Why does the paper instead require this equality conditional on each realized candidate tree? The fact that candidates are generated before verification does not itself justify this stronger requirement. Since it generally excludes standard sampled-proposal rejection sampling, including tree methods such as SpecInfer, Sequoia, and EAGLE-1, what motivates this restriction independently of the resulting coverage ceiling, and how does the restricted result answer the broader verification question posed in the introduction?
-
Section 3 devotes approximately two pages to formalizing a coverage constraint that is already directly reflected in existing fixed-candidate verification procedures. What substantive insight does this analysis add beyond the elementary observation that an exact verifier cannot accept more target probability mass than its candidate prefixes cover, and that existing procedures already exhaust this mass? Why does this warrant a major theoretical contribution rather than a brief background observation supporting the training objective?
-
Section 6.2 states that the training comparisons use matched initialization, data, and compute. Could the authors provide the exact configurations corresponding to each Table 2 row and explain how compute was matched, for example, by iterations, epochs, or GPU-hours? Please also clarify the relative training throughput and total cost, including rollout and tree evaluation, and whether a comparison with other training methods such as D-PACE under a comparable training budget was considered.
7 comments
Rebuttal reply for Reviewer p9ih (2/4)
W2: Comparison with SGLang and the timing boundary
1. Competitive verification and its connection to ExitTrain
Thank you for suggesting SGLang's target-only verifier. We have adapted it to the same Transformers backend and compared it with Predraw and TEV using the same trees, target distributions, and cache-commit procedure. This isolates the verifier comparison from differences between serving systems.
The table reports decision-and-commit latency after preprocessing, averaged equally over MT-Bench, HumanEval, GSM8K, and MATH-500. All values are in milliseconds; lower is better.
| Temperature | Tree budget | Predraw | SGLang target-only | TEV |
|---|---|---|---|---|
| 0.6 | 32 | 1.4068 | 1.2330 | 1.2248 |
| 0.6 | 64 | 1.4525 | 1.2345 | 1.2310 |
| 0.6 | 128 | 1.6590 | 1.2573 | 1.2545 |
| 1.0 | 32 | 1.3660 | 1.2080 | 1.2008 |
| 1.0 | 64 | 1.4210 | 1.2353 | 1.2218 |
| 1.0 | 128 | 1.5345 | 1.2783 | 1.2720 |
TEV achieves decision-and-commit latency comparable to SGLang across all six temperature/budget settings. Both perform target-only verification and use the same cache-commit procedure here. These results establish competitive stage latency against an optimized existing verifier, extending the evidence beyond our original Predraw comparison.
Our contribution also connects verification to drafter training through the same exit law. TEV's formulation identifies where target probability mass leaves the current tree. A prefix probability measures how often the target reaches a node; its exit weight additionally measures how much of that flow is not covered by the available children. Thus, nodes with similar reach probabilities can receive different weights depending on their continuation coverage. This supplies the node-level feedback used to design ExitTrain's auxiliary draft-token objective. Table 4 isolates this training benefit while retaining Predraw as the verifier. The exit formulation therefore provides both an exact verification construction and a concrete training signal for improving future trees. Its value extends beyond the runtime difference between two verifier implementations.
2. A consistent timing boundary for both methods
Table 3 compares verification decisions and cache commit after preprocessing. TEV's softmax, candidate-probability gathering, and preparatory D2H transfer were excluded; Predraw likewise excluded its corresponding preprocessing and preparatory transfers. The shared target forward was also excluded. This is a consistent component-level timing convention, not a selective exclusion of TEV's costs.
We will specify the timer boundaries and prepared inputs for each method in the revised paper. We will also supplement this stage comparison with latency from target logits being ready through completed cache commit, including each method's required preprocessing, sampling, transfers, and synchronization. The revision will present the SGLang comparison alongside the connection between the exit formulation and ExitTrain.
Rebuttal reply for Reviewer p9ih (3/4)
Q1: Why condition on the realized candidate tree?
Thank you for asking us to clarify the motivation and scope of our paper.
1. A verification guarantee independent of the builder
Conditioning on the tree lets us specify a correct verifier without requiring the tree's joint generation probability. The builder can change its top-k selection, node budget, or pruning rule while the verifier uses the resulting support and target probabilities. This is an independent motivation for the definition, rather than a consequence of the coverage ceiling.
We follow the deterministic candidate-selection setting used in EAGLE-3 and DFlash with DDTree. Given the context , the tree is determined: . Conditioning additionally on that tree adds no restriction to the target law. The justification is this construction rule, not simply that construction precedes verification.
Within this setting, our analysis answers the introduction's question: can changing the verifier increase acceptance on the same tree? Predraw already attains the ceiling; TEV preserves its acceptance law while providing another verification construction. Improving acceptance instead requires better target-prefix coverage.
2. Connection to rejection sampling and standard losslessness
Fixed-candidate rejection sampling is included in our analysis (Appendix A.4.2). At each sequential test, the fixed candidate has the point-mass proposal . Against the current target residual , its acceptance probability is ; rejection removes that token and renormalizes the residual.
For example, let the target probabilities of be and the root's children be . Sequential correction accepts with probability , or rejects it and accepts with probability . The remaining probability produces as the root's residual bonus. TEV gives the same first-token distribution through its exit events.
The scope distinction is between this fixed-support correction and verifiers that rely on random proposal generation to obtain marginal exactness. The latter need not satisfy Eq. (3); the fixed-tree ceiling is not asserted for that broader class. TEV itself satisfies standard losslessness: averaging its conditional target-law guarantee over candidate trees preserves the target law, and Eq. (3)'s stopping-time condition supports composition across rounds.
3. Checking the complete round-output distribution
For a fixed tree, target autoregressive sampling until the first departure from the tree has the exit-node/bonus-token law
Our theory establishes . Because sibling tokens are unique, uniquely determines the emitted block . This joint law therefore describes the complete output of a verification round, not just a token marginal.
We also checked the actual verifier using Qwen3-4B at temperature . For each benchmark, we used 128 fixed tree/logit snapshots and ran TEV 100,000 times per snapshot with fresh randomness. We compared the empirical joint distribution with the independently computed analytical reference and averaged the per-snapshot TV distances:
| Benchmark | Mean empirical TV distance |
|---|---|
| MT-Bench | |
| HumanEval | |
| GSM8K | |
| MATH-500 |
These measurements supplement the theoretical distribution-preservation result with an implementation-level check. We will clarify the scope in the introduction and Section 3, expand the rejection-sampling discussion in Appendix A.5, and document this experiment in the revision.
Rebuttal reply for Reviewer p9ih (1/4)
Thank you for the careful review and for recognizing ExitTrain's potential. We address W1 and Q2 together below.
1. What does the full-tree characterization add beyond the OT special case?
Our focus is the joint verification outcome of an entire multi-level draft tree, not only a one-step acceptance probability. The main theory of Hu et al. (2025) optimizes the coupling between a candidate tuple and a single target token. We agree that deterministic-candidate coverage follows from their OT special case and that summing prefix coverage gives a natural length bound. We do not claim these observations alone as our contribution.
The additional result is Theorem 1's equality characterization. In our conditional-exactness class, attaining the total length bound forces every prefix to attain its individual acceptance bound. A verifier cannot accept one prefix less often and compensate by accepting another beyond its bound. Consequently, all saturated verifiers must have the same joint distribution of accepted paths, exit nodes, and bonus tokens, not merely the same expected length. Hu et al. also apply their method to EAGLE trees, but their single-step result does not establish this characterization across saturated tree verifiers.
2. Why is this more than background for training?
We agree that the elementary probability identities do not, by themselves, warrant a major theoretical claim. We disagree that Section 3 only formalizes those identities: the full exit law directly supports both TEV's construction and ExitTrain's training design.
For verification, Predraw and sequential residual methods recover the exit through local samples or child tests. The characterized law permits sampling the exit directly and then its residual bonus. This is the basis of TEV, rather than simply a restatement of the acceptance bound.
For training, the theory identifies both what to improve and where to obtain feedback. Once verification is saturated, higher acceptance requires better tree coverage. Prefix probability P(v) measures reachability; exit mass w(v)=P(v)ρ(v) additionally identifies target mass missing from the node's children. For expandable nodes, Eqs. (12)-(13) relate this weight to the summed coverage gains of individually adding missing children, before enforcing the budget.
ExitTrain evaluates the current tree under actual target-model parent contexts and uses these detached weights in an auxiliary draft-token likelihood loss. This is a tree-allocation surrogate, not an exact coverage gradient. The theory therefore supplies a specific training signal, not just a retrospective explanation of training gains.
3. How does our definition relate to rejection sampling and tree construction?
Our analysis is independent of the specific tree-construction algorithm; it applies to the resulting candidate support. Top-k expansion and best-first selection are both admissible when they produce a tree satisfying our structural assumptions. We follow the deterministic selection setting used in EAGLE-3 and DFlash with DDTree. Given the context, the tree is determined, so conditioning additionally on it does not further restrict the target law.
Rejection sampling is compatible with this setting. In sequential verification, each fixed candidate xj can be treated as a point-mass proposal qj=δxj, so qj(xj)=1. With current target residual rj, the acceptance probability becomes min{1,rj(xj)/qj(xj)}=rj(xj); rejection removes xj and renormalizes the residual. This is the fixed-candidate procedure analyzed in Appendix A.4.2, which shares TEV's saturated exit and bonus law.
Here qj is the deterministic proposal at that test, not the drafter's original softmax distribution. The theorem concerns exactness for each fixed support; it does not assert equivalence to the original proposal-aware verifier of a random drafter, whose losslessness may hold only after averaging over candidate generation.
We cited HSD to motivate whether verification improvements for sampled chains also remain possible on fixed trees, not to attribute probability-ratio verification to HSD.
4. What will change in the manuscript?