acceptodds
Under review as a conference paper at ICLR 2027

Where Draft Trees Lose Target Mass: Exit-Guided Speculative Decoding

Abstract

Tree-based speculative decoding organizes draft tokens into a bounded tree so that multiple continuations can be verified in one target-model pass. However, the tree is constructed from draft-side scores while its usefulness is ultimately determined by the target model, creating a fundamental draft–target mismatch under a finite tree budget. We study two questions: once a draft tree is fixed, can a better exact verifier accept more draft tokens, and if not, how can target feedback improve the tree itself? We answer both through a target-flow view of the fixed draft tree. We identify a canonical exit law that describes where target continuations leave the tree, and prove that one plus the resulting target coverage is a sharp upper bound on the expected output-block length, including the bonus token, of any exact path verifier. Moreover, every verifier attaining this ceiling must realize the same exit and bonus-token law; representative predraw-and-follow and sequential residual verifiers already attain it. This theory directly yields Tree Exit Verification (TEV), an exact verifier that realizes the canonical law through one exit-node decision and one bonus-token decision, exposing a regular, level-parallel verification procedure. The same exit law also localizes where target probability is missing from the current tree. We use it as node-level feedback to train the drafter on its inference-time draft trees, reallocating finite tree budget toward target-relevant regions. Experiments across dialogue, code, and mathematical reasoning validate the fixed-tree equivalence, show that the Exit-Guided Draft-Tree Training (ExitTrain) improves average output-block length by 13%, and that TEV reduces verifier-stage latency by 15%, yielding 14% end-to-end speedup over DDTree. Together, our results separate the two remaining opportunities in bounded tree speculative decoding: better draft trees for higher acceptance, and more direct verification for lower latency. Code is available at https://anonymous.4open.science/r/TEV-ICLR-348C/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Inference Acceleration, Speculative decoding, Draft Model Training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_p9ihOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies verification and draft training for tree-based speculative decoding, with three main claimed contributions:

Fixed-tree characterization (Sec 3): It characterizes the acceptance ceiling through target-prefix coverage under its fixed-tree exactness definition, and shows that predraw and sequential residual verification attain the same ceiling and exit distribution.

Tree Exit Verification (Sec 4): It proposes sampling an exit node from this distribution, followed by a residual bonus token, to reduce verification overhead without changing expected acceptance.

ExitTrain (Sec 5): It introduces an auxiliary training objective that uses target-derived exit weights to reinforce tokens in the draft tree, aiming to redirect a limited tree budget toward insufficiently covered continuations.

Critical strengths

ExitTrain is an interesting attempt to use tree-level target feedback to guide draft training. The reported experiments suggest that this approach can improve accepted draft length and decoding throughput.

Critical weaknesses

Two of the paper’s three main contributions lack sufficient novelty and significance, and its positioning relative to prior verification methods raises concerns.

  1. The fixed-tree characterization offers limited theoretical insight, and its scope relative to standard speculative sampling is insufficiently explained. For deterministically selected candidate trees, such as those constructed by EAGLE-3 and DDTree, the basic verification constraint is the target probability mass covered by candidate children at each node. Different tree construction methods do not change this fact; under the paper’s conditional exactness requirement, the acceptance ceiling follows directly from this probability-mass constraint. Nevertheless, the paper devotes substantial space to this characterization and presents it as a major contribution without establishing sufficient additional insight. This observation can also be understood as a degenerate case of the optimal-transport formulation in Towards Optimal Multi-draft Speculative Decoding (Hu et al., 2025). For a deterministic candidate set, the optimal acceptance probability reduces to its target-probability coverage. The paper's fixed-tree ceiling follows naturally by summing this coverage across tree depths. It remains unclear what substantive theoretical insight the proposed characterization adds beyond this elementary case.

Furthermore, the paper repeatedly uses HSD to represent verification mechanisms that exploit target-to-draft probability ratios, although this mechanism already appears in Leviathan et al.’s original speculative decoding work. SpecInfer, Sequoia, and EAGLE-1 also use sampled candidates with corresponding rejection-sampling tree verification. The paper’s exactness definition requires the target distribution to be preserved conditional on the realized candidate tree, which standard sampled-proposal rejection sampling generally does not satisfy. This distinction is not specific to HSD. Discussing this fundamental scope restriction primarily through HSD, without adequately relating it to original speculative sampling and existing tree verifiers, risks overstating the generality and novelty of the results.

  1. TEV’s efficiency contribution is not established against existing implementations, and the released code conflicts with its execution claims and raises substantial timing concerns. Section 4 claims that globally sampling an exit node eliminates per-node sampling and data-dependent child tests, enabling level-parallel verification. However, The paper does not sufficiently distinguish TEV from SGLang’s existing target-only tree verifier, which already performs similar verification procedure inside a CUDA kernel. Comparing only against DDTree’s predraw implementation does not establish a contribution over existing efficient verification methods. Additionally, Table 3 claims to measure verifier decision and cache-commit latency while excluding the shared target forward pass. However, the released code appears to place some TEV-specific overhead (such as softmax, candidate-probability gathering, and D2H transfer) outside this timing interval, raising concerns about the fairness of the comparison.

These concerns affect two of the three claimed contributions, which together occupy the majority of the paper’s technical content and framing. A version centered on ExitTrain, with Sec 3 reduced to background and TEV either dropped or properly benchmarked against existing verifiers, would be a stronger submission. Therefore, I recommend a clear rejection for this manuscript.

Questions for authors
  1. Standard lossless speculative decoding requires the output distribution, marginalized over both draft sampling and verification randomness, to match the target distribution. Why does the paper instead require this equality conditional on each realized candidate tree? The fact that candidates are generated before verification does not itself justify this stronger requirement. Since it generally excludes standard sampled-proposal rejection sampling, including tree methods such as SpecInfer, Sequoia, and EAGLE-1, what motivates this restriction independently of the resulting coverage ceiling, and how does the restricted result answer the broader verification question posed in the introduction?

  2. Section 3 devotes approximately two pages to formalizing a coverage constraint that is already directly reflected in existing fixed-candidate verification procedures. What substantive insight does this analysis add beyond the elementary observation that an exact verifier cannot accept more target probability mass than its candidate prefixes cover, and that existing procedures already exhaust this mass? Why does this warrant a major theoretical contribution rather than a brief background observation supporting the training objective?

  3. Section 6.2 states that the training comparisons use matched initialization, data, and compute. Could the authors provide the exact configurations corresponding to each Table 2 row and explain how compute was matched, for example, by iterations, epochs, or GPU-hours? Please also clarify the relative training throughput and total cost, including rollout and tree evaluation, and whether a comparison with other training methods such as D-PACE under a comparable training budget was considered.

7 comments
AuthorsOpenReview ↗

Rebuttal reply for Reviewer p9ih (1/4)

W1 & Q2: Theoretical contribution and scope

Thank you for the careful review and for recognizing ExitTrain's potential. We address W1 and Q2 together below.

1. What does the full-tree characterization add beyond the OT special case?

Our focus is the joint verification outcome of an entire multi-level draft tree, not only a one-step acceptance probability. The main theory of Hu et al. (2025) optimizes the coupling between a candidate tuple and a single target token. We agree that deterministic-candidate coverage follows from their OT special case and that summing prefix coverage gives a natural length bound. We do not claim these observations alone as our contribution.

The additional result is Theorem 1's equality characterization. In our conditional-exactness class, attaining the total length bound forces every prefix to attain its individual acceptance bound. A verifier cannot accept one prefix less often and compensate by accepting another beyond its bound. Consequently, all saturated verifiers must have the same joint distribution of accepted paths, exit nodes, and bonus tokens, not merely the same expected length. Hu et al. also apply their method to EAGLE trees, but their single-step result does not establish this characterization across saturated tree verifiers.

2. Why is this more than background for training?

We agree that the elementary probability identities do not, by themselves, warrant a major theoretical claim. We disagree that Section 3 only formalizes those identities: the full exit law directly supports both TEV's construction and ExitTrain's training design.

For verification, Predraw and sequential residual methods recover the exit through local samples or child tests. The characterized law permits sampling the exit directly and then its residual bonus. This is the basis of TEV, rather than simply a restatement of the acceptance bound.

For training, the theory identifies both what to improve and where to obtain feedback. Once verification is saturated, higher acceptance requires better tree coverage. Prefix probability P(v)P(v) measures reachability; exit mass w(v)=P(v)ρ(v)w(v)=P(v)\rho(v) additionally identifies target mass missing from the node's children. For expandable nodes, Eqs. (12)-(13) relate this weight to the summed coverage gains of individually adding missing children, before enforcing the budget.

ExitTrain evaluates the current tree under actual target-model parent contexts and uses these detached weights in an auxiliary draft-token likelihood loss. This is a tree-allocation surrogate, not an exact coverage gradient. The theory therefore supplies a specific training signal, not just a retrospective explanation of training gains.

3. How does our definition relate to rejection sampling and tree construction?

Our analysis is independent of the specific tree-construction algorithm; it applies to the resulting candidate support. Top-k expansion and best-first selection are both admissible when they produce a tree satisfying our structural assumptions. We follow the deterministic selection setting used in EAGLE-3 and DFlash with DDTree. Given the context, the tree is determined, so conditioning additionally on it does not further restrict the target law.

Rejection sampling is compatible with this setting. In sequential verification, each fixed candidate xjx_j can be treated as a point-mass proposal qj=δxjq_j=\delta_{x_j}, so qj(xj)=1q_j(x_j)=1. With current target residual rjr_j, the acceptance probability becomes min⁡{1,rj(xj)/qj(xj)}=rj(xj)\min\{1,r_j(x_j)/q_j(x_j)\}=r_j(x_j); rejection removes xjx_j and renormalizes the residual. This is the fixed-candidate procedure analyzed in Appendix A.4.2, which shares TEV's saturated exit and bonus law.

Here qjq_j is the deterministic proposal at that test, not the drafter's original softmax distribution. The theorem concerns exactness for each fixed support; it does not assert equivalence to the original proposal-aware verifier of a random drafter, whose losslessness may hold only after averaging over candidate generation.

We cited HSD to motivate whether verification improvements for sampled chains also remain possible on fixed trees, not to attribute probability-ratio verification to HSD.

4. What will change in the manuscript?

  • Introduction and Section 3: state the scope and OT relationship explicitly; shorten the preliminary coverage discussion while retaining Theorem 1's equality characterization and the appendix proofs.
  • Sections 3.3-5: explain the common exit law's role in TEV's construction and ExitTrain's weighting more directly.
  • Appendix A.5: broaden the discussion beyond HSD to Leviathan et al., SpecInfer, Sequoia, and EAGLE-1, with a main-text pointer.
AuthorsOpenReview ↗

Rebuttal reply for Reviewer p9ih (2/4)

W2: Comparison with SGLang and the timing boundary

1. Competitive verification and its connection to ExitTrain

Thank you for suggesting SGLang's target-only verifier. We have adapted it to the same Transformers backend and compared it with Predraw and TEV using the same trees, target distributions, and cache-commit procedure. This isolates the verifier comparison from differences between serving systems.

The table reports decision-and-commit latency after preprocessing, averaged equally over MT-Bench, HumanEval, GSM8K, and MATH-500. All values are in milliseconds; lower is better.

TemperatureTree budgetPredrawSGLang target-onlyTEV
0.6321.40681.23301.2248
0.6641.45251.23451.2310
0.61281.65901.25731.2545
1.0321.36601.20801.2008
1.0641.42101.23531.2218
1.01281.53451.27831.2720

TEV achieves decision-and-commit latency comparable to SGLang across all six temperature/budget settings. Both perform target-only verification and use the same cache-commit procedure here. These results establish competitive stage latency against an optimized existing verifier, extending the evidence beyond our original Predraw comparison.

Our contribution also connects verification to drafter training through the same exit law. TEV's formulation identifies where target probability mass leaves the current tree. A prefix probability measures how often the target reaches a node; its exit weight additionally measures how much of that flow is not covered by the available children. Thus, nodes with similar reach probabilities can receive different weights depending on their continuation coverage. This supplies the node-level feedback used to design ExitTrain's auxiliary draft-token objective. Table 4 isolates this training benefit while retaining Predraw as the verifier. The exit formulation therefore provides both an exact verification construction and a concrete training signal for improving future trees. Its value extends beyond the runtime difference between two verifier implementations.

2. A consistent timing boundary for both methods

Table 3 compares verification decisions and cache commit after preprocessing. TEV's softmax, candidate-probability gathering, and preparatory D2H transfer were excluded; Predraw likewise excluded its corresponding preprocessing and preparatory transfers. The shared target forward was also excluded. This is a consistent component-level timing convention, not a selective exclusion of TEV's costs.

We will specify the timer boundaries and prepared inputs for each method in the revised paper. We will also supplement this stage comparison with latency from target logits being ready through completed cache commit, including each method's required preprocessing, sampling, transfers, and synchronization. The revision will present the SGLang comparison alongside the connection between the exit formulation and ExitTrain.

AuthorsOpenReview ↗

Rebuttal reply for Reviewer p9ih (3/4)

Q1: Why condition on the realized candidate tree?

Thank you for asking us to clarify the motivation and scope of our paper.

1. A verification guarantee independent of the builder

Conditioning on the tree lets us specify a correct verifier without requiring the tree's joint generation probability. The builder can change its top-k selection, node budget, or pruning rule while the verifier uses the resulting support and target probabilities. This is an independent motivation for the definition, rather than a consequence of the coverage ceiling.

We follow the deterministic candidate-selection setting used in EAGLE-3 and DFlash with DDTree. Given the context cc, the tree is determined: T=F(c)\mathcal T=F(c). Conditioning additionally on that tree adds no restriction to the target law. The justification is this construction rule, not simply that construction precedes verification.

Within this setting, our analysis answers the introduction's question: can changing the verifier increase acceptance on the same tree? Predraw already attains the ceiling; TEV preserves its acceptance law while providing another verification construction. Improving acceptance instead requires better target-prefix coverage.

2. Connection to rejection sampling and standard losslessness

Fixed-candidate rejection sampling is included in our analysis (Appendix A.4.2). At each sequential test, the fixed candidate xjx_j has the point-mass proposal qj=δxjq_j=\delta_{x_j}. Against the current target residual rjr_j, its acceptance probability is rj(xj)r_j(x_j); rejection removes that token and renormalizes the residual.

For example, let the target probabilities of (a,b,c)(a,b,c) be (0.6,0.3,0.1)(0.6,0.3,0.1) and the root's children be {a,b}\{a,b\}. Sequential correction accepts aa with probability 0.60.6, or rejects it and accepts bb with probability 0.4×0.75=0.30.4\times0.75=0.3. The remaining probability 0.10.1 produces cc as the root's residual bonus. TEV gives the same first-token distribution through its exit events.

The scope distinction is between this fixed-support correction and verifiers that rely on random proposal generation to obtain marginal exactness. The latter need not satisfy Eq. (3); the fixed-tree ceiling is not asserted for that broader class. TEV itself satisfies standard losslessness: averaging its conditional target-law guarantee over candidate trees preserves the target law, and Eq. (3)'s stopping-time condition supports composition across rounds.

3. Checking the complete round-output distribution

For a fixed tree, target autoregressive sampling until the first departure from the tree has the exit-node/bonus-token law

Qref(v,a)=P(v)pv(a)1[a∉A(v)].Q_{\mathrm{ref}}(v,a)=P(v)p_v(a)\mathbf{1}[a\notin\mathcal A(v)].

Our theory establishes QTEV=QrefQ_{\mathrm{TEV}}=Q_{\mathrm{ref}}. Because sibling tokens are unique, (v,a)(v,a) uniquely determines the emitted block π(v)∥a\pi(v)\Vert a. This joint law therefore describes the complete output of a verification round, not just a token marginal.

We also checked the actual verifier using Qwen3-4B at temperature 11. For each benchmark, we used 128 fixed tree/logit snapshots and ran TEV 100,000 times per snapshot with fresh randomness. We compared the empirical joint distribution with the independently computed analytical reference and averaged the per-snapshot TV distances:

BenchmarkMean empirical TV distance
MT-Bench3.24×10−43.24\times10^{-4}
HumanEval2.17×10−42.17\times10^{-4}
GSM8K1.78×10−41.78\times10^{-4}
MATH-5001.93×10−41.93\times10^{-4}

These measurements supplement the theoretical distribution-preservation result with an implementation-level check. We will clarify the scope in the introduction and Section 3, expand the rejection-sampling discussion in Appendix A.5, and document this experiment in the revision.

Reviewer_6ygnOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies two questions to improve speculative deocding performance. First they ask the question, can existing verification methods be improved for the given tree structure? Secondly can we train the drafter better to obtain better trees?

Critical strengths

The paper studies an important question and it is easy to follow.

Critical weaknesses

I have three major concerns about the paper:

(a) Use of heavy new terminology for simple terms in probability: For example, Exact path verifier is same as standard speculative sampling, Saturated verifier for lossless drafter verification. These new terms makes the paper hard to read and obfusicates the real contributions.

(b) Theoretical results are not that interesting: Most theorems and Lemmas follow by standard probability definitions e.g., Lemma 1 follows by simple conditional probability definitions. While technical complexity is not required, I would not count theoretical results as a significant contribution of the paper.

(c) The source of gains is unclear: the abstract reads that "TEV reduces verififer stage latency by 15%" But if I understanding correctly, majority of gains come from drafter training (e.g., see that the gains from TEV over previous drafter is marginal in Table 4"

Questions for authors

In addition to the above concerns, it would be great if authors can expand on how much training time it took for drafter training and how it compares to previous methods of drafter training.

4 comments
AuthorsOpenReview ↗

Rebuttal reply for Reviewer 6ygn (1/4)

(a) Use of heavy new terminology for simple terms in probability: For example, Exact path verifier is same as standard speculative sampling, Saturated verifier for lossless drafter verification. These new terms makes the paper hard to read and obfusicates the real contributions.

Thank you for recognizing the importance of the problem. We take the concern about terminology seriously and will make the definitions easier to read. The distinction they express, however, is necessary for the question the paper asks.

Losslessness concerns output correctness; saturation additionally requires attaining the fixed-tree acceptance ceiling. A verifier can ignore the entire draft tree and directly sample one target token per round. It remains lossless, but accepts zero draft tokens, so it is not saturated whenever the tree has positive target-prefix coverage. These terms therefore distinguish whether verification is correct from whether it has exhausted the tree's acceptance potential.

We will introduce the terms with the plain-language descriptions "preserving the target distribution for a fixed tree" and "attaining the fixed-tree acceptance ceiling," add the example above, and reduce repeated terminology in Section 3.

AuthorsOpenReview ↗

Rebuttal reply for Reviewer 6ygn (2/4)

(b) Theoretical results are not that interesting: Most theorems and Lemmas follow by standard probability definitions e.g., Lemma 1 follows by simple conditional probability definitions. While technical complexity is not required, I would not count theoretical results as a significant contribution of the paper.

Lemma 1 uses elementary probability. Our contribution is the characterization of optimal verification behavior on the entire tree and its consequences for verification and training, rather than the difficulty of that identity.

Theorem 1 shows that, within our conditionally exact class, attaining the total acceptance bound forces every prefix to attain its individual bound. No verifier can accept one prefix less often and compensate elsewhere, because other prefixes cannot exceed their own bounds. Consequently, all saturated verifiers share the same joint law of accepted paths, exit nodes, and bonus tokens:

Pr⁡(V=v,Y=a∣c,T)=P(v)pv(a)1[a∉A(v)].\Pr(V=v,Y=a\mid c,\mathcal T)=P(v)p_v(a)\mathbf{1}[a\notin\mathcal A(v)].

This goes beyond observing that two existing procedures have equal average acceptance. It specifies the distribution that every saturated alternative must realize. Predraw and sequential residual verification find an exit through local samples or child tests; the joint-law characterization permits sampling the exit directly and then its residual bonus, giving TEV's construction.

The training insight is equally concrete: the exit law identifies where a tree loses target probability mass. As an illustrative example, suppose two nodes each have reach probability P=0.2P=0.2, but their children cover 90% and 50% of the respective next-token distributions. Prefix weighting treats them equally, whereas their exit weights are 0.020.02 and 0.100.10. Exit weighting thus distinguishes frequently reached prefixes whose continuations are already covered from those with larger coverage gaps. Eqs. (12)-(13) connect these weights to the gains from adding missing children before imposing the tree budget.

ExitTrain evaluates the current draft tree under actual target-model parent contexts and uses detached exit weights in an auxiliary draft-token likelihood loss alongside the token loss. This is a tree-allocation surrogate, not an exact gradient through the discrete builder. The prefix-versus-exit experiment below tests this choice of feedback under the same training cost and verifier. We will shorten the elementary derivations and foreground this chain from characterization to algorithm design in Sections 3-5.

AuthorsOpenReview ↗

Rebuttal reply for Reviewer 6ygn (3/4)

(c) The source of gains is unclear: the abstract reads that "TEV reduces verififer stage latency by 15%" But if I understanding correctly, majority of gains come from drafter training (e.g., see that the gains from TEV over previous drafter is marginal in Table 4"

Thank you for pointing out the need to clarify the source of the gains. The end-to-end gains are indeed primarily driven by ExitTrain. Table 4 separates the two effects: with Predraw fixed, training increases mean SR from 4.95 to 5.55 (+12.1%), showing that the training benefit does not depend on using TEV at inference. TEV provides a smaller, complementary efficiency benefit rather than higher acceptance on a fixed tree.

The reported 15% reduction refers specifically to Table 3’s decision-and-cache-commit interval after preprocessing, not overall decoding latency. We will clarify this measurement scope and the attribution of gains in the abstract, and supplement the results with complete logits-ready-to-cache-commit timing.

AuthorsOpenReview ↗

Rebuttal reply for Reviewer 6ygn (4/4)

In addition to the above concerns, it would be great if authors can expand on how much training time it took for drafter training and how it compares to previous methods of drafter training.

For Qwen3-4B, all four objectives were trained for two epochs. The costs below include target rollout and drafter optimization; the tree-weighted objectives also include target-tree evaluation. We have added D-PACE continuation training. All inference configurations use the same builder and B=64B=64; the table identifies the verifier and reports mean τ\tau over the four benchmarks.

Training objectiveEpochsGPU-hoursVerifierτ\tau, T=0T=0τ\tau, T=1T=1
Token-only finetuning2~200Predraw8.3607.468
D-PACE2~200Predraw8.5487.803
Prefix weighting2~280TEV8.2087.715
Exit weighting (ExitTrain)2~280TEV8.8488.063

ExitTrain's mean accepted length exceeds D-PACE's by 3.51% / 3.33% at T=0/1T=0/1, with higher values on all four benchmarks at both temperatures. Predraw and TEV have equal expected accepted length on the same tree, so verifier choice does not supply this acceptance advantage.

The prefix-versus-exit control holds both training cost and verifier fixed. Both consume approximately 280 GPU-hours and use TEV. Exit weighting improves mean τ\tau by 7.80% / 4.50% and mean SR by 7.40% / 4.31%. This supports the particular exit-based feedback, beyond simply spending more compute on target-tree evaluation.

The protocol matches epochs, not total GPU-hours. ExitTrain's additional approximately 80 GPU-hours are a one-time training investment that can be amortized with sufficient inference reuse. We will provide the row-wise configurations and clarify the cost accounting in Section 6.2, the Table 2 caption, and the training appendix.

We would appreciate your reassessment of the contribution in light of the distinction between correctness and saturation, the constructive role of the theory, and the separately measured training benefits.

Reviewer_b1ghOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the acceptance ceiling of fixed draft trees. The authors prove that the two existing exact verifiers already attain this ceiling. Building on this, they propose TEV, which directly samples an exit node, and ExitTrain, which trains with exit-probability weighting. Experiments show that the main speedup comes from ExitTrain

Critical strengths
  1. The motivation of the paper is clearly argued. The method is simple and requires no architectural modification to the drafter.
  2. The design of TEV is simple and elegant, avoiding node-wise sampling and sequential verification dependencies. ExitTrain also delivers solid gains in average output-block length and speedup.
Critical weaknesses

I have some concerns about the ablation study. The training-objective ablation is conducted only on Qwen3-4B. However, the gains vary substantially across models in the main experiments, from 17.7% on LLaMA to 5.6% on Gemma. It is therefore unclear whether exit weighting also outperforms prefix weighting on other models. An ablation on at least one additional model would strengthen this claim.

Questions for authors

Was exit weighting compared with prefix weighting on any model other than Qwen3-4B? Given the varying gains across model families, it is unclear whether the advantage in Table 2 generalizes.

2 comments
AuthorsOpenReview ↗

Rebuttal reply for Reviewer b1gh

Thank you for suggesting a cross-model ablation. We have added a controlled comparison of exit weighting and prefix weighting on Llama-3.1-8B, extending Table 2 beyond the Qwen family.

The two runs use matched initialization, training data, training schedule, and budget. We retain the auxiliary-loss coefficient and normalization procedure, changing only the weighting rule. Inference uses the same verifier, tree builder, and node budget B=64B=64, so the comparison isolates the training weights rather than changes in verification or tree capacity.

The table reports arithmetic means over MT-Bench, HumanEval, GSM8K, and MATH-500. As in the paper, τ\tau is the average output-block length including the bonus token, and SR is the speedup ratio relative to autoregressive decoding.

TemperatureTraining weightsMean SRMean τ\tau
0Prefix weight4.286.54
0Exit weight4.466.97
1Prefix weight3.034.80
1Exit weight3.165.01

At T=0T=0, exit weighting improves mean τ\tau by 6.57% and mean SR by 4.32%. At T=1T=1, the improvements are 4.32% and 4.17%, respectively. These percentages use the unrounded means. Both metrics improve on all four benchmarks at both temperatures, rather than the average gain being driven by a single task.

This provides additional evidence that the benefit of exit weighting is not specific to Qwen3-4B: the positive comparison with prefix weighting also holds on Llama-3.1-8B. The main-table gains compare complete configurations against a baseline, whereas this ablation measures the incremental effect of exit versus prefix weighting within a model. Different overall gains across model families therefore need not imply a reversal of the weighting comparison.

We will incorporate the Llama-3.1-8B ablation into Section 6.2 and provide the full per-benchmark results and controlled training configuration in the appendix. The revised discussion will explicitly base the cross-family conclusion on the Qwen3-4B and Llama-3.1-8B comparisons.

Thanks for running the Llama ablation. Seeing exit weighting improve speedup on a second model answers my main question. My score stays the same.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.