acceptodds
Under review as a conference paper at ICLR 2027

A Sampling-Centric Framework for Tree Speculative Decoding

Abstract

Tree speculative decoding accelerates large language model inference by drafting and verifying multiple candidate continuations in parallel. Top-candidate dynamic-tree methods such as EAGLE-2/3 follow a topology-first paradigm: they expand deterministic candidates and globally prune them by score, so draft–target overlap outside the retained set cannot contribute to acceptance. Introducing sampled candidates into this pipeline creates a selection problem: under global pruning, a sample's realized identity can affect its own survival. Existing sampling extensions such as RheoSampling retain expand-then-prune and therefore must explicitly control how samples are scored and pruned to preserve losslessness. We introduce Solvo, a sampling-centric framework that replaces expand-then-prune with allocate-then-sample. At each layer, a global path-score threshold commits slots to parents before any token is drawn, and slots are never reranked afterward, so a sampled token can condition subsequent expansion but not its own retention. Under stochastic decoding, each non-root candidate group places top-ranked deterministic tokens in every slot except the last, which is filled with a sample from the remaining draft mass. Using a martingale argument over tree growth, we prove that Solvo preserves the exact target-model distribution under adaptive growth and cost-aware stopping. Across five target models and six tasks, Solvo improves end-to-end throughput by 7.3-10.2% over EAGLE-3 under stochastic decoding, and still by 2.9-7.8% under greedy decoding, where sampling plays no role.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Speculative Decoding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aXHKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents Solvo, a sampling-centric framework for tree speculative decoding. It allocates tree slots before sampling token identities, combines deterministic candidates with residual sampling, and introduces cost-aware tree growth. A causal verification analysis establishes preservation of the target distribution under the stated construction. Experiments across five target models and six tasks, together with component ablations and tuned-baseline comparisons, evaluate decoding throughput.

Critical strengths
  1. The separation of allocation from sampled token identity is technically valuable. It addresses a concrete source of selection bias, and the causal verification argument connects the tree-construction rule to distributional correctness.
  2. The shared-backend comparisons and cumulative ablations strengthen the empirical validation. The stochastic and greedy evaluations, including the additional TALON tuning comparison, give useful evidence that the construction improves an implemented decoder.
Critical weaknesses
  1. The main limitation is the demonstrated deployment scope. The throughput evidence uses one GPU platform, batch size one, and no concurrent requests; it does not yet establish how the cost-aware stopping rule behaves under serving contention or larger batches.
  2. The margin over the tuned TALON baseline is relatively small in greedy decoding. This makes sensitivity to the model-specific threshold and latency calibration important when judging how portable the improvement is.
Questions for authors
  1. How much throughput is lost when the threshold and stopping-cost calibration are fixed on a separate workload and transferred without retuning?
  2. Does the advantage over tuned TALON remain under concurrent requests or a larger serving batch?
1 comment

Thank you for your review! We are glad to answer your questions.

Q1/W2. How much throughput is lost when the threshold and stopping-cost calibration are fixed on a separate workload and transferred without retuning?

In all our experiments, the stopping cost is calibrated on only 4 Alpaca prompts and then frozen across all six datasets (480 prompts). For θ, the value selected on MT-Bench-101, an independent dataset not among the six in our paper, matches the paper setting (see our response to Reviewer wLg1). This also addresses the single-GPU-platform concern in W1: we further test transfer from the RTX PRO 6000 to an RTX 3090.

Using Llama3.1-8B-Instruct, we transfer the threshold (θ = 0.01) and the stopping-cost calibration (Ld = 0.717 ms, Lb = 22.947 ms) obtained on the RTX PRO 6000 directly to an RTX 3090, and compare with recalibrating on the RTX 3090 (Ld = 1.298 ms, Lb = 30.427 ms), keeping θ = 0.01. All six datasets (480 prompts) are used.

TemperatureTransferred TPSRecalibrated TPSThroughput change from transfer
T=0139.707140.256−0.39%
T=1122.723122.325+0.33%

The difference is negligible.

For θ, we use the recalibrated Ld/Lb and re-sweep θ on all six datasets (480 prompts) on the RTX 3090, alongside the RTX PRO 6000 results:

θTPS (RTX 3090)TPS (RTX PRO 6000)
0.002132.687200.633
0.005137.713208.856
0.01140.256215.454
0.015138.894213.738
0.02136.779208.405

The optimal θ remains 0.01, and the trend over θ is essentially the same on both GPUs.

In summary:

  1. The global threshold θ generally does not need recalibration, since it mainly depends on the properties of the draft and target models rather than on the hardware.
  2. Early stopping is more hardware-dependent, but even without recalibration the impact on our RTX 3090 is small. Since calibration is cheap (a few minutes, using only 4 Alpaca prompts), we recommend doing it anyway.

Q2. Does the advantage over tuned TALON remain under concurrent requests or a larger serving batch?

Yes, it does.

We ran continuous serving with SGLang using Llama3.1-8B on MT-Bench. "Batch" denotes the running batch size. TPS:

BatchT=0 TALONT=0 SolvoT=1 TALONT=1 Solvo
2565.87581.48423.09454.63
4925.27957.56700.21771.55
81261.661297.55950.901034.57
161502.361544.231095.481228.20
321731.431793.561275.131427.54

TALON uses μ = 0.03, the single-request optimum from our sweep in Table 9. Solvo reuses the parameters from the paper without any change.

Since TALON's optimal μ may shift with batch size, we additionally swept μ at batch = 8:

TALON μT=0 TPST=1 TPS
0.011229.04950.06
0.021245.74950.51
0.031261.66950.90
0.041259.40933.99
0.051265.20900.63
0.061262.95922.24
0.071254.27916.03

At T=1, TALON's optimal μ is still 0.03. At T=0 the optimum is 0.05, only marginally above 0.03. Even with this optimum, TALON reaches 1265.20, still below Solvo's 1297.55.

W1. How does the cost-aware stopping rule behave under serving contention or larger batches?

Thank you for this insightful question.

Under continuous serving, the stopping decision can take the other requests in the batch into account. For example, if most requests have already been stopped by the per-request rule while a few continue drafting, the additional draft forward may no longer pay off for the batch as a whole.

A natural extension for Solvo is therefore to continue drafting only if another layer improves the speedup of the whole batch. This yields further gains:

BatchPer-request early stopBatched early stopImprovement
2581.48585.35+0.67%
4957.56972.80+1.59%
81297.551330.20+2.52%
161544.231573.79+1.91%
321793.561809.45+0.89%
Reviewer_8yyXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies that EAGLE-style expand-then-prune construction can bias sampled candidates because a sample’s realized identity may affect whether it survives pruning.

The proposed method (Solvo) instead allocates tree slots before sampling, then fills those slots with deterministic and residual-sampled candidates, together with a cost-aware stopping rule for adaptive depth.

The paper proves that this procedure preserves the exact target-model distribution and shows experimentally that it improves end-to-end decoding speed over EAGLE-3, TALON, and RheoSampling, especially under stochastic decoding.

Critical strengths
  • Clear methodological novelty with a principled design insight.

Solvo replaces the usual expand-then-prune dynamic-tree construction with allocate-then-sample, so sampled token identities cannot affect their own retention while still influencing subsequent tree growth.

  • Strong correctness and empirical validation.

The paper gives a nontrivial martingale-based proof that Solvo preserves the exact target-model distribution under adaptive growth and stopping, and it validates the method across five target models and six tasks with end-to-end speedup over baselines.

Critical weaknesses

The method appears to rely on a bag of empirical tricks, and the key source of improvement is unclear.

Solvo combines threshold-based tree construction, cost-aware stopping, mixed stochastic proposals, and UniVer, with each component contributing some gain in the ablation.

As a result, it is difficult to determine whether the main improvement actually comes from the paper’s central allocate-then-sample idea or from the accumulation of several auxiliary design choices.

Questions for authors

How much of Solvo’s gain comes specifically from the proposed allocate-then-sample construction?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1 of 2)

Thank you for your review! We are glad to answer your question.

Q1. Key Source of Improvement

Effect of sampled candidates on acceptance

We first use a 2-candidate example to show how adding a sampled candidate affects the acceptance rate of speculative decoding. On Llama3.1-8B-Instruct and MT-Bench, we compare two candidate sets:

  1. the top-2 tokens of the draft distribution q;
  2. the top-1 token of q, plus one token sampled from the residual of q.

Both rates below are expected acceptance rates computed on the same trajectories:

Candidate setAcceptance rate
Top-278.72%
Top-1 + 1 residual sample85.27%

Adding a sampled candidate clearly raises the acceptance rate. Intuitively, this is the advantage of mixed proposals: deterministic candidates capture the acceptance in the head of the distribution, while a residual sample explores the draft–target overlap in the tail, which pure Top-K cannot do. Mixed proposals and the optimal transport (OT) view of speculative sampling are analyzed in detail in Towards Optimal Multi-Draft Speculative Decoding (Hu et al., ICLR 2025).

Effect of the verifier on acceptance length

The example above shows that single-layer acceptance depends on the proposal scheme. Given a fixed proposal scheme (e.g., top-1 + 1 sample everywhere), the verifier also affects acceptance.

Take UniVer (Weng et al., NeurIPS 2026) as an example. For tree decoding, UniVer allows probability allocation across layers, whereas the method of Hu et al. only allocates within a layer (single-layer OT). Both verifiers are lossless, but UniVer has more freedom in allocating probability mass, so it achieves a longer acceptance length even on exactly the same tree with exactly the same sampled nodes.

We use a binary tree with 5 draft layers below the root, with top-1 + 1 sample at every node, on the 80 MT-Bench prompts. Both values are expected acceptance lengths on the same trees:

VerifierAcceptance length
Single-layer OT4.160
UniVer4.263

UniVer achieves a clearly longer acceptance length than per-layer allocation. This is the gain an advanced verifier brings.

Note that an advanced verifier is effective only when the tree contains sampled candidates. Without them, the verifier cannot exploit any flexibility in the probability allocation: from the OT perspective the feasible set is fixed, and only token matching remains. As reported in Appendix F of our paper, applying UniVer to EAGLE-3 brings no acceptance gain.

(continued in the next comment)

AuthorsOpenReview ↗

Official Comment by Authors (2 of 2)

Q2. What allocate-then-sample enables

The key point is that neither gain above (mixed proposals and UniVer) is available by default in dynamic trees such as EAGLE-3. Sampled candidates cannot simply be added to an expand-then-prune dynamic tree: as discussed in our paper, if sampled nodes are present without special treatment, global pruning breaks losslessness. In short, EAGLE-3's expand-then-prune effectively forces Top-K candidates, so the gains of sampling and advanced verification cannot be obtained there.

Our question was therefore: if the tree must accommodate sampled candidates and an advanced verifier from the outset, how should it be built? rather than how can sampled candidates be inserted into EAGLE-3's expand-then-prune construction? (the latter is what RheoSampling does). This is why we call Solvo a sampling-centric framework: allocate-then-sample is what makes flexible sampling and advanced verification straightforward to realize in a dynamic tree, and thereby collects these gains.

Table 2 of our paper:

ConfigurationτSR
EAGLE-35.083.99×
+ Threshold tree5.244.10×
+ Cost-aware stopping5.234.16×
+ Mixed proposals5.344.21×
+ Advanced verifier5.504.33×

All four steps follow from the allocate-then-sample design:

  • Threshold tree (3.99× → 4.10×). To accommodate sampled candidates and an advanced verifier, we replace global pruning with allocation by a global threshold.
  • Cost-aware stopping (4.10× → 4.16×). Layer-wise allocation makes the frontier of each layer known as soon as that layer is allocated. Using draft q as the proxy for expected acceptance (the standard practice in dynamic trees, also used by EAGLE-3; see our response to Reviewer wLg1), the frontier approximates an upper bound on the expected gain of the next drafting layer. A stopping test then comes naturally: drafting stops once the estimated gain no longer justifies another layer.
  • Mixed proposals and advanced verifier (4.16× → 4.33×). These are the original goal of allocate-then-sample: slots are allocated first, so lossless sampling and verification can then be applied freely.

To answer your question directly: every step from 3.99× to 4.33× in Table 2 follows from allocate-then-sample. The allocation itself contributes +0.11×, the stopping rule that layer-wise allocation makes natural contributes +0.06×, and sampling plus verification contribute +0.17×. This last gain is unavailable in EAGLE-3, whose trees contain no sampled candidates (Appendix F). Under expand-then-prune, it can only be obtained through construction-specific control of the sampling–pruning coupling, as in RheoSampling.

A direct comparison is between Solvo and RheoSampling. Both perform sampling and advanced verification in dynamic trees; the difference is that Solvo is designed around sampled candidates through allocate-then-sample, whereas RheoSampling keeps the expand-then-prune paradigm and inserts sampled candidates into it. Solvo not only supports more flexible sampling and verification (RheoSampling supports only a specific proposal scheme), but also builds better trees: RheoSampling can degrade the tree structure at low temperatures and may even underperform EAGLE-3 (see the analysis of single degeneracy in Sec. 5.4 of the RheoSampling paper), while Solvo keeps a stable advantage over EAGLE-3 at low temperatures and at T=0. In our experiments, Solvo consistently achieves higher speedup than RheoSampling.

Thanks for your detailed feedback. I would like to keep the current rating for the paper.

Reviewer_wLg1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new task ICET for tracking one participant’s emotions in raw conversational video. It drops pre-segmentation and speaker-centric assumptions: a model gets the video and a target person, then must segment utterances, classify that person’s emotions consistently, and extract the utterances that cause them. The authors release EmoTrack, a Chinese–English benchmark with 22,585 utterance-level annotations over 21 hours, covering speakers and non-speakers. They benchmark several open and closed MLLMs. Performance drops on longer videos, audio matters most, simple visual fusion helps little, transcripts amplify both correct and noisy audio cues, and GRPO gives mixed gains. The paper provides a diagnostic benchmark: it shows where current MLLMs fail rather than offering a solution.

Critical strengths
  1. ICET is a sensible task formulation: it removes pre-segmentation and speaker-centric assumptions and proposed a new evaluation method including the benchmark of utterance segmentation, identity-consistent emotion classification, and emotion-cause extraction jointly on continuous video.
  2. EmoTrack provides a bilingual resource with 22,585 utterance-level annotations covering both speakers and non-speakers, which supports this setting and enables controlled comparisons.
  3. The benchmark of diverse MLLMs is broad and diagnostic rather than a single leaderboard: the modality ablations, transcript analysis, multi-party comparisons, and attention visualizations help locate where current models fail.
Critical weaknesses

The authors do not include identity tracking or consistency as a separate evaluation metric. In long, multi-person video sequences, identity tracking errors could affect the accuracy and reliability of the overall evaluation results. From the eq(1), ICET only evaluates utterance segmentation(USeg), emotion classification(EC), and emotion-cause extraction(ECE) for the designated target participant p. But what happen if some USeg result is correct but the predicted emotion is for another participant p+1? Figure 5 shows a clear drop in EC and cause extraction performance in multi-party scenarios, while USeg remains relatively stable. It is unclear whether this degradation comes from identity confusion or difficulties in emotion reasoning. In long videos with complex multi-person interactions, identity confusion could affect both emotion classification and emotion-cause extraction. This raises concerns about the reliability of ICET's evaluation results in such scenarios, and whether the benchmark can adequately assess models under complex multi-person interactions.

Questions for authors
  1. In page8 and fig. 4, author makes a conclusion that the utterance boundary errors are no longer the main bottleneck for EC and ECE once a coarse boundary is achieved. However, varying the IoU threshold does not actually change the predicted utterance boundaries. On one hand, the IoU vary is not directly mapping the error rate of USeg, right? On the other hand, will it be too limited by the samples used in this benchmark?
  2. Fig. 5 shows that EC and ECE performance drops in multi-party scenarios, while USeg remains relatively stable. How much of this degradation is caused by identity confusion rather than difficulties in emotion recognition or cause extraction? Could the authors provide a separate analysis for it?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1 of 3)

Note: The review shown on this page (Reviewer wLg1) concerns Submission 39547, not our paper. The review intended for our paper appears on the Submission 39547 page (as Reviewer W2i3); below we respond to that review.

Thank you for your review! We are glad to answer your questions.

Q1. Threshold sensitivity at T=1, and how should the threshold be chosen for a new model?

Using Llama3.1-8B, the averages over the six datasets (480 prompts) in our paper are (TPS):

θT=0T=1
0.002200.63181.46
0.005208.86186.98
0.010 (paper setting)215.45184.31
0.015213.74183.56
0.020208.41178.30

The default θ = 0.01 in our paper is transferred directly from T=0 to T=1, without retuning for T=1. On sensitivity: with this direct transfer, TPS is only 1.43% below the optimum. Conversely, retuning θ for T=1 would yield some additional gain.

On how θ should be chosen for a new model, there is a simple recipe: after obtaining a new model, determine θ on a single dataset. When transferred to other datasets, the gap to the optimum is small, and θ may well already be optimal.

As an example, we use MT-Bench-101, which is not among the six datasets in our paper. We take 100 samples and test only the first turn:

θT=0 TPST=1 TPS
0.002198.51184.94
0.005217.61190.54
0.010228.48196.06
0.015226.99190.76
0.020223.09188.46

At both T=0 and T=1, the optimum is 0.01, the same θ we actually use.

Q2. How well does the scoring method reflect actual acceptance?

Using draft q as the proxy for acceptance is standard practice in current dynamic trees: EAGLE-3 and TALON, both compared in our paper, build their dynamic trees on this proxy, and we follow the same common design.

To show this correlation directly, we measured the relationship between structural scores and actual acceptance rates on Llama3.1-8B over MT-Bench, GSM8K, and HumanEval:

Score bucketT=0 mean structural scoreT=0 mean acceptanceT=1 mean structural scoreT=1 mean acceptance
[0.0, 0.1)2.94%2.65%2.89%2.83%
[0.1, 0.2)14.03%13.49%14.02%11.09%
[0.2, 0.3)24.46%23.92%24.42%18.96%
[0.3, 0.4)34.60%33.65%34.64%26.39%
[0.4, 0.5)44.75%43.01%44.74%34.16%
[0.5, 0.6)54.75%51.16%54.81%42.65%
[0.6, 0.7)64.99%60.68%64.89%51.63%
[0.7, 0.8)74.97%71.39%75.05%62.38%
[0.8, 0.9)85.17%81.91%85.18%73.40%
[0.9, 1.0]97.52%95.99%97.48%93.93%

At T=0, the structural score closely matches the actual acceptance rate (unweighted mean absolute gap across the ten buckets: 2.03 percentage points). At T=1, it overestimates acceptance by 8.07 percentage points on average, but at both temperatures the mean acceptance rate increases monotonically with the structural score, supporting its use for prioritizing candidates.

An ideal tree construction would require the target model's probabilities, but the target forward pass has not yet been run when the tree is built, so true acceptance rates are unavailable. Under this constraint, the draft probability is already a reliable signal.

To answer your question: the structural score reflects actual acceptance well, which is exactly why it is the mainstream choice.

AuthorsOpenReview ↗

Official Comment by Authors (2 of 3)

W2. Batch serving

We ran batch serving experiments and compared against TALON in our response to Reviewer aXHK. For your convenience, we repeat the results here.

We ran continuous serving with SGLang using Llama3.1-8B on MT-Bench. "Batch" denotes the running batch size. TPS:

BatchT=0 TALONT=0 SolvoT=1 TALONT=1 Solvo
2565.87581.48423.09454.63
4925.27957.56700.21771.55
81261.661297.55950.901034.57
161502.361544.231095.481228.20
321731.431793.561275.131427.54

TALON uses μ = 0.03, the single-request optimum from our sweep in Table 9. Solvo reuses the parameters from the paper without any change.

Since TALON's optimal μ may shift with batch size, we additionally swept μ at batch = 8:

TALON μT=0 TPST=1 TPS
0.011229.04950.06
0.021245.74950.51
0.031261.66950.90
0.041259.40933.99
0.051265.20900.63
0.061262.95922.24
0.071254.27916.03

At T=1, TALON's optimal μ is still 0.03. At T=0 the optimum is 0.05, only marginally above 0.03. Even with this optimum, TALON reaches 1265.20, still below Solvo's 1297.55.

Under batch serving, Solvo's advantage remains stable.

W1. How much of the improvement can be attributed specifically to the proposed allocate-then-sample design?

In one sentence: every step from 3.99× to 4.33× in Table 2 follows from allocate-then-sample. The allocation itself contributes +0.11×, the stopping rule that layer-wise allocation makes natural contributes +0.06×, and sampling plus verification contribute +0.17×. This last gain is unavailable in EAGLE-3, whose trees contain no sampled candidates (Appendix F). Under expand-then-prune, it can only be obtained through construction-specific control of the sampling–pruning coupling, as in RheoSampling.

We detail the sources of improvement below.

Effect of sampled candidates on acceptance

We first use a 2-candidate example to show how adding a sampled candidate affects the acceptance rate of speculative decoding. On Llama3.1-8B-Instruct and MT-Bench, we compare two candidate sets:

  1. the top-2 tokens of the draft distribution q;
  2. the top-1 token of q, plus one token sampled from the residual of q.

Both rates below are expected acceptance rates computed on the same trajectories:

Candidate setAcceptance rate
Top-278.72%
Top-1 + 1 residual sample85.27%

Adding a sampled candidate clearly raises the acceptance rate. Intuitively, this is the advantage of mixed proposals: deterministic candidates capture the acceptance in the head of the distribution, while a residual sample explores the draft–target overlap in the tail, which pure Top-K cannot do. Mixed proposals and the optimal transport (OT) view of speculative sampling are analyzed in detail in Towards Optimal Multi-Draft Speculative Decoding (Hu et al., ICLR 2025).

Effect of the verifier on acceptance length

The example above shows that single-layer acceptance depends on the proposal scheme. Given a fixed proposal scheme (e.g., top-1 + 1 sample everywhere), the verifier also affects acceptance.

Take UniVer (Weng et al., NeurIPS 2026) as an example. For tree decoding, UniVer allows probability allocation across layers, whereas the method of Hu et al. only allocates within a layer (single-layer OT). Both verifiers are lossless, but UniVer has more freedom in allocating probability mass, so it achieves a longer acceptance length even on exactly the same tree with exactly the same sampled nodes.

We use a binary tree with 5 draft layers below the root, with top-1 + 1 sample at every node, on the 80 MT-Bench prompts. Both values are expected acceptance lengths on the same trees:

VerifierAcceptance length
Single-layer OT4.160
UniVer4.263

UniVer achieves a clearly longer acceptance length than per-layer allocation. This is the gain an advanced verifier brings.

Note that an advanced verifier is effective only when the tree contains sampled candidates. Without them, the verifier cannot exploit any flexibility in the probability allocation: from the OT perspective the feasible set is fixed, and only token matching remains. As reported in Appendix F of our paper, applying UniVer to EAGLE-3 brings no acceptance gain.

(continued in the next comment)

AuthorsOpenReview ↗

Official Comment by Authors (3 of 3)

W1 (continued)

What allocate-then-sample enables

The key point is that neither gain above (mixed proposals and UniVer) is available by default in dynamic trees such as EAGLE-3. Sampled candidates cannot simply be added to an expand-then-prune dynamic tree: as discussed in our paper, if sampled nodes are present without special treatment, global pruning breaks losslessness. In short, EAGLE-3's expand-then-prune effectively forces Top-K candidates, so the gains of sampling and advanced verification cannot be obtained there.

Our question was therefore: if the tree must accommodate sampled candidates and an advanced verifier from the outset, how should it be built? rather than how can sampled candidates be inserted into EAGLE-3's expand-then-prune construction? (the latter is what RheoSampling does). This is why we call Solvo a sampling-centric framework: allocate-then-sample is what makes flexible sampling and advanced verification straightforward to realize in a dynamic tree, and thereby collects these gains.

Table 2 of our paper:

ConfigurationτSR
EAGLE-35.083.99×
+ Threshold tree5.244.10×
+ Cost-aware stopping5.234.16×
+ Mixed proposals5.344.21×
+ Advanced verifier5.504.33×

All four steps follow from the allocate-then-sample design:

  • Threshold tree (3.99× → 4.10×). To accommodate sampled candidates and an advanced verifier, we replace global pruning with allocation by a global threshold.
  • Cost-aware stopping (4.10× → 4.16×). Layer-wise allocation makes the frontier of each layer known as soon as that layer is allocated. Using draft q as the proxy for expected acceptance (the standard practice in dynamic trees, also used by EAGLE-3; see Q2 above), the frontier approximates an upper bound on the expected gain of the next drafting layer. A stopping test then comes naturally: drafting stops once the estimated gain no longer justifies another layer.
  • Mixed proposals and advanced verifier (4.16× → 4.33×). These are the original goal of allocate-then-sample: slots are allocated first, so lossless sampling and verification can then be applied freely.

To answer your question directly: every step from 3.99× to 4.33× in Table 2 follows from allocate-then-sample. The allocation itself contributes +0.11×, the stopping rule that layer-wise allocation makes natural contributes +0.06×, and sampling plus verification contribute +0.17×. This last gain is unavailable in EAGLE-3, whose trees contain no sampled candidates (Appendix F). Under expand-then-prune, it can only be obtained through construction-specific control of the sampling–pruning coupling, as in RheoSampling.

A direct comparison is between Solvo and RheoSampling. Both perform sampling and advanced verification in dynamic trees; the difference is that Solvo is designed around sampled candidates through allocate-then-sample, whereas RheoSampling keeps the expand-then-prune paradigm and inserts sampled candidates into it. Solvo not only supports more flexible sampling and verification (RheoSampling supports only a specific proposal scheme), but also builds better trees: RheoSampling can degrade the tree structure at low temperatures and may even underperform EAGLE-3 (see the analysis of single degeneracy in Sec. 5.4 of the RheoSampling paper), while Solvo keeps a stable advantage over EAGLE-3 at low temperatures and at T=0. In our experiments, Solvo consistently achieves higher speedup than RheoSampling.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion

We thank the reviewers for their constructive feedback. They highlighted Solvo’s allocate-then-sample design, the proof of losslessness under adaptive tree growth and cost-aware stopping, and the empirical evaluation across five target models and six tasks.

Note on Reviewer wLg1 and review status. The official review fields under wLg1 still contain a review of another submission. Reviewer wLg1 posted their review of our paper as an official comment on this page (rating 3). Following our response, neither 8yyX nor aXHK raised further questions, and both maintained their ratings of 3.

  • Source of improvement (8yyX, wLg1). Better proposals raise acceptance: mixing deterministic candidates with a residual sample captures draft–target overlap that Top-K misses (78.72% → 85.27% in our two-candidate example). Solvo is designed around sampling (allocate-then-sample), so it can add sampled nodes to a dynamic tree without losing losslessness. Table 2 decomposes the gain: the threshold tree itself raises speedup from 3.99× to 4.10×, cost-aware stopping, a natural companion of the layer-wise threshold tree, brings it to 4.16×, and sampling plus verification bring it to 4.33×. The last gain does not exist without sampled candidates (Appendix F: UniVer on EAGLE-3’s deterministic trees yields no acceptance gain); this is precisely what allocate-then-sample enables.
  • Continuous serving (aXHK, wLg1). In new SGLang experiments on Llama 3.1-8B and MT-Bench, Solvo outperformed TALON at all five running batch sizes tested (2–32); sweeping TALON’s μ at batch size 8 did not change this result.
  • Hardware and workload transfer (aXHK). On the RTX 3090, reusing the RTX PRO 6000 stopping-cost calibration gave throughput within 0.4% of recalibrating on the 3090 at both temperatures. A separate threshold sweep on MT-Bench-101, an independent dataset, selected the paper’s default θ = 0.01.
  • Threshold sensitivity (wLg1). Across the six evaluation datasets, the default θ, transferred from T=0 without retuning, achieved throughput within 1.43% of the best measured at T=1.
  • Score reliability (wLg1). At both temperatures, the bucketed results show that the structural score tracks actual acceptance, supporting its use for tree construction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.