acceptodds
Under review as a conference paper at ICLR 2027

FlashMTP: Flash Multi-Token Prediction

Abstract

The autoregressive nature of large language models (LLMs) fundamentally constrains inference speed. Standard autoregressive decoding generates one token per forward pass, and its sequential dependency prevents parallel prediction of future tokens. The key challenge is to exploit information to predict multiple future tokens while effectively allocating candidate tokens per forward pass. To address this problem, we propose FlashMTP (Flash Multi-Token Prediction), a plug-and-play, training-free adaptive multi-token prediction method. FlashMTP expands the effective candidate set using the outputs at all evaluated candidate positions and dynamically allocates the candidate tokens of each pass according to path acceptance probabilities. By improving the coverage of future tokens and the utilization of candidate tokens, FlashMTP accepts more consecutive tokens per pass. FlashMTP is lossless under exact arithmetic in the sense that greedy decoding yields exactly the autoregressive output, and stochastic decoding samples every token from the model's conditional distribution rather than from the candidate set. Empirical evaluations across multiple generation benchmarks demonstrate that FlashMTP improves mean accepted tokens per decoding step (MAT) and generation throughput, with speedups of up to over vanilla autoregressive decoding. This work paves the way for efficient LLM inference through adaptive multi-token prediction. Code will be released upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-token prediction, inference acceleration, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fjwEOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes FlashMTP, a training-free method to generate several tokens in one model pass. It reuses successor predictions from evaluated candidates and allocates the next pass's candidates using estimated path acceptance rates. The paper reports more accepted tokens and faster generation than LookAhead and LogitSpec on three benchmarks and eight model settings.

Critical strengths
  1. Reusing predictions from evaluated candidate positions is a useful idea. The adaptive allocation rule gives these predictions a clear role in the next pass.

  2. The decoding procedure is described clearly. The appendix explains why it preserves greedy outputs and sampling distributions under exact arithmetic.

  3. The speedups appear across several model sizes and tasks. The component ablations show that both successor reuse and allocation contribute to the result.

Critical weaknesses
  1. The main comparison does not separate proposal quality from candidate budget. FlashMTP uses up to 63 candidate positions per pass, while the reported LookAhead setup cannot accept more than four tokens per pass. A matched-budget comparison is needed to judge the source of the speedup.

  2. The practical losslessness claim is not quantified. The main speed tests use float16, but Appendix C reports only that float16 outputs are almost identical to autoregressive outputs. Exact-match rates and task-quality differences are needed for the setting used in Table 1.

  3. The evaluation uses batch size one and does not report peak memory or how cost changes with context length. A large candidate tree and saved successor index may be expensive in other serving settings. This limits the deployment claim.

Questions for authors

Please address the above weaknesses and answer the following questions.

  1. What candidate budgets and maximum depths were used for LookAhead and LogitSpec? Can you compare the methods at a matched number of candidate positions or matched per-pass latency?

  2. For float16 decoding, what fraction of full outputs exactly matches autoregressive decoding? Do the first token differences affect HumanEval, GSM8K, or Spec-Bench scores?

  3. What are the peak GPU memory and index-memory costs as context length grows? Does the speedup hold when more than one request is decoded in a batch?

8 comments

Response to Reviewer fjwE (Part 1 / 6)

We thank the reviewer for the positive assessment and constructive suggestions. We respond to each comment as follows and sincerely hope that our responses could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.

W1: The main comparison does not separate proposal quality from candidate budget. FlashMTP uses up to 63 candidate positions per pass, while the reported LookAhead setup cannot accept more than four tokens per pass. A matched-budget comparison is needed to judge the source of the speedup.

A-W1: We thank the reviewer for raising this question. The LookAhead setting W=7, N=5, G=7 is the configuration given in the official LookAhead implementation, and with N=5 each guess has four tokens. To separate the quality of the candidates from their number, we ran each method with several numbers of rows, including 56 and 64 rows for all three methods. Tables R1 and R2 show the results. We will add them to the appendix of the revised paper.

Table R1: Matched candidate counts.

RowsCandidate tokensMethodHumanEval MATHumanEval SpeedupGSM8K MATGSM8K SpeedupSpec-Bench MATSpec-Bench Speedup
5655LookAhead (7,5,7)2.222.03×2.021.69×1.781.46×
5655LogitSpec2.742.50×2.141.80×1.901.56×
5655FlashMTP4.573.71×3.462.56×3.062.15×
6463LookAhead (8,5,8)2.272.06×2.071.81×1.821.50×
6463LogitSpec2.772.46×2.171.70×1.931.57×
6463FlashMTP4.713.78×3.552.57×3.152.17×

Table R2: MAT and speedup with different numbers of candidate tokens per forward pass on Qwen2.5-7B.

Method, RowsCandidate tokensHumanEval MATHumanEval SpeedupGSM8K MATGSM8K SpeedupSpec-Bench MATSpec-Bench Speedup
LookAhead (7,5,7), 56552.222.03×2.021.69×1.781.46×
LookAhead (8,5,8), 64632.272.06×2.071.81×1.821.50×
LookAhead (7,7,7), 84832.442.09×2.111.68×1.871.38×
LogitSpec, 32312.592.46×1.951.70×1.781.44×
LogitSpec, 56552.742.50×2.141.80×1.901.56×
LogitSpec, 64632.772.46×2.171.70×1.931.57×
FlashMTP, 16153.553.12×2.602.11×2.341.84×
FlashMTP, 32314.103.62×3.052.44×2.732.08×
FlashMTP, 56554.573.71×3.462.56×3.062.15×
FlashMTP, 64634.713.78×3.552.57×3.152.17×

At the same number of rows, FlashMTP accepts 1.6–2.1 times as many tokens per pass as LookAhead and LogitSpec. Allowing LookAhead to accept up to six tokens per step (84 rows) raises its MAT by at most 0.22, and FlashMTP with only 16 rows (15 candidate tokens) is faster than every LookAhead and LogitSpec setting on all three benchmarks.

LookAhead and LogitSpec fix the shape of a draft in advance: an n-gram window with a preset guess length, or a retrieved segment below each ranked next-next token. Adding positions mostly adds more guesses of the same kind, so their MAT saturates. FlashMTP proposes the candidates of each pass for the specific path below every admitted candidate, using the distributions the model computed at all evaluated positions, and gives each additional row to the path most likely to be accepted according to the acceptance it has measured, so its MAT keeps growing with the number of rows. The speedup therefore comes from the quality and allocation of the candidates, not from evaluating more candidates.

Response to Reviewer fjwE (Part 2 / 6)

W2: The practical losslessness claim is not quantified. The main speed tests use float16, but Appendix C reports only that float16 outputs are almost identical to autoregressive outputs. Exact-match rates and task-quality differences are needed for the setting used in Table 1.

A-W2: We thank the reviewer for raising this question. Table R3 reports sequence agreement and task quality for Qwen2.5-7B. We will add it to the appendix of the revised paper.

Table R3: Output agreement with AutoRegressive decoding and task quality on Qwen2.5-7B.

MethodHumanEval identical to AutoRegressiveHumanEval pass@1GSM8K identical to AutoRegressiveGSM8K accuracy
AutoRegressive164/16482.3%300/30088.3%
LookAhead(f16)156/164(95.1%)82.3%277/300 (92.3%)88.3%
FlashMTP(f16)159/164 (97.0%)82.3%276/300(92.0%)88.3%

Theoretically, due to the verification mechanism, FlashMTP does not change the output of greedy decoding. Under single precision (FP32), the output of FlashMTP is identical to that of AutoRegressive decoding. However, due to numerical precision, under half precision (FP16), the output of FlashMTP is not completely identical to that of AutoRegressive decoding.

Other lossless acceleration methods in this field behave in the same way. For example, under half-precision (FP16) inference, 8 of the 164 LookAhead answers on HumanEval are inconsistent with the AutoRegressive output.

Response to Reviewer fjwE (Part 3 / 6)

W3: The evaluation uses batch size one and does not report peak memory or how cost changes with context length. A large candidate tree and saved successor index may be expensive in other serving settings. This limits the deployment claim.

A-W3: We thank the reviewer for raising these questions.

(i) Batch size.

Current mainstream methods in this field are mainly evaluated with a batch size of 1. For example, LookAhead and LogitSpec both operate with a batch size of 1. Supporting larger batch sizes requires substantial modifications to the underlying codebase.

(ii) Peak memory.

Table R4 reports the peak GPU memory on HumanEval. We will add it to the appendix of the revised paper.

Table R4: Peak GPU memory on HumanEval with Qwen2.5-7B (nvidia-smi, MiB).

Peak GPU memory (MiB)
Autoregressive16,063
LookAhead15,963
LogitSpec15,963
FlashMTP15,963

All methods peak at about 16.0 GB. nvidia-smi includes memory cached by PyTorch's allocator, so differences below 1% are within allocator granularity and do not reflect real savings. A precise measurement of allocated memory shows that a forward pass over 64 rows uses at most 0.03 GB more than one row. The candidate tree therefore adds no meaningful GPU memory.

(iii) Cost as context length grows.

Table R5 reports how the cost changes with context length (Qwen2.5-7B, summarization inputs, 256 new tokens). We will add it to the appendix of the revised paper.

Table R5: Cost of decoding as the context grows.

Context length1k2k4k
AR, time per step (ms)15.616.216.9
FlashMTP, time per step (ms)21.926.132.6
One forward pass, 1 row (ms)16.716.416.3
One forward pass, 64 rows (ms)17.819.623.2
Peak allocated GPU memory, 1 row / 64 rows (GB)14.32 / 14.3414.38 / 14.3914.48 / 14.50
Successor index (host memory)≤ 5.2 MB≤ 5.2 MB≤ 5.2 MB

(iv) The saved successor index.

The index lives in host memory and is bounded by 8,192 keys with 8 next tokens each, about 5.2 MB when full, regardless of context length. With many concurrent sessions, it grows linearly with the number of sessions and can be capped or cleared per session.

Reviewer_edjfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FlashMTP is a training-free, draft-model-free decoding method that combines successor reuse with adaptive candidate-tree construction. It stores top-ranked successor tokens from all evaluated positions, including rejected branches, and retrieves proposals using token and token-pair keys together with context matching and last-logit candidates. It estimates conditional acceptance rates online and uses products of these estimates to prioritize candidate paths under a fixed token budget. The paper provides an exact-arithmetic losslessness argument for greedy and stochastic decoding. Experiments cover eight model configurations on HumanEval, GSM8K, and Spec-Bench, primarily using batch size one and FP16 on an A800 GPU. The reported speedups over autoregressive decoding range from approximately 2.13× to 3.77× in the main table. Ablations support the usefulness of the proposed components.

Critical strengths

S1. A coherent and practically appealing adaptive decoding design. FlashMTP combines candidate reuse with online acceptance feedback to allocate a limited verification budget across candidate paths. Path-level scoring accounts for the requirement that all preceding candidates must be accepted. The method requires neither additional model training nor a separate draft model, making it potentially useful for deployments where these requirements would be costly. S2. Consistent reported improvements across the evaluated settings. FlashMTP outperforms both included baselines in MAT and speedup across all model–benchmark combinations in Table 1. The evaluation includes reasoning and non-reasoning configurations, and the ablations show that removing each major component reduces performance. These results provide encouraging evidence that the combined design is effective in the tested low-batch setting.

Critical weaknesses

W1. The evaluation does not establish the incremental contribution over the closest prior work. Token Recycling already reuses successor predictions from evaluated draft positions, including rejected branches, stores token-to-successor mappings, and recursively constructs trees for verification. FlashMTP cites this work but does not include it as an experimental baseline. Consequently, the comparison with LookAhead and LogitSpec does not isolate the benefit of adaptive allocation over an existing recycling-based approach. Removing rejected-branch observations from FlashMTP is not a substitute for evaluating the complete prior method. A controlled comparison with Token Recycling is important to assessing the significance of the contribution. W2. Important implementation and validation details remain insufficiently specified. Appendix E lists mechanisms such as sibling normalization, candidate-set coverage estimation, supervision of unselected candidates, and multi-source evidence aggregation, but Algorithms 1–5 do not fully specify these mechanisms. In particular, Algorithm 5 describes ordered deduplication without specifying the multi-source probability aggregation listed in Table 7. This makes the implementation behind the main results difficult to reconstruct. Furthermore, the main experiments use FP16, while Table 4 reports only “almost” exact agreement with autoregressive decoding. The authors correctly distinguish this from exact-arithmetic losslessness, but quantitative disagreement rates and task-quality measurements are still needed to establish the practical effect of these differences.

Questions for authors

Q1 — Contribution and comparison. Can you provide a comparison with Token Recycling using the same target models, candidate-token budgets, evaluation protocol, and implementation backend? How much of the improvement remains when adaptive allocation is added to a recycling-based baseline? Q2 — Algorithm specification. Which mechanisms in Appendix E are enabled in the main experiments? Please specify the sibling normalization and coverage estimates, multi-source aggregation, supervision of unselected candidates, and construction of the marginal forecast. How should the ordered deduplication in Algorithm 5 be reconciled with Table 7? Q3 — Practical output preservation. For the FP16 configuration used in the main table, what fraction of examples differ from autoregressive decoding? Please quantify sequence agreement and report HumanEval pass@1 and GSM8K accuracy for the two decoding methods. Aggregate output-length ratios do not establish token agreement or equivalent task quality. Q4 — Evaluation controls. What candidate budgets and tree configurations were used for the baselines, and how were they selected? Were persistent successor indices and acceptance statistics reset between benchmarks, models, and repeated runs? How sensitive are the results to clearing state between requests or changing request order?

7 comments

Response to Reviewer edjf (Part 1 / 6)

We thank the reviewer for the careful review and constructive suggestions. We respond to each comment as follows and sincerely hope that our responses could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.

W1: The evaluation does not establish the incremental contribution over the closest prior work ... A controlled comparison with Token Recycling is important to assessing the significance of the contribution.

A-W1: We thank the reviewer for raising this question. To address your concerns, we have added Token Recycling [1] as a baseline. Table R1 shows the results. Both methods evaluate 64 rows per forward pass (the root and 63 candidate tokens). We will add it to the appendix of the revised paper.

Table R1: Token Recycling and FlashMTP with the same number of candidate tokens per forward pass (64 rows). Speedups are relative to autoregressive decoding in the same session.

ModelMethodHumanEval MATHumanEval SpeedupGSM8K MATGSM8K SpeedupSpec-Bench MATSpec-Bench Speedup
Qwen2.5-7BTokenRecycling2.842.36×2.882.39×2.471.97×
Qwen2.5-7BFlashMTP4.713.78×3.552.57×3.152.17×
Qwen2.5-14BTokenRecycling2.522.05×2.641.91×2.351.74×
Qwen2.5-14BFlashMTP3.322.64×3.352.22×2.982.06×

Table R1 demonstrates that FlashMTP outperforms Token Recycling. On MAT it improves by 33.7% on average and by up to 66.0%. On speedup it improves by 23.6% on average and by up to 60.2%.

The gain comes from two limitations of Token Recycling that FlashMTP removes.

  • Coverage. Token Recycling proposes, below each node, the top tokens last seen after that single token, so the second and third future tokens are not conditioned on the path leading to them. FlashMTP stores the top tokens of every evaluated position together with their confidence, under one-token and two-token keys, adds context matches, and queries all of these sources again below every admitted candidate. Deeper candidates are therefore proposed for the specific path they extend.

  • Utilization. Token Recycling fills the same static tree in every step. FlashMTP measures how often each kind of candidate is accepted at each depth and gives the candidate tokens of each pass to the paths most likely to be accepted.

[1] Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling, Luo et al., ACL 2025.

Response to Reviewer edjf (Part 2 / 6)

W2: Important implementation and validation details remain insufficiently specified ... The authors correctly distinguish this from exact-arithmetic losslessness, but quantitative disagreement rates and task-quality measurements are still needed to establish the practical effect of these differences.

A-W2: We thank the reviewer for raising these questions.

(i) Specification.

The mechanisms in Appendix E are all enabled in the main experiments. In short:

  • Sibling normalization and coverage estimation. Candidates below the same parent are scored together, and their acceptance rates are rescaled to add up to the measured probability that the model's token is among them.
  • Supervision of unselected candidates. After each pass, every proposed sibling, including those that received no row, is labelled as matched or not, and these labels update the rates.
  • Multi-source aggregation. A token proposed by several sources is kept once, with the union of its sources and the highest of their probabilities.
  • Marginal forecast. The next-token lists of the rejected siblings at the last accepted position are combined into up to 8 extra candidates for the next root.

(ii) Algorithm 5 and Table 7.

Algorithm 5 shows that a token enters the candidate list once. In the implementation, a repeated token is merged into the existing entry: its source is added to the entry's set of sources, and its probability becomes the highest among its sources, which is what Table 7 describes.

(iii) Float16 agreement and task quality.

Table R2 reports sequence agreement and task quality for Qwen2.5-7B. We will add it to the appendix of the revised paper.

Table R2: Output agreement with AutoRegressive decoding and task quality on Qwen2.5-7B.

MethodHumanEval identical to AutoRegressiveHumanEval pass@1GSM8K identical to AutoRegressiveGSM8K accuracy
AutoRegressive164/16482.3%300/30088.3%
LookAhead(f16)156/164(95.1%)82.3%277/300 (92.3%)88.3%
FlashMTP(f16)159/164 (97.0%)82.3%276/300(92.0%)88.3%

Theoretically, due to the verification mechanism, FlashMTP does not change the output of greedy decoding. Under single precision (FP32), the output of FlashMTP is identical to that of AutoRegressive decoding. However, due to numerical precision, under half precision (FP16), the output of FlashMTP is not completely identical to that of AutoRegressive decoding.

Other lossless acceleration methods in this field behave in the same way. For example, under half-precision (FP16) inference, 8 of the 164 LookAhead answers on HumanEval are inconsistent with the AutoRegressive output.

Response to Reviewer edjf (Part 3 / 6)

Q1: Contribution and comparison. Can you provide a comparison with Token Recycling using the same target models, candidate-token budgets, evaluation protocol, and implementation backend? How much of the improvement remains when adaptive allocation is added to a recycling-based baseline?

A-Q1: We thank the reviewer for raising this question. As in A-W1, we have added Token Recycling [1] as a baseline. Table R3 shows the results. Both methods evaluate 64 rows per forward pass (the root and 63 candidate tokens). We will add it to the appendix of the revised paper.

Table R3: Token Recycling and FlashMTP with the same number of candidate tokens per forward pass (64 rows). Speedups are relative to autoregressive decoding in the same session.

ModelMethodHumanEval MATHumanEval SpeedupGSM8K MATGSM8K SpeedupSpec-Bench MATSpec-Bench Speedup
Qwen2.5-7BTokenRecycling2.842.36×2.882.39×2.471.97×
Qwen2.5-7BFlashMTP4.713.78×3.552.57×3.152.17×
Qwen2.5-14BTokenRecycling2.522.05×2.641.91×2.351.74×
Qwen2.5-14BFlashMTP3.322.64×3.352.22×2.982.06×

Table R3 demonstrates that FlashMTP outperforms Token Recycling. On MAT it improves by 33.7% on average and by up to 66.0%. On speedup it improves by 23.6% on average and by up to 60.2%.

The gain comes from two limitations of Token Recycling that FlashMTP removes.

  • Coverage. Token Recycling proposes, below each node, the top tokens last seen after that single token, so the second and third future tokens are not conditioned on the path leading to them. FlashMTP stores the top tokens of every evaluated position together with their confidence, under one-token and two-token keys, adds context matches, and queries all of these sources again below every admitted candidate. Deeper candidates are therefore proposed for the specific path they extend.

  • Utilization. Token Recycling fills the same static tree in every step. FlashMTP measures how often each kind of candidate is accepted at each depth and gives the candidate tokens of each pass to the paths most likely to be accepted.

To measure how much of the improvement remains, we added adaptive allocation to the original Token Recycling code. Table R4 shows the results. We will add it to the appendix of the revised paper.

Table R4: Token Recycling with adaptive allocation on Qwen2.5-7B (64 rows for all methods).

MethodHumanEval MATHumanEval SpeedupGSM8K MATGSM8K SpeedupSpec-Bench MATSpec-Bench Speedup
TokenRecycling2.842.36×2.882.39×2.471.97×
TokenRecycling + adaptive allocation2.942.51×2.912.41×2.511.98×
FlashMTP4.713.78×3.552.57×3.152.17×

Table R4 demonstrates the advantage of FlashMTP over Token Recycling with adaptive allocation. The improvement that remains for FlashMTP over this stronger baseline is +50.7%, +6.9% and +9.9% speedup, and +60.4%, +21.8% and +25.6% MAT.

Most of FlashMTP's gain is therefore not explained by adding adaptive allocation to Token Recycling. It comes from candidates that are conditioned on the path below each admitted candidate and drawn from sources of very different reliability, combined with statistics that separate these sources by depth and evidence strength.

[1] Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling, Luo et al., ACL 2025.

Reviewer_GrixOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary
  1. FlashMTP involves two main ideas in regular speculative decoding setup a): it does smart reuse of rejected candidates since computations have already taken place and b): it learn to dynamically allocate candidates across width and depth of candidate "beams"
  2. The dynamicity comes from the observation that the early candidate tokens in sequence are more sensitive to if longer candidate will be accepted or not and thus it make sense to have higher coverage i.e. wider beam speculation on those tokens.
  3. Empirically at best it reports 3.77x speedup over regular autoreg decoding HumanEval, using an NVIDIA A800.
Critical strengths
  1. The paper make sense and the principles and intuition is well grounded.
  2. Paper applies it methods to empirically assess multiple model families with decent coverage of benchmark.
Critical weaknesses

There are multiple missing baselines and insufficient details to establish fair comparisons with the two included baselines, LookAhead and LogitSpec.

  1. Token Recycling [1] already proposes reusing predictions from rejected candidate positions for future speculative decoding. Although it uses static BFS-based tree construction, it is the most direct training-free baseline and is missing.
  2. Goose [2] already studies adaptive allocation of speculative tree width/depth based on candidate-source acceptance rates. This directly overlaps with FlashMTP's second main idea, but no empirical comparison is provided.
  3. FlashMTP uses 63 candidate tokens per pass while the candidate budgets and hyperparameters used for LookAhead and LogitSpec are not clearly reported.
  4. The evaluated LookAhead configuration allows at most 4 accepted tokens per step, whereas FlashMTP can accept longer sequences. This potentially disadvantages LookAhead, without any justification or hyperparameter sweep showing that its configuration is optimal.
  5. Strong training-based speculative decoding baselines, particularly EAGLE-2 [3] and EAGLE-3 [4], are missing. Although these require additional training, they are important for establishing practical competitiveness against existing speculative decoding methods.
  6. Fair comparisons should include matched candidate budgets, independently tuned hyperparameters, and budget/depth sweeps. Additionally, end-to-end throughput, verification cost, candidate construction overhead, and memory usage should be reported. Equal candidate counts alone may not imply equal compute across methods.

[1] Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling, Luo et al., ACL 2025.

[2] Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding, Jin et al., COLM 2026.

[3] EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees, Li et al., EMNLP 2024.

[4] EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test, Li et al., NeurIPS 2025.

Questions for authors

Can you please look at the weaknesses and add as much experimental evidence needed to support the fair comparison please? Currently evaluation is severely incomplete to be able to make any concrete claim about FlashMTP.

7 comments

Response to Reviewer Grix (Part 1 / 6)

We thank the reviewer for the careful review and constructive suggestions. We respond to each comment as follows and sincerely hope that our responses could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.

W1: Token Recycling [1] already proposes reusing predictions from rejected candidate positions for future speculative decoding. Although it uses static BFS-based tree construction, it is the most direct training-free baseline and is missing.

A-W1: We thank the reviewer for raising this question. To address your concerns, we have added Token Recycling [1] as a baseline. Table R1 shows the results. Both methods evaluate 64 rows per forward pass (the root and 63 candidate tokens). We will add it to the appendix of the revised paper.

Table R1: Token Recycling and FlashMTP with the same number of candidate tokens per forward pass (64 rows). Speedups are relative to autoregressive decoding in the same session.

ModelMethodHumanEval MATHumanEval SpeedupGSM8K MATGSM8K SpeedupSpec-Bench MATSpec-Bench Speedup
Qwen2.5-7BTokenRecycling2.842.36×2.882.39×2.471.97×
Qwen2.5-7BFlashMTP4.713.78×3.552.57×3.152.17×
Qwen2.5-14BTokenRecycling2.522.05×2.641.91×2.351.74×
Qwen2.5-14BFlashMTP3.322.64×3.352.22×2.982.06×

Table R1 demonstrates that FlashMTP outperforms Token Recycling. On MAT it improves by 33.7% on average and by up to 66.0%. On speedup it improves by 23.6% on average and by up to 60.2%.

The gain comes from two limitations of Token Recycling that FlashMTP removes.

  • Coverage. Token Recycling proposes, below each node, the top tokens last seen after that single token, so the second and third future tokens are not conditioned on the path leading to them. FlashMTP stores the top tokens of every evaluated position together with their confidence, under one-token and two-token keys, adds context matches, and queries all of these sources again below every admitted candidate. Deeper candidates are therefore proposed for the specific path they extend.

  • Utilization. Token Recycling fills the same static tree in every step. FlashMTP measures how often each kind of candidate is accepted at each depth and gives the candidate tokens of each pass to the paths most likely to be accepted.

[1] Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling, Luo et al., ACL 2025.

Response to Reviewer Grix (Part 2 / 6)

W2: Goose [2] already studies adaptive allocation of speculative tree width/depth based on candidate-source acceptance rates. This directly overlaps with FlashMTP's second main idea, but no empirical comparison is provided.

A-W2: We thank the reviewer for raising this question. We will cite and discuss Goose [2]. Goose and FlashMTP differ in two respects.

  • Goose fixes source roles. FlashMTP allocates per candidate. Goose uses two sources, n-gram matches copied from the context and predictions from earlier forward passes, and gives each a fixed role: the more reliable source forms a deep spine, and the other spreads as wide branches along it. FlashMTP gives no source a fixed role. It keeps acceptance statistics per source, depth and evidence strength, values every candidate by the product of these rates along its path, and admits candidates best-first, so the shape of the tree is decided again in every pass.

  • Goose extends a draft. FlashMTP re-predicts below every admitted candidate. In FlashMTP, every source is queried again with the hypothesized path as context below every admitted candidate. The model's distributions at all evaluated positions, rejected ones included, also become path-conditioned predictions for the following passes.

According to published results, we compare FlashMTP with Goose. Table R2 shows the results. We will add it to the appendix of the revised paper.

Table R2: Comparison between FlashMTP and Goose.

MethodImplementation DetailsQwen3-8B HumanEval MATQwen3-8B HumanEval Speedup
GooseConstructs a fixed tree structure with a deep main trunk and wide branches based on the acceptance rate by source.2.782.34×
FlashMTPPerforms separate statistics based on source × depth × evidence strength, allocates resources one by one using the probability product of the entire path, and does not predefine the structure.3.753.10×

[2] Goose: Anisotropic Speculation Trees for Training-Free Speculative Decoding, Jin et al., COLM 2026.

Response to Reviewer Grix (Part 3 / 6)

W3: FlashMTP uses 63 candidate tokens per pass while the candidate budgets and hyperparameters used for LookAhead and LogitSpec are not clearly reported.

A-W3: We thank the reviewer for raising this question. Table R3 lists the number of candidate tokens and the hyperparameters of all methods. We will add it to the appendix of the revised paper. All counts include the root, and rows is the number of positions evaluated in one forward pass. For both baselines we use the defaults of their official implementations: W=7, N=5, G=7 is the configuration given in the official LookAhead code, and a maximum n-gram size of 3, 20 retrieved tokens and tree capacity 64 are the defaults of the official LogitSpec code.

Table R3: Candidate counts and hyperparameters.

MethodRows per passCandidate tokensHyperparameters
LookAhead5655 (6 + 21 lookahead-window tokens, 28 guess tokens)W=7, N=5, G=7
LogitSpec6463max n-gram 3, 20 retrieved tokens, tree capacity 64

Comments on the paper

1 comment

Dear Area Chair,

We would like to provide a brief summary of our responses to assist with the meta-review.

We posted our responses to all three reviewers and uploaded the revised paper. All new results are in Appendix G of the revised paper (G.1–G.6). As of now, we have not received further questions from the reviewers.

The main concerns of each reviewer and our responses are summarized below.

  • Reviewer Grix (missing baselines and fair comparison). We added Token Recycling with the same number of candidate tokens, and FlashMTP is faster in all six settings (G.1). We also compared FlashMTP with Goose, EAGLE-2 and EAGLE-3 according to published results (G.2). Matched candidate counts and sweeps show that FlashMTP with only 16 rows is faster than every LookAhead and LogitSpec setting (G.3), and throughput, memory and per-step latency are reported in G.4.
  • Reviewer edjf (contribution over Token Recycling and FP16 fidelity). With the same models, number of candidate tokens, protocol and backend, FlashMTP remains 6.9–50.7% faster than Token Recycling even after adaptive allocation is added to Token Recycling (G.1). In FP16, HumanEval pass@1 and GSM8K accuracy are identical to those of autoregressive decoding (G.5). The reset policy and the sensitivity to the decoding state are reported in G.6.
  • Reviewer fjwE (candidate count, FP16 exactness and cost). At the same 56 or 64 rows, FlashMTP accepts 1.6–2.1 times as many tokens per pass as LookAhead and LogitSpec, and it is also faster at matched per-step latency (G.3, G.4). FP16 agreement and task scores are reported in G.5, and peak memory and the cost as the context grows are reported in G.4.

We would be grateful if you could take our responses and the revision into account in the meta-review.

Thank you for your time and for handling our submission.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.