- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes FlashMTP, a training-free method to generate several tokens in one model pass. It reuses successor predictions from evaluated candidates and allocates the next pass's candidates using estimated path acceptance rates. The paper reports more accepted tokens and faster generation than LookAhead and LogitSpec on three benchmarks and eight model settings.
Critical strengths
-
Reusing predictions from evaluated candidate positions is a useful idea. The adaptive allocation rule gives these predictions a clear role in the next pass.
-
The decoding procedure is described clearly. The appendix explains why it preserves greedy outputs and sampling distributions under exact arithmetic.
-
The speedups appear across several model sizes and tasks. The component ablations show that both successor reuse and allocation contribute to the result.
Critical weaknesses
-
The main comparison does not separate proposal quality from candidate budget. FlashMTP uses up to 63 candidate positions per pass, while the reported LookAhead setup cannot accept more than four tokens per pass. A matched-budget comparison is needed to judge the source of the speedup.
-
The practical losslessness claim is not quantified. The main speed tests use float16, but Appendix C reports only that float16 outputs are almost identical to autoregressive outputs. Exact-match rates and task-quality differences are needed for the setting used in Table 1.
-
The evaluation uses batch size one and does not report peak memory or how cost changes with context length. A large candidate tree and saved successor index may be expensive in other serving settings. This limits the deployment claim.
Questions for authors
Please address the above weaknesses and answer the following questions.
-
What candidate budgets and maximum depths were used for LookAhead and LogitSpec? Can you compare the methods at a matched number of candidate positions or matched per-pass latency?
-
For float16 decoding, what fraction of full outputs exactly matches autoregressive decoding? Do the first token differences affect HumanEval, GSM8K, or Spec-Bench scores?
-
What are the peak GPU memory and index-memory costs as context length grows? Does the speedup hold when more than one request is decoded in a batch?
8 comments
Response to Reviewer fjwE (Part 2 / 6)
W2: The practical losslessness claim is not quantified. The main speed tests use float16, but Appendix C reports only that float16 outputs are almost identical to autoregressive outputs. Exact-match rates and task-quality differences are needed for the setting used in Table 1.
A-W2: We thank the reviewer for raising this question. Table R3 reports sequence agreement and task quality for Qwen2.5-7B. We will add it to the appendix of the revised paper.
Table R3: Output agreement with AutoRegressive decoding and task quality on Qwen2.5-7B.
| Method | HumanEval identical to AutoRegressive | HumanEval pass@1 | GSM8K identical to AutoRegressive | GSM8K accuracy |
|---|---|---|---|---|
| AutoRegressive | 164/164 | 82.3% | 300/300 | 88.3% |
| LookAhead(f16) | 156/164(95.1%) | 82.3% | 277/300 (92.3%) | 88.3% |
| FlashMTP(f16) | 159/164 (97.0%) | 82.3% | 276/300(92.0%) | 88.3% |
Theoretically, due to the verification mechanism, FlashMTP does not change the output of greedy decoding. Under single precision (FP32), the output of FlashMTP is identical to that of AutoRegressive decoding. However, due to numerical precision, under half precision (FP16), the output of FlashMTP is not completely identical to that of AutoRegressive decoding.
Other lossless acceleration methods in this field behave in the same way. For example, under half-precision (FP16) inference, 8 of the 164 LookAhead answers on HumanEval are inconsistent with the AutoRegressive output.
Response to Reviewer fjwE (Part 3 / 6)
W3: The evaluation uses batch size one and does not report peak memory or how cost changes with context length. A large candidate tree and saved successor index may be expensive in other serving settings. This limits the deployment claim.
A-W3: We thank the reviewer for raising these questions.
(i) Batch size.
Current mainstream methods in this field are mainly evaluated with a batch size of 1. For example, LookAhead and LogitSpec both operate with a batch size of 1. Supporting larger batch sizes requires substantial modifications to the underlying codebase.
(ii) Peak memory.
Table R4 reports the peak GPU memory on HumanEval. We will add it to the appendix of the revised paper.
Table R4: Peak GPU memory on HumanEval with Qwen2.5-7B (nvidia-smi, MiB).
| Peak GPU memory (MiB) | |
|---|---|
| Autoregressive | 16,063 |
| LookAhead | 15,963 |
| LogitSpec | 15,963 |
| FlashMTP | 15,963 |
All methods peak at about 16.0 GB. nvidia-smi includes memory cached by PyTorch's allocator, so differences below 1% are within allocator granularity and do not reflect real savings. A precise measurement of allocated memory shows that a forward pass over 64 rows uses at most 0.03 GB more than one row. The candidate tree therefore adds no meaningful GPU memory.
(iii) Cost as context length grows.
Table R5 reports how the cost changes with context length (Qwen2.5-7B, summarization inputs, 256 new tokens). We will add it to the appendix of the revised paper.
Table R5: Cost of decoding as the context grows.
| Context length | 1k | 2k | 4k |
|---|---|---|---|
| AR, time per step (ms) | 15.6 | 16.2 | 16.9 |
| FlashMTP, time per step (ms) | 21.9 | 26.1 | 32.6 |
| One forward pass, 1 row (ms) | 16.7 | 16.4 | 16.3 |
| One forward pass, 64 rows (ms) | 17.8 | 19.6 | 23.2 |
| Peak allocated GPU memory, 1 row / 64 rows (GB) | 14.32 / 14.34 | 14.38 / 14.39 | 14.48 / 14.50 |
| Successor index (host memory) | ≤ 5.2 MB | ≤ 5.2 MB | ≤ 5.2 MB |
(iv) The saved successor index.
The index lives in host memory and is bounded by 8,192 keys with 8 next tokens each, about 5.2 MB when full, regardless of context length. With many concurrent sessions, it grows linearly with the number of sessions and can be capped or cleared per session.
Response to Reviewer fjwE (Part 1 / 6)
We thank the reviewer for the positive assessment and constructive suggestions. We respond to each comment as follows and sincerely hope that our responses could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.
A-W1: We thank the reviewer for raising this question. The LookAhead setting W=7, N=5, G=7 is the configuration given in the official LookAhead implementation, and with N=5 each guess has four tokens. To separate the quality of the candidates from their number, we ran each method with several numbers of rows, including 56 and 64 rows for all three methods. Tables R1 and R2 show the results. We will add them to the appendix of the revised paper.
Table R1: Matched candidate counts.
Table R2: MAT and speedup with different numbers of candidate tokens per forward pass on Qwen2.5-7B.
At the same number of rows, FlashMTP accepts 1.6–2.1 times as many tokens per pass as LookAhead and LogitSpec. Allowing LookAhead to accept up to six tokens per step (84 rows) raises its MAT by at most 0.22, and FlashMTP with only 16 rows (15 candidate tokens) is faster than every LookAhead and LogitSpec setting on all three benchmarks.
LookAhead and LogitSpec fix the shape of a draft in advance: an n-gram window with a preset guess length, or a retrieved segment below each ranked next-next token. Adding positions mostly adds more guesses of the same kind, so their MAT saturates. FlashMTP proposes the candidates of each pass for the specific path below every admitted candidate, using the distributions the model computed at all evaluated positions, and gives each additional row to the path most likely to be accepted according to the acceptance it has measured, so its MAT keeps growing with the number of rows. The speedup therefore comes from the quality and allocation of the candidates, not from evaluating more candidates.