- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies block size in diffusion-based speculative decoding with DFlash drafters. Sweeping block sizes per sample, it reports that the size maximizing acceptance length varies across inputs but stays close to the drafter's training block size. BlockPilot treats block-size selection as classification over a small neighborhood of the training size. A small MLP reads the target model's logits at the final prefill position and predicts the block size once per request. It is trained on labels obtained by trying each candidate size on a set of prompts. Validation covers four target models on math, code and chat benchmarks at two temperatures. BlockPilot reports higher speedup and acceptance length than EAGLE-3 and every fixed DFlash block size, with further results on predictor variants, concurrent serving, a per-round feedback variant and offline cost.
Critical strengths
BlockPilot adds almost no inference cost, since it runs once after prefill and leaves the drafter, target, and verification unchanged. In the complete appendix tables, which include every intermediate block size (Tables 11 and 12), it gives the highest average speedup of any fixed block size for all models and both temperatures. This supports acceptance because the gain holds against the strongest fixed setting.
The observation that the best block size varies across prompts but stays in a narrow band around the training size is documented with a per-sample sweep (Figure 3), and the choice of predictor input is compared against scalar confidence rules and hidden states (Table 6). This matters because the observation is useful to anyone deploying block-diffusion drafters, independent of this particular predictor.
Critical weaknesses
The main comparison does not separate per-sample selection from simply choosing a better global block size. Table 2 and Figure 2 compare against a few widely spaced block sizes, and the text measures gains against the training size. The appendix shows that neighboring sizes are often stronger than the training size, and BlockPilot's margin over the best fixed size is clearly smaller, especially in greedy decoding. A per-sample oracle that uses the measured labels is also missing. This is consequential because the paper claims sample adaptivity drives the gain, and the main-text evidence overstates how much of the gain comes from adaptivity. The predictor's reported near-perfect classification accuracy (Table 6) would place it almost at the oracle, so the oracle row is needed to check that claim.
The contribution is narrow. It selects one discrete hyperparameter of one drafter family from a handful of values, and it requires hours of offline enumeration and training for each target and drafter pair (Appendix I). Adaptive draft length has been studied extensively for autoregressive drafters, and the "policy learning" framing describes a supervised classifier. Together with the smaller margin over the best fixed size, this makes the advance incremental relative to tuning the block size once per model, which weighs against acceptance.
Questions for authors
- Which data are the Table 6 accuracies measured on? The predictor is trained on general-purpose prompts, yet accuracy is reported per evaluation benchmark. Were the prompts used to construct labels disjoint from those used to measure speedup?
- Neighboring block sizes often give nearly equal acceptance lengths, and with sampling the labels come from stochastic decoding. How stable are the labels across repeated runs, and how does that ceiling compare with the reported accuracy?
- What speedups do a per-sample oracle and the best single fixed block size per model (chosen on held-out data) reach? Both belong in the main table.
- The SGLang serving results (Table 7) at a single request appear lower than Table 2 for what looks like the same model, dataset, and temperature. Do the two tables use different stacks or generation lengths, and which should be read as the main result?
- The per-round feedback variant is slower (Table 8). Training the predictor on mid-generation states would show whether this comes from per-round overhead or from applying a prefill-trained predictor outside its training distribution.
6 comments
Q1. Training data and prediction-accuracy evaluation
Table 6 measures the Qwen3-4B predictor's accuracy on test samples from the seven evaluation benchmarks. Training uses ShareGPT, WSC, and COPA with the configuration in lines 369–377; the resulting predictor is directly applied to these benchmarks, without additional task-specific training or adjustment. Reference labels are obtained by enumerating candidate block sizes for each evaluation input and selecting the size with the highest measured acceptance length; accuracy is the exact-match rate against these labels.
We distinguish the prompt template, which specifies how an input is presented, from the underlying sample, which contains the actual question or task content. Label construction and speedup evaluation use the appropriate input format for each dataset. Across the training sources and evaluation benchmarks, the inputs differ in task content as well as presentation format. The predictor transfers across these task contents and input formats without task-specific retraining, supporting generalization beyond the general-purpose training sources. This evidence concerns cross-task transfer rather than prompt-format variation alone.
Q2. Label stability under repeated stochastic decoding
When constructing the per-instance block-size sweeps in Figure 3, we repeatedly decoded the same inputs using different random seeds and compared the acceptance lengths and optimal choices across candidates. Within the samples and repetitions covered by these experiments, the optimal block-size labels remained unchanged, yielding an observed label agreement of 100%. For candidates with exactly equal acceptance lengths, a fixed priority order determines the label consistently.
The 98.7% in Table 6 measures agreement between the predictor and the measured reference labels; the observed 100% measures label agreement across repeated decoding runs. These measure prediction accuracy and empirical label stability, respectively. The repeated sweeps support label stability for the tested inputs and repetitions. The 98.7% is an exact-match accuracy against the constructed labels, not a theoretical accuracy ceiling.
Q4. Single-request SGLang results versus Table 2
The tables use different execution environments and timing scopes. Table 2 measures decode-stage speedup, primarily reflecting TPOT reduction. Table 7 uses the SGLang serving framework and measures the complete serving pipeline, including request scheduling, batch management, drafting, and verification. Scheduling and management operations remain present even at concurrency one. These costs need not receive the same proportional reduction as decoding, so serving-level speedup can be lower than decode-stage speedup.
The additional benefit of BlockPilot over fixed-block DFlash remains similar. For Qwen3-8B on AIME24 at temperature 0, the improvement over DFlash(16) is approximately 10.6% in Table 2 (5.57× to 6.16×), and 10.9% at concurrency one in Table 7 (4.50× to 4.99×). Thus, the advantage of adaptive selection is retained in the serving framework.
Table 2 is the main result for decoding acceleration; Table 7 is a complementary serving evaluation across concurrency levels. Each should be interpreted within its respective timing scope.
Q5. Per-round feedback and training on decoding states
The predictor for the per-round feedback variant in Table 8 was trained with decoding-stage intermediate states included specifically for this comparison. This training setup is exclusive to that experiment. The variant therefore does not simply apply a predictor trained only on prefill states to intermediate generation states. This corrects Appendix D's description of reusing the same prefill-trained predictor: the feedback variant has a distinct training-data setup, so the comparison does not change prediction frequency alone.
Per-round feedback requires extracting predictor inputs, running the MLP, and updating the block size at every iteration. The prediction cost therefore accumulates with decoding rounds, whereas BlockPilot incurs it once after prefill. Table 8 reports lower net speedup for per-round feedback even when training includes decoding states. This is consistent with the gains from repeated selection being insufficient to offset its execution costs, although the experiment does not isolate each cost component.
Dear Reviewer P4TC,
Thank you once again for your time and valuable feedback on our manuscript.
We have carefully considered all of your comments and provided detailed responses to your concerns.
We sincerely hope that our responses address your concerns. We would be happy to clarify any remaining questions and engage in further discussion.
Best regards,
Authors
Response to Reviewer P4TC
Thank you for your detailed review and constructive feedback. Please find our responses below.
W1 & Q3. Best fixed-block configurations and the per-instance oracle
Tables 11–12 already provide the complete comparisons with neighboring fixed block sizes, allowing the benefit of instance-level selection to be assessed beyond global fixed-block tuning. Relative to the best reported fixed configuration for each model–temperature combination, BlockPilot achieves a higher overall average speedup in all eight combinations, with relative improvements of 3.2%–13.7%. This comparison separates the additional benefit of instance-level selection from the total improvement over the training block size. Under greedy decoding, the gains over the best reported fixed configuration are approximately 3.2% for Qwen3-4B and 5.4% for Qwen3-8B.
We additionally provide a per-instance acceptance-length oracle comparison. The oracle directly uses the block size with the highest measured acceptance length within the predictor's candidate set for each input, bypassing the predictor. All entries below are overall average speedup ratios.
The fixed baseline is the configuration with the highest overall average speedup in Table 11 for each model–temperature combination; the selected block sizes are 17, 17, 16, and 17 in the row order above. It is a retrospective best fixed configuration on the reported test results, rather than a configuration selected on an independent held-out set. For the same candidate set, test samples, timing measurements, and aggregation rule, this retrospective maximum is no lower than the test-result performance of any fixed configuration selected on held-out data. It therefore supplies a stronger fixed-configuration comparison on these results.
BlockPilot exceeds this fixed baseline in all four settings, and its reported overall average speedup is close to that of the acceptance-length oracle. These comparisons show both the benefit over global fixed-block tuning and the performance gap between prediction and direct selection using measured optimal labels.
W2. Contribution, supervised policy learning, and offline cost
BlockPilot's contribution is the discovery of block-size characteristics in diffusion drafters and a request-level adaptive selection method derived from them. To our knowledge, this work is the first to systematically show that the optimal block size varies across inputs while remaining concentrated near the training block size in the evaluated settings. Figure 3 supports this finding across multiple models and datasets. The small candidate set is a consequence of this empirical structure, enabling efficient instance-level selection.
Relative to prior adaptive draft-length work on autoregressive drafters, the novelty lies in this diffusion-drafter finding and the resulting one-time request-level selection mechanism. “Policy learning” is implemented through supervised classification; the contribution is not a new classifier architecture. BlockPilot makes one prediction from prefill logits and applies the selected block size throughout generation.
Dataset construction and predictor training take approximately 12.15 hours in the reported setting, without retraining the target model or diffusion drafter. For a given model pair, this preparation is a one-time cost, whereas inference benefits recur across subsequent requests. The 3.2%–13.7% improvements over the best reported fixed configurations in Tables 11–12 demonstrate additional benefits beyond tuning a single block size once per model.