- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This manuscript introduces BV Loss, a block verification-aware training objective for diffusion-based speculative decoding. The key idea is to align the drafter training objective with the sequence-level acceptance criterion used by block verification rather than optimizing token-level acceptance.
Critical strengths
-
The paper is clearly written and well organized.
-
The motivation is intuitive and well aligned with the problem setting.
Critical weaknesses
-
While the paper is motivated by the mismatch between token-level objectives and block verification, most experiments only show improvements in accepted length and speedup. I would like to see a deeper analysis of why the proposed objective specifically benefits block verification and which properties of the verification mechanism are being exploited.
-
The paper reports acceptance length and speedup, but does not report benchmark quality metrics such as GSM8K accuracy, HumanEval pass@1, MBPP pass@1, or MT-Bench scores. Since the drafter is trained with a different objective, it would be helpful to verify that the efficiency gains do not come with quality degradation.
-
The empirical evaluation lacks variance estimates. Results appear to be based on a single training seed, and no standard deviation or confidence interval is reported. This makes it difficult to assess the robustness of some of the reported gains.
-
BV Loss seems to be the natural objective implied by the block verification mechanism. While the idea is well motivated, the contribution feels more like a principled extension of existing acceptance-aware objectives to block verification rather than a fundamentally new optimization framework.
Questions for authors
- Can the authors provide benchmark quality metrics (e.g., GSM8K accuracy, HumanEval pass@1, MT-Bench scores) to verify that BV does not negatively affect downstream performance?
- Have the authors evaluated measured variance statistics?
- Can the authors provide additional analysis explaining which aspects of block verification contribute most to the gains observed with BV Loss?
8 comments
W2 / Q1: Preservation of Benchmark Performance
Our method modifies only the training objective and leaves the inference and verification procedures unchanged. It therefore preserves the lossless guarantee of the underlying verification algorithm.
In our evaluations, our method produced outputs identical to those of the original target model, resulting in identical GSM8K accuracy, HumanEval pass@1, and MT-Bench scores (using GPT-OSS-120B as the judge). Stochastic decoding can cause benchmark scores to vary across runs due to sampling randomness, even when the underlying model performance remains unchanged. To avoid this source of variability, we used greedy decoding for both the evaluated model and the judge model. The detailed benchmark results are shown in Table A.
Table A. Benchmark performance for the target-only reference and speculative decoding with a BV-trained drafter.
| Decoding | GSM8K accuracy | HumanEval pass@1 | MT-Bench |
|---|---|---|---|
| Target only | 91.736% | 84.146% | 6.813 |
| Speculative decoding with BV | 91.736% | 84.146% | 6.813 |
The matching benchmark scores confirm that our implementation preserves the target model’s outputs and benchmark performance under the evaluated settings, consistent with the theoretical lossless guarantee.
W3 / Q2: Statistical Robustness and Generalization
We agree that experiments with a single training seed do not provide sufficient evidence to assess statistical robustness. It is reasonable to check whether the observed improvements depend on a particular seed. To address this concern, we conducted additional experiments using training seeds 42, 43, and 44, comparing CE, LK, and BV under the same evaluation setting.
Table B. Average accepted length across seven benchmarks for three training seeds on Qwen3-4B / DFlash.
| Loss | Seed 42 (paper) | Seed 43 | Seed 44 | Mean ± SD |
|---|---|---|---|---|
| CE | 4.09 | 4.08 | 4.05 | 4.07 ± 0.02 |
| LK | 4.25 | 4.22 | 4.25 | 4.24 ± 0.02 |
| BV | 4.95 | 4.86 | 4.91 | 4.91 ± 0.05 |
Table C. Per-benchmark accepted length, reported as mean ± standard deviation across the same three training seeds on Qwen3-4B / DFlash. All results use T = 1 and block verification.
| Loss | GSM8K | MATH-500 | AIME25 | HumanEval | MBPP | LCB | MT-Bench | Avg. |
|---|---|---|---|---|---|---|---|---|
| CE | 5.55 ± 0.01 | 4.81 ± 0.02 | 3.83 ± 0.04 | 4.02 ± 0.04 | 3.75 ± 0.02 | 3.95 ± 0.05 | 2.62 ± 0.03 | 4.07 ± 0.02 |
| LK | 5.75 ± 0.09 | 5.00 ± 0.07 | 3.96 ± 0.09 | 4.20 ± 0.04 | 3.92 ± 0.01 | 4.09 ± 0.05 | 2.74 ± 0.02 | 4.24 ± 0.02 |
| BV | 6.80 ± 0.05 | 5.86 ± 0.06 | 4.72 ± 0.07 | 4.74 ± 0.05 | 4.49 ± 0.06 | 4.62 ± 0.09 | 3.11 ± 0.03 | 4.91 ± 0.05 |
Across the three seeds, BV achieves an average accepted length of 4.91 ± 0.05, compared with 4.07 ± 0.02 for CE and 4.24 ± 0.02 for LK. Its mean improvements of 0.84 over CE and 0.67 over LK are substantially larger than the observed variation across seeds. BV outperforms both baselines at every evaluated seed and achieves the highest mean accepted length on all seven benchmarks. These results provide evidence that the improvements are not driven by a single favorable training seed.
We also conducted additional experiments on Llama-3.1-8B-Instruct to assess whether the benefits generalize beyond the Qwen model family.
Table D. Average accepted length on Llama-3.1-8B-Instruct / DFlash. All results use T = 1 and block verification.
| Loss | GSM8K | MATH-500 | AIME25 | HumanEval | MBPP | LCB | MT-Bench | Avg. |
|---|---|---|---|---|---|---|---|---|
| CE | 3.92 | 3.35 | 2.75 | 4.19 | 3.96 | 3.28 | 2.45 | 3.41 |
| LK | 4.11 | 3.53 | 2.83 | 4.44 | 4.21 | 3.50 | 2.50 | 3.59 |
| BV | 4.89 | 4.00 | 3.04 | 5.05 | 4.85 | 3.97 | 2.76 | 4.08 |
BV consistently outperforms both CE and LK across all seven benchmarks on Llama. Its average accepted length reaches 4.08, compared with 3.41 for CE and 3.59 for LK. Together with the multi-seed results, these findings provide further evidence that our method remains effective across training seeds and model families.
W4 / Q3: Technical Contributions
We agree that BV loss shares its motivation with existing acceptance-aware objectives: aligning the training objective with the acceptance criterion used at inference. However, the objective that block verification naturally implies does not work on its own. Directly maximizing the expected accepted length, which we refer to as pure BV (Eq. 10), actually underperforms a drafter trained with CE. Our contribution lies in identifying the optimization difficulties responsible for this behavior and developing a training method that addresses them.
Table E. Average accepted length for different objectives on Qwen3-4B / DFlash, with T = 1 and block verification.
| Objective | GSM8K | MATH-500 | AIME25 | HumanEval | MBPP | LCB | MT-Bench | Avg. |
|---|---|---|---|---|---|---|---|---|
| CE | 5.55 | 4.82 | 3.88 | 4.01 | 3.76 | 3.96 | 2.65 | 4.09 |
| TV | 2.39 | 2.43 | 2.16 | 1.71 | 1.76 | 1.62 | 1.62 | 1.96 |
| Pure BV | 3.03 | 2.97 | 2.54 | 2.00 | 2.18 | 1.82 | 1.79 | 2.34 |
| LK | 5.74 | 5.04 | 3.90 | 4.24 | 3.93 | 4.14 | 2.73 | 4.25 |
| Log-BV w/o annealing | 6.35 | 5.67 | 4.68 | 4.54 | 4.32 | 4.55 | 3.03 | 4.74 |
| Log-BV + annealing | 6.85 | 5.92 | 4.80 | 4.79 | 4.53 | 4.68 | 3.09 | 4.95 |
1. The natural extension does not work well.
Pure BV achieves an average accepted length of 2.34, better than TV (1.96) but well below CE (4.09). Deriving the block-level objective is therefore insufficient to outperform CE.
2. Pure BV introduces two optimization difficulties arising from the block structure.
Under pure BV, the gradient contribution of each prefix is proportional to its acceptance score (Eq. 12), and acceptance scores are determined by cumulative likelihood ratios.
- Vanishing gradients: The size of the overall gradient scales with the acceptance scores of all prefixes combined. When acceptance is low, this total is very small, so the entire drafter learns slowly.
- Early-training signal imbalance across positions: Because likelihood ratios accumulate along a prefix, low acceptance at earlier positions drives the scores of longer prefixes close to zero. Early in training, the gradient is therefore concentrated almost entirely on the earlier positions, and later positions receive almost no learning signal. This imbalance is absent from TV and LK, whose loss terms evaluate individual positions independently.
3. Log-BV and annealing address these two difficulties.
- Log-BV (Eq. 13): We apply a logarithmic transformation to the overall expected accepted length. This preserves the minimizers while normalizing the gradient by the expected accepted length, removing the acceptance-dependent attenuation of the overall gradient.
- Annealing (Eq. 15): At the beginning of training, we assign equal weights to all prefixes so that later positions also receive sufficient learning signal, and then gradually transition back to log-BV.
4. The gains can be attributed to the individual components as follows.
- Block-aware objective: This is the component through which the two properties described in W1 / Q3 enter training. Holding the other conditions fixed, moving from a token-level objective to a block-level objective improves the average accepted length from 1.96 with TV to 2.34 with pure BV.
- Log scaling: The average accepted length improves from 2.34 with pure BV to 4.74 with log-BV. This is the largest improvement and makes optimization of the block-aware objective effective.
- Annealing: Annealing provides a further improvement on all seven benchmarks, increasing the average accepted length from 4.74 to 4.95.
The prefix-level acceptance structure of block verification is what the objective exploits, but directly optimizing it underperforms even CE. Once log scaling and annealing address the two optimization difficulties, the block-aware objective becomes effective, reaching an average accepted length of 4.95. Our contribution therefore lies in the analysis and training methodology that make the block-level extension effective, rather than in the extension alone. We will include Table E as an ablation study in the revised manuscript and clarify the relationship between BV loss and existing acceptance-aware objectives.
Thank you again for the constructive feedback. We hope these additional results and clarifications address your concerns, and please let us know if you have any remaining concerns.
Thank you for your careful and constructive review. We address each of your points below.
W1 / Q3: Properties of Block Verification Exploited by BV Loss
BV loss exploits two properties of block verification that work together.
Property 1: Prefix-aware acceptance. Acceptance is determined by the cumulative likelihood ratio over an entire prefix, rather than by evaluating each token independently (Eq. 5).
Property 2: Compensation across positions. When the drafter assigns a higher probability than the target to an earlier token on a target-generated trajectory, that surplus can compensate for a lower draft-to-target probability ratio at a later position.
The first property means that block verification considers both the state of the preceding prefix and the likelihood ratio of the current token when determining the acceptance probability of the extended prefix. A prediction at one position therefore affects both the acceptance of that token and the acceptance of subsequent prefixes that depend on it. BV loss incorporates these prefix-level acceptance scores into the training objective, jointly evaluating predictions across positions according to their contributions to the accepted length.
The second property can be understood by considering the accumulation of draft-to-target probability ratios along target-generated training trajectories. A ratio greater than one at an earlier position can compensate for a lower ratio at a later position (Eqs. 7,8). Consequently, the acceptance probability of a prefix depends on how the probability ratios at different positions combine. BV captures these compensatory relationships, which cannot be captured by considering individual token acceptance alone.
An example sequence of length 2 gives some intuition for when BV loss has an advantage over TV loss. Suppose the target's probabilities for the proposed tokens are (0.8, 0.8) and the drafter's probabilities are (0.9, 0.7). TV loss trains the drafter to lower its first probability from 0.9 to 0.8 and to raise its second from 0.7 to 0.8. Under BV loss, in contrast, since the sequence-level probabilities satisfy 0.8 * 0.8 > 0.9 * 0.7, training pushes up the probabilities of both the first and the second token. This case illustrates that TV loss and BV loss optimize different objectives.
These two properties work together: prefix-aware acceptance connects predictions across positions, while cumulative likelihood ratios capture how those predictions can compensate for one another. BV loss thus provides a learning signal that encourages these predictions to jointly sustain longer accepted prefixes, in addition to improving local token acceptance.