acceptodds
Under review as a conference paper at ICLR 2027

BV Loss: Block Verification Aware Loss for Block Diffusion Speculative Decoding

Abstract

Diffusion drafters accelerate speculative decoding by proposing multiple tokens in parallel. Despite recent advances in speculative decoding through sequence-level drafting and verification, existing training objectives remain largely designed around token-level verification. To address this mismatch, we introduce **Block Verification-aware loss (BV loss)**, a training objective designed to maximize the expected acceptance length of a drafted sequence. BV loss is directly derived from the block verification acceptance rule, providing a principled connection between the drafter training objective and the inference-time verification mechanism at the sequence level. Across math, code, and chat benchmarks, BV loss increases the mean number of tokens accepted per verification call under block verification by 13.0–21.0% over cross-entropy loss training for DFlash and DSpark with Qwen3-4B and Qwen3-8B without changing the inference procedure. BV loss also outperforms tokenwise acceptance objectives such as TV loss and LK loss, and its gains extend to token verification and greedy decoding. These results demonstrate the benefit of training block diffusion drafters with an objective aligned with sequence-level verification, rather than optimizing each token independently.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
speculative decoding, block diffusion drafter, large language model, block verification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mx5xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript introduces BV Loss, a block verification-aware training objective for diffusion-based speculative decoding. The key idea is to align the drafter training objective with the sequence-level acceptance criterion used by block verification rather than optimizing token-level acceptance.

Critical strengths
  1. The paper is clearly written and well organized.

  2. The motivation is intuitive and well aligned with the problem setting.

Critical weaknesses
  1. While the paper is motivated by the mismatch between token-level objectives and block verification, most experiments only show improvements in accepted length and speedup. I would like to see a deeper analysis of why the proposed objective specifically benefits block verification and which properties of the verification mechanism are being exploited.

  2. The paper reports acceptance length and speedup, but does not report benchmark quality metrics such as GSM8K accuracy, HumanEval pass@1, MBPP pass@1, or MT-Bench scores. Since the drafter is trained with a different objective, it would be helpful to verify that the efficiency gains do not come with quality degradation.

  3. The empirical evaluation lacks variance estimates. Results appear to be based on a single training seed, and no standard deviation or confidence interval is reported. This makes it difficult to assess the robustness of some of the reported gains.

  4. BV Loss seems to be the natural objective implied by the block verification mechanism. While the idea is well motivated, the contribution feels more like a principled extension of existing acceptance-aware objectives to block verification rather than a fundamentally new optimization framework.

Questions for authors
  1. Can the authors provide benchmark quality metrics (e.g., GSM8K accuracy, HumanEval pass@1, MT-Bench scores) to verify that BV does not negatively affect downstream performance?
  2. Have the authors evaluated measured variance statistics?
  3. Can the authors provide additional analysis explaining which aspects of block verification contribute most to the gains observed with BV Loss?
8 comments

Thank you for your careful and constructive review. We address each of your points below.

W1 / Q3: Properties of Block Verification Exploited by BV Loss

BV loss exploits two properties of block verification that work together.

Property 1: Prefix-aware acceptance. Acceptance is determined by the cumulative likelihood ratio over an entire prefix, rather than by evaluating each token independently (Eq. 5).

Property 2: Compensation across positions. When the drafter assigns a higher probability than the target to an earlier token on a target-generated trajectory, that surplus can compensate for a lower draft-to-target probability ratio at a later position.

The first property means that block verification considers both the state of the preceding prefix and the likelihood ratio of the current token when determining the acceptance probability of the extended prefix. A prediction at one position therefore affects both the acceptance of that token and the acceptance of subsequent prefixes that depend on it. BV loss incorporates these prefix-level acceptance scores into the training objective, jointly evaluating predictions across positions according to their contributions to the accepted length.

The second property can be understood by considering the accumulation of draft-to-target probability ratios along target-generated training trajectories. A ratio greater than one at an earlier position can compensate for a lower ratio at a later position (Eqs. 7,8). Consequently, the acceptance probability of a prefix depends on how the probability ratios at different positions combine. BV captures these compensatory relationships, which cannot be captured by considering individual token acceptance alone.

An example sequence of length 2 gives some intuition for when BV loss has an advantage over TV loss. Suppose the target's probabilities for the proposed tokens are (0.8, 0.8) and the drafter's probabilities are (0.9, 0.7). TV loss trains the drafter to lower its first probability from 0.9 to 0.8 and to raise its second from 0.7 to 0.8. Under BV loss, in contrast, since the sequence-level probabilities satisfy 0.8 * 0.8 > 0.9 * 0.7, training pushes up the probabilities of both the first and the second token. This case illustrates that TV loss and BV loss optimize different objectives.

These two properties work together: prefix-aware acceptance connects predictions across positions, while cumulative likelihood ratios capture how those predictions can compensate for one another. BV loss thus provides a learning signal that encourages these predictions to jointly sustain longer accepted prefixes, in addition to improving local token acceptance.

W2 / Q1: Preservation of Benchmark Performance

Our method modifies only the training objective and leaves the inference and verification procedures unchanged. It therefore preserves the lossless guarantee of the underlying verification algorithm.

In our evaluations, our method produced outputs identical to those of the original target model, resulting in identical GSM8K accuracy, HumanEval pass@1, and MT-Bench scores (using GPT-OSS-120B as the judge). Stochastic decoding can cause benchmark scores to vary across runs due to sampling randomness, even when the underlying model performance remains unchanged. To avoid this source of variability, we used greedy decoding for both the evaluated model and the judge model. The detailed benchmark results are shown in Table A.

Table A. Benchmark performance for the target-only reference and speculative decoding with a BV-trained drafter.

DecodingGSM8K accuracyHumanEval pass@1MT-Bench
Target only91.736%84.146%6.813
Speculative decoding with BV91.736%84.146%6.813

The matching benchmark scores confirm that our implementation preserves the target model’s outputs and benchmark performance under the evaluated settings, consistent with the theoretical lossless guarantee.

W3 / Q2: Statistical Robustness and Generalization

We agree that experiments with a single training seed do not provide sufficient evidence to assess statistical robustness. It is reasonable to check whether the observed improvements depend on a particular seed. To address this concern, we conducted additional experiments using training seeds 42, 43, and 44, comparing CE, LK, and BV under the same evaluation setting.

Table B. Average accepted length across seven benchmarks for three training seeds on Qwen3-4B / DFlash.

LossSeed 42 (paper)Seed 43Seed 44Mean ± SD
CE4.094.084.054.07 ± 0.02
LK4.254.224.254.24 ± 0.02
BV4.954.864.914.91 ± 0.05

Table C. Per-benchmark accepted length, reported as mean ± standard deviation across the same three training seeds on Qwen3-4B / DFlash. All results use T = 1 and block verification.

LossGSM8KMATH-500AIME25HumanEvalMBPPLCBMT-BenchAvg.
CE5.55 ± 0.014.81 ± 0.023.83 ± 0.044.02 ± 0.043.75 ± 0.023.95 ± 0.052.62 ± 0.034.07 ± 0.02
LK5.75 ± 0.095.00 ± 0.073.96 ± 0.094.20 ± 0.043.92 ± 0.014.09 ± 0.052.74 ± 0.024.24 ± 0.02
BV6.80 ± 0.055.86 ± 0.064.72 ± 0.074.74 ± 0.054.49 ± 0.064.62 ± 0.093.11 ± 0.034.91 ± 0.05

Across the three seeds, BV achieves an average accepted length of 4.91 ± 0.05, compared with 4.07 ± 0.02 for CE and 4.24 ± 0.02 for LK. Its mean improvements of 0.84 over CE and 0.67 over LK are substantially larger than the observed variation across seeds. BV outperforms both baselines at every evaluated seed and achieves the highest mean accepted length on all seven benchmarks. These results provide evidence that the improvements are not driven by a single favorable training seed.

We also conducted additional experiments on Llama-3.1-8B-Instruct to assess whether the benefits generalize beyond the Qwen model family.

Table D. Average accepted length on Llama-3.1-8B-Instruct / DFlash. All results use T = 1 and block verification.

LossGSM8KMATH-500AIME25HumanEvalMBPPLCBMT-BenchAvg.
CE3.923.352.754.193.963.282.453.41
LK4.113.532.834.444.213.502.503.59
BV4.894.003.045.054.853.972.764.08

BV consistently outperforms both CE and LK across all seven benchmarks on Llama. Its average accepted length reaches 4.08, compared with 3.41 for CE and 3.59 for LK. Together with the multi-seed results, these findings provide further evidence that our method remains effective across training seeds and model families.

W4 / Q3: Technical Contributions

We agree that BV loss shares its motivation with existing acceptance-aware objectives: aligning the training objective with the acceptance criterion used at inference. However, the objective that block verification naturally implies does not work on its own. Directly maximizing the expected accepted length, which we refer to as pure BV (Eq. 10), actually underperforms a drafter trained with CE. Our contribution lies in identifying the optimization difficulties responsible for this behavior and developing a training method that addresses them.

Table E. Average accepted length for different objectives on Qwen3-4B / DFlash, with T = 1 and block verification.

ObjectiveGSM8KMATH-500AIME25HumanEvalMBPPLCBMT-BenchAvg.
CE5.554.823.884.013.763.962.654.09
TV2.392.432.161.711.761.621.621.96
Pure BV3.032.972.542.002.181.821.792.34
LK5.745.043.904.243.934.142.734.25
Log-BV w/o annealing6.355.674.684.544.324.553.034.74
Log-BV + annealing6.855.924.804.794.534.683.094.95

1. The natural extension does not work well.

Pure BV achieves an average accepted length of 2.34, better than TV (1.96) but well below CE (4.09). Deriving the block-level objective is therefore insufficient to outperform CE.

2. Pure BV introduces two optimization difficulties arising from the block structure.

Under pure BV, the gradient contribution of each prefix is proportional to its acceptance score (Eq. 12), and acceptance scores are determined by cumulative likelihood ratios.

  • Vanishing gradients: The size of the overall gradient scales with the acceptance scores of all prefixes combined. When acceptance is low, this total is very small, so the entire drafter learns slowly.
  • Early-training signal imbalance across positions: Because likelihood ratios accumulate along a prefix, low acceptance at earlier positions drives the scores of longer prefixes close to zero. Early in training, the gradient is therefore concentrated almost entirely on the earlier positions, and later positions receive almost no learning signal. This imbalance is absent from TV and LK, whose loss terms evaluate individual positions independently.

3. Log-BV and annealing address these two difficulties.

  • Log-BV (Eq. 13): We apply a logarithmic transformation to the overall expected accepted length. This preserves the minimizers while normalizing the gradient by the expected accepted length, removing the acceptance-dependent attenuation of the overall gradient.
  • Annealing (Eq. 15): At the beginning of training, we assign equal weights to all prefixes so that later positions also receive sufficient learning signal, and then gradually transition back to log-BV.

4. The gains can be attributed to the individual components as follows.

  • Block-aware objective: This is the component through which the two properties described in W1 / Q3 enter training. Holding the other conditions fixed, moving from a token-level objective to a block-level objective improves the average accepted length from 1.96 with TV to 2.34 with pure BV.
  • Log scaling: The average accepted length improves from 2.34 with pure BV to 4.74 with log-BV. This is the largest improvement and makes optimization of the block-aware objective effective.
  • Annealing: Annealing provides a further improvement on all seven benchmarks, increasing the average accepted length from 4.74 to 4.95.

The prefix-level acceptance structure of block verification is what the objective exploits, but directly optimizing it underperforms even CE. Once log scaling and annealing address the two optimization difficulties, the block-aware objective becomes effective, reaching an average accepted length of 4.95. Our contribution therefore lies in the analysis and training methodology that make the block-level extension effective, rather than in the extension alone. We will include Table E as an ablation study in the revised manuscript and clarify the relationship between BV loss and existing acceptance-aware objectives.

Thank you again for the constructive feedback. We hope these additional results and clarifications address your concerns, and please let us know if you have any remaining concerns.

Reviewer_ab5BOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces BV loss, a training objective for block diffusion drafters derived from the block-verification acceptance rule. It reformulates expected accepted draft length as an expectation over target-generated trajectories, enabling training without differentiating through draft sampling. The linear objective reduces to TV loss for a single-token block. For practical optimization, the authors use a vocabulary-integrated estimator, a blockwise logarithmic surrogate, and an annealing schedule. Experiments with DFlash and DSpark drafters for Qwen3-4B/8B show improvements over several tokenwise objectives across math, code, and chat benchmarks. The gains also extend to token verification and greedy decoding.

Critical strengths

A well-motivated training objective. The change-of-measure formulation provides a direct connection between drafter training and expected accepted length, while allowing estimation from target-generated data. Improvements are observed across Qwen drafter architecture, both target sizes, and multiple decoding settings, with corresponding gains in measured inference speed. This improvement introduces no additional inference overhead.

Critical weaknesses
  1. The effectiveness at larger training scales remains unclear. The experiments use 100K training examples, whereas block-diffusion drafters are often trained on substantially larger datasets. It would be helpful to evaluate whether BV’s gains over CE and LK persist when scaling to approximately 1M examples.
  2. The evaluation is limited to Qwen3-4B and Qwen3-8B. Results on another model family would help establish whether BV’s gains generalize beyond Qwen-specific distributional characteristics.
Questions for authors

BV shows similar relative improvements under token-wise and block verification. It would be helpful if the authors could offer some insight into this observation. Compared with BV loss, TV and LK seem more closely aligned with maximizing the expected accepted draft length under token-wise verification.

3 comments

Thank you for your careful and constructive review. We address each of your points below.

Weakness 1: Results with a Larger Training Dataset

Thank you for this suggestion. We agree that evaluating our method with a larger training dataset would provide valuable additional evidence of its effectiveness. Due to the substantial time required for training, our original experiments used a dataset of 100K examples. To address your concern, we conducted additional experiments using 800K training examples, the same dataset size as in the DFlash paper. Following our original data configuration, we used data generated by the target model based on the OpenPerfectBlend dataset.

Table A. Average accepted length on Qwen3-4B DFlash with 800K training examples. All results use T = 1 and block verification.

LossGSM8KMATH-500AIME25HumanEvalMBPPLCBMT-BenchAvg.
CE6.465.634.535.094.695.013.074.93
LK6.605.804.665.314.955.163.225.10
BV (ours)8.047.035.815.905.685.883.726.01

BV loss achieves the highest accepted length on all seven benchmarks. Its average accepted length reaches 6.01, compared with 4.93 for CE and 5.10 for LK. These results show that the benefits of BV loss remain substantial even when the training dataset is scaled up, further supporting the effectiveness of our method. We will include these additional results in the final version of the paper.

Weakness 2: Evaluation on Another Model Family

We agree that the experiments in our original submission were limited to the Qwen model family. Since our method is designed to be applicable across model families, we conducted additional experiments using Llama-3.1-8B-Instruct following our original experiment configs in paper. The results are shown below.

Table B. Average accepted length on Llama-3.1-8B-Instruct DFlash. All results use T = 1 and block verification.

LossGSM8KMATH-500AIME25HumanEvalMBPPLCBMT-BenchAvg.
CE3.923.352.754.193.963.282.453.41
LK4.113.532.834.444.213.502.503.59
BV (ours)4.894.003.045.054.853.972.764.08

BV loss consistently outperforms both baselines across all seven benchmarks. Its average accepted length is 4.08, compared with 3.41 for CE and 3.59 for LK. These results demonstrate that our method remains effective on Llama, extending our findings beyond the Qwen model family. We hope this additional evaluation addresses your concern about generalization across model families. In the final version, we will include these results into main paper Table 2.

Q1: Reason for the Improvement Under Token-Wise Verification

Thank you for this insightful question. Given that TV and LK are derived from the acceptance rule for token verification, it is natural to ask why BV loss can outperform them even under token verification. The key distinction is that optimizing the acceptance of individual tokens and optimizing the expected accepted length of a block are different objectives. TV optimizes single-token acceptance, but it does not directly optimize the expected accepted length of an entire draft block under token-wise verification.

In diffusion-based speculative decoding, tokens are drafted and verified in blocks even when token-wise verification is used. A later token contributes to the accepted length only if all previous tokens have been accepted. Consequently, acceptance at earlier positions is especially important because it determines whether later tokens can contribute at all.

TV and LK, by themselves, treat different token positions equally. Their per-token losses are therefore multiplied by fixed positional weight decay to account for these differences in importance. In contrast, BV loss is inherently prefix-aware and accounts for the greater importance of earlier positions without requiring a separate fixed positional weighting scheme. By capturing this prefix dependence, an objective derived for block verification can also benefit token-wise verification of draft blocks.

This interpretation is also consistent with Figure 3. The gains in prefix acceptance are relatively small at the beginning of the draft block and become more pronounced at later positions, indicating improved retention of longer prefixes.

We therefore interpret the improvements observed under token verification as evidence that learning the prefix structure shared by both verification schemes can generalize across verifiers. In the revised manuscript, we will make the distinction between single-token acceptance and expected accepted length more explicit and explain this generalization in greater detail.

We hope these additional experiments and analyses clarify the reviewer’s concerns.

Thank you for the detailed rebuttal and additional experiments. The larger-scale training results, evaluation on another model family, and clarification of the gains under token-wise verification largely address my concerns. I will raise my score to 3 and encourage the authors to incorporate these results and explanations into the revised paper.

We sincerely thank the reviewer for carefully considering our rebuttal and for raising the score. We are glad that the larger-scale training results, the evaluation on an additional model family, and the clarification of the gains under token-wise verification addressed your concerns.

As suggested, we will incorporate these results and explanations into the revised paper. Your constructive feedback has helped us substantially improve the paper, and we appreciate it.

Reviewer_sCdyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes BV Loss, a block verification-aware training objective for diffusion-based speculative decoding. By directly optimizing the expected accepted draft length under block verification, the proposed method addresses the mismatch between token-level training objectives and sequence-level verification. Experiments across multiple models and benchmarks demonstrate consistent improvements in acceptance length and decoding speed.

Critical strengths
  1. The paper identifies a meaningful mismatch between existing token-level training objectives and block-level verification, which is relevant to improving LLM inference efficiency.

  2. BV Loss is derived from the block verification acceptance rule, providing a clear theoretical connection between the training objective and the inference-time evaluation criterion.

  3. The method consistently outperforms several baselines, including CE, KL, TV, and LK losses, across different drafter architectures and target models, without modifying the inference procedure.

Critical weaknesses
  1. The general idea of aligning drafter training objectives with verification acceptance has already been explored in prior work, particularly LK Losses. The proposed BV Loss can be viewed as a natural extension of acceptance-aware optimization from token-level to block-level verification. The authors should better clarify the non-trivial technical challenges and conceptual advances beyond existing approaches.

  2. While the theoretical BV Loss is equivalent to maximizing the expected accepted length, the practical implementation employs a logarithmic surrogate with annealing. It is unclear whether this practical objective preserves the same theoretical optimality properties.

  3. It would be helpful to further disentangle the contributions of the block-aware objective, logarithmic transformation, and annealing strategy to determine which components are primarily responsible for the observed improvements.

Questions for authors
  1. What are the key conceptual or technical advances of BV Loss beyond a straightforward block-level extension of existing acceptance-aware objectives such as LK Losses?

  2. Does the practical log-scaled BV objective with annealing preserve the optimality properties established for the original BV objective?

  3. How does the performance advantage of BV Loss vary with block size? Could the authors provide additional experiments to support its generalizability?

  4. Could the authors provide a more detailed ablation study separating the effects of the block-aware objective, log scaling, and annealing?

4 comments

Thank you for your detailed review of our paper. We respond to your concerns and questions. We hope these responses address the points you raised.

W1, W3 / Q1, Q4: Straightforward extension, log Scaling, and Annealing

We agree that aligning the training objective with the verifier is a natural starting point, and our work is motivated by the same principle. However, BV loss cannot be obtained simply by replacing token-level acceptance with block-level acceptance. Such a direct extension actually underperforms a drafter trained with CE. Our contribution lies in identifying the optimization difficulties responsible for this behavior and developing a method to address them. Table A illustrates this progression and provides an ablation that isolates the contribution of each component.

Table A. Average accepted length for different objectives on Qwen3-4B / DFlash, with T = 1 and block verification.

ObjectiveGSM8KMATH-500AIME25HumanEvalMBPPLCBMT-BenchAvg.
CE5.554.823.884.013.763.962.654.09
TV2.392.432.161.711.761.621.621.96
Pure BV3.032.972.542.002.181.821.792.34
LK5.745.043.904.243.934.142.734.25
Log-BV w/o annealing6.355.674.684.544.324.553.034.74
Log-BV + annealing6.855.924.804.794.534.683.094.95

1. A straightforward extension does not work well.

The most direct extension is pure BV (Eq. 10), which maximizes the expected accepted length without further modification. Pure BV achieves an average accepted length of 2.34, better than TV (1.96) but falling below CE (4.09). Simply moving to a block-level objective is therefore insufficient to outperform CE.

2. Pure BV introduces two optimization difficulties arising from the block structure.

Under pure BV, the gradient contribution of each prefix is proportional to its acceptance score (Eq. 12), and acceptance scores are determined by cumulative likelihood ratios.

  • Vanishing gradients: The size of the overall gradient scales with the acceptance scores of all prefixes combined. When acceptance is low, this total is very small, so the entire drafter learns slowly.
  • Early-training signal imbalance across positions: Because likelihood ratios accumulate along a prefix, low acceptance at earlier positions drives the scores of longer prefixes close to zero. Early in training, the gradient is therefore concentrated almost entirely on the earlier positions, and later positions receive almost no learning signal. This imbalance is absent from TV and LK, whose loss terms evaluate individual positions independently.

3. Log-BV and annealing address these two difficulties.

  • Log-BV (Eq. 13): We apply a logarithmic transformation to the overall expected accepted length. This preserves the minimizers while normalizing the gradient by the expected accepted length, removing the acceptance-dependent attenuation of the overall gradient. In Table A, the average accepted length increases from 2.34 to 4.74.
  • Annealing (Eq. 15): At the beginning of training, we assign equal weights to all prefixes so that later positions also receive sufficient learning signal, and then gradually transition back to log-BV. This increasing the average accepted length from 4.74 to 4.95.

4. The contributions of the individual components can be separated as follows.

  • Block-aware objective: Holding the other conditions fixed, this comparison isolates the effect of moving from a token-level objective to a block-level objective. Without log scaling, the average accepted length improves from 1.96 with TV to 2.34 with pure BV.
  • Log scaling: The average accepted length improves from 2.34 with pure BV to 4.74 with log-BV. This is the largest improvement and makes optimization of the block-aware objective effective.
  • Annealing: Annealing provides a further improvement, increasing the average accepted length from 4.74 to 4.95.

The block-aware objective alone underperforms CE. However, once log scaling and annealing address the optimization difficulties introduced by the block structure, it achieves higher accepted lengths than both CE and LK. Our contribution therefore lies in the analysis and training methodology that make the block-level extension effective. We will include Table A as an ablation study in the revised manuscript.

W2 / Q2: Effects of Log Scaling and Annealing on the Objective

We clarify below how log scaling and annealing affect the objective.

Log scaling preserves the optimal solutions of the theoretical BV objective. The log-BV objective defined in Section 3.3 is a strictly increasing transformation of the pure-BV loss. Therefore, the two objectives have the same set of minimizing parameters. At the population level, this transformation also preserves the gradient direction while rescaling its magnitude, strengthening the learning signal when acceptance is low while preserving the direction that improves the expected accepted length:

log-BV gradient = pure BV gradient * (B / expected acceptance length)

As this shows, the gradient direction is the same, but the gradient is scaled up when acceptance is low.

Annealing does not change the objective optimized in the later stages of training. The annealing coefficient increases from 0 to 1 over the earlier stage of training and is fixed at 1 thereafter. From that point onward, training optimizes the same final log-BV objective as it would without annealing. The role of annealing is to give prefixes with low acceptance scores sufficient weight early in training, thereby adjusting the training trajectory to facilitate effective optimization of the final objective.

We will explain these theoretical properties of log scaling and the scope of annealing more clearly in the revised manuscript.

Q3: Generalization Across Block Sizes

We agree that evaluating BV loss across different block sizes is an informative way to assess its generalizability. We therefore trained and evaluated models with block sizes of 8, 16, and 32, comparing BV against CE and LK. The results are shown in Table B.

Table B. Average accepted length across different block sizes on Qwen3-4B / DFlash, with T = 1 and block verification.

Block sizeLossGSM8KMATH-500AIME25HumanEvalMBPPLCBMT-BenchAvg.
8CE4.824.433.773.903.733.832.653.87
8LK4.994.583.894.063.814.062.804.03
8BV (Ours)5.374.924.314.384.164.293.034.35
16CE5.554.823.884.013.763.962.654.09
16LK5.745.043.904.243.934.142.734.25
16BV (Ours)6.855.924.804.794.534.683.094.95
32CE5.534.693.643.803.643.752.533.94
32LK5.714.853.794.033.773.832.654.09
32BV (Ours)6.985.874.644.564.394.432.994.84

BV consistently outperforms both CE and LK across all seven benchmarks at every evaluated block size. At block size 8, however, the improvement over LK is smaller: the average accepted length increases from 4.03 to 4.35 (+0.32), compared with 4.25 to 4.95 (+0.70) at block size 16. This suggests that the advantage of BV loss is less pronounced at smaller block sizes.

Across all three objectives, the average accepted length decreases slightly when the block size increases from 16 to 32. One possible explanation is that allocating training signal to positions beyond 16 may adversely affect overall performance under the current training setup. Nevertheless, BV’s absolute advantage over LK increases slightly at block size 32, where the average accepted length improves from 4.09 to 4.84 (+0.75). This suggests that BV loss could provide additional benefits as future advances in speculative decoding enable more effective use of larger draft blocks.

These results show that the benefits of BV loss persist with both smaller and larger block sizes, although the magnitude of the improvement varies. We will include these results in the final version of the paper.

Thank you again for the constructive feedback. We hope these additional results address your concerns, and please let us know if you have any remaining concerns.

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal Discussion

We sincerely thank all reviewers for their time and constructive feedback, which has helped us substantially improve the paper.

The reviewers recognized the meaningful motivation behind the BV loss (sCdy, ab5B, mx5x). They also noted the meaningful LLM inference efficiency gains it achieves without modifying the inference procedure (sCdy, ab5B).

Below, we summarize how we addressed each reviewer's concerns during the rebuttal period.

Reviewer sCdy

  1. Clarifying the technical challenge: We explained why a naive extension of existing approaches does not work and described how our method addresses this challenge.
  2. Effects of the block-aware objective, log transformation, and annealing: We provided ablation studies isolating each component and explained the role each one plays.
  3. Sensitivity to block size: We ran additional experiments with block sizes of 8, 16, and 32. The results show that BV loss generalizes well across block sizes.

Reviewer ab5B

  1. Scaling to larger datasets: We trained on a dataset 8x larger than in our original experiments. The gains held at this larger scale.
  2. Generalization beyond the Qwen family: We added experiments on Llama-3.1-8B-Instruct. These show that BV loss generalizes to a different model family.
  3. Why BV loss also benefits token-level verification: We explained that this stems from the difference between verifying a single token and verifying tokens within a draft block.

After this discussion, Reviewer ab5B indicated that their concerns had been largely addressed and raised their rating from weak reject to weak accept.

Reviewer mx5x

  1. Which properties of block verification BV loss exploits: We explained the roles of prefix-aware acceptance and compensation across positions. We also gave a concrete example where BV loss outperforms TV loss.
  2. Benchmark quality: We clarified that BV loss does not change the verification procedure, so it cannot affect the output quality of speculative decoding. We confirmed this empirically: under T=0, benchmark quality is identical to that of the target model.
  3. Single training seed: We extended our experiments to three training seeds. The performance improvements are substantially larger than the variance across seeds.
  4. Additional analysis of BV loss: We presented results for pure BV loss, the log transformation, and annealing. We used them to explain why pure BV loss underperforms and why the log transformation and annealing are necessary.

Reviewer mx5x stated that most of their concerns had been addressed and raised one minor follow-up question. Other papers report slight variations between the outputs of the target model and speculative decoding, so how could our results be exactly identical? We explained that under greedy decoding (T=0), lossless speculative decoding produces exactly the same outputs as the target model. We also ran additional experiments at T=1. These showed only small variations, which were not statistically significant and are expected from sampling stochasticity.

We are grateful to all reviewers for the constructive discussion. We will incorporate the additional results and discussions from the rebuttal period into the final version of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.