acceptodds
Under review as a conference paper at ICLR 2027

Block Verification for Speculative Coupled Decoding

Abstract

Autoregressive image and video models generate long sequences of visual tokens, making token-by-token decoding slow. Speculative Coupled Decoding (SCD) accelerates generation without a separate draft model by reusing draft tokens across Jacobi iterations. Its token-wise verifier stops at the first rejection, whereas Block Verification can accept longer prefixes using the same target-model evaluation. We replace SCD's verifier with Block Verification. However, verification changes the distribution of the tokens left for reuse, so applying SCD's updates with their original stored distributions can bias the output. We derive the exact post-verification distributions and use them to update the tokens before reuse. The method requires only a small code change to SCD, with no training or extra target-model passes per iteration. We prove that the resulting decoder is lossless: it preserves the target-model sampling distribution exactly. Across two image generation models and one video generation model, our method reduces mean target-model calls by 2.1%-20.7% and mean generation time by 1.6%-19.9% compared with SCD at matching window sizes.

Primary area:
generative models
Keywords:
speculative decoding, block verification, autoregressive visual generation, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_daUKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines block verification with SCD and corrects the distribution of draft tokens reused across rounds. It speeds up image and video generation while preserving the target distribution, without training or extra target-model calls. Experiments cover three models and several window sizes.

Critical strengths
  1. The paper identifies an easily overlooked issue in combining block verification with draft reuse and provides a correction supported by theory.

  2. The method is simple and effective, with little added cost and speed improvements across all tested configurations.

Critical weaknesses
  1. A comparison with SJD-PAC and a discussion of SJD++ would help show the advantage over related methods.

  2. The benefit of reusing the draft is unclear without a comparison with resampling.

  3. The gains are small on two models, and the differences across models are not well explained. Tests are limited to one GPU and batch size one.

  4. The background is long, and the workflow figure could make the proposed changes easier to see.

Questions for authors
  1. How does the method compare with SJD-PAC and SJD++ under similar settings?

  2. How much does corrected reuse help compared with resampling the remaining draft?

  3. Why do the gains vary across models, and do they hold with other batch sizes or sampling settings?

3 comments

Thank you for your thoughtful review and questions.

W1 / Q1. SJD-PAC and SJD++

We have now run the official SJD-PAC code, which supports only Lumina-mGPT, under the same settings as our method. The full table is in our shared response to all reviewers. At window 64, which is the default in the SJD-PAC paper and also its fastest window in our runs, SCD + BV needs 13.8% fewer target-model calls and 15.7% less time than SJD-PAC.

Our method is lossless. We believe that the fair comparison is with other lossless methods, because a method that gives up this guarantee can trade accuracy for speed. SJD++ is not lossless, and to our knowledge its authors have not published code to reproduce it, so we do not compare with it.

W2. Cross-round effects

All results in the paper are measured over complete generations. The effect of a higher acceptance on the suffix that is reused in later rounds is therefore already included in every number that we report.

Nevertheless, we measured this effect for this rebuttal. We ran SCD on Lumina-mGPT at window 64 (1,000 prompts). At every call, we also computed how many tokens Block Verification would generate on average from the same draft tokens. This is the gain of Block Verification in a single round, without its effect on later rounds. In a single round, Block Verification generates 50% more tokens per call than the token-wise verification of SCD (about 6.4 compared with 4.28). Over complete generations, SCD + BV generates 19% more tokens per call than SCD (5.11 compared with 4.28). The gain over complete generations is smaller than the gain in a single round. The reason is the cross-round effect that you describe. After a round of SCD + BV, the draft tokens that remain for the next round are accepted less often than the draft tokens that remain after a round of SCD.

Q2. Reuse compared with resampling

We understand this question as a comparison of SCD + BV with SJD that uses Block Verification (SJD + BV). SJD resamples the draft tokens that are not accepted (the suffix), so SJD + BV is lossless without our correction. We ran this comparison for this rebuttal. The result is that SCD + BV needs 48% fewer calls than SJD + BV. The table gives the number of target-model calls per image on Lumina-mGPT at window 64, over 1,000 prompts, with the speedup over autoregressive generation in parentheses.

Suffix resampledSuffix reused
Token-wise verificationSJD: 1071.0 (2.01x)SCD: 550.2 (3.67x)
Block VerificationSJD + BV: 892.1 (2.35x)SCD + BV (ours): 461.2 (4.29x)

Reusing the suffix gives the larger gain. It removes about half of the calls with either verifier. Block Verification removes 16% to 17% of the calls, whether the suffix is resampled or reused. Our correction is what allows the two to be combined and remain lossless.

W3 / Q3. Variation of the gain

The speedup of SCD + BV over SCD is modest on some models and larger on others. However, it is present in every setting, and it is free. The 95% confidence interval of the reduction relative to SCD is above zero in all 18 combinations of model and window, for both calls and time. The method changes a few lines of SCD, remains provably lossless, and adds negligible overhead. Inference is expensive, so saving even a few percent of it matters.

We do not have an explanation for why the gain differs between the models, and this is an interesting question for future work. It does not change the contributions of the paper. We show how to combine Block Verification with SCD, we prove that the combination is lossless, and we show that it is faster than SCD on all three models.

The number of target-model calls does not depend on the hardware, so the reductions in calls carry over to any GPU.

We generate one image or video at a time, as in the SJD and SCD papers. The throughput of a batch is the average number of tokens generated per call divided by the time of a call. A larger batch can change the time of a call, with or without Block Verification. It does not change the gain of Block Verification in tokens per call, because verification acts on each sequence separately. Hence, the gain in throughput carries over to larger batches and to other hardware.

We use the sampling settings of the SCD paper, which differ between the models. SCD + BV improves on SCD under all three.

W4. Presentation

Thank you for these suggestions. We will make the contributions clearer and state them earlier in the introduction. We will also change Figure 1(b) so that it shows clearly which steps are the same as in SCD and which step is new in our method. We would like to keep the background, because many readers do not know both SCD and Block Verification.


In the revised paper, we will include the new experiments and the clarifications from this response. We believe that our answers resolve the concerns that you raised. We would be happy to answer any further question.

Thank you for the additional SJD-PAC comparison, suffix-resampling ablation, and cross-round analysis. These experiments address my concerns about the missing baseline and clarify the value of corrected suffix reuse. However, the practical improvement remains modest on two of the three models, and the reasons for the model-dependent gains remain unexplained. The claim that throughput benefits extend to larger batches is also not established by per-sequence acceptance gains alone. While the response strengthens the evidence, the practical significance and scope of the improvement remain insufficiently established. I therefore maintain my original score.

Thank you for the reply, and for stating that the new experiments address your concerns about the missing baseline and clarify the value of corrected suffix reuse.

On the remaining concerns, we want to stress one point. SCD + BV is a "free lunch" over SCD. It is a small code change with no training and no extra target-model passes, and it comes with a proof of losslessness. It was faster than SCD in all 24 combinations of model and window that we tested, and every difference is statistically significant. For a model that is deployed at scale, a saving of a few percent of the inference cost is large in absolute terms, and it can make a real difference to profit margins. We argue that this merits publication at ICLR.

Reviewer_QDuxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper combines Block Verification with Speculative Coupled Decoding (SCD) to accelerate autoregressive image and video generation. The main observation is that directly replacing SCD’s token-wise verifier with Block Verification is not sufficient, because the verification outcome changes the conditional distribution of the uncommitted suffix tokens that SCD reuses. The authors derive a post-verification distribution for these suffix tokens and use it in the subsequent MRS update. They also provide a losslessness argument and evaluate the method on two image models and one video model. Across the tested window sizes, the method consistently improves block efficiency and wall-clock generation time over SCD.

Critical strengths
  1. The paper identifies a subtle issue that arises specifically when Block Verification is combined with SCD-style suffix reuse.

  2. The proposed change is lightweight and practically useful. It requires no additional draft model, training, or extra target-model pass, and it improves over SCD in all tested model-and-window configurations.

Critical weaknesses
  1. Could the authors provide a small enumerable experiment comparing the output distribution of SCD+BV, naive SCD+BV without the corrected distribution, and exact autoregressive sampling?

  2. Why were SJD++, SJD-PAC, or other recent training-free visual autoregressive decoding methods not included in the experimental comparison?

  3. It seems this method is unrelated to images or videos; does it not work for text? If so, why?

  4. The relationship between speedup and model size. Could you include results for LlamaGen or other models that offer a range of different sizes?

Questions for authors

See Weakness.

1 comment

Thank you for your helpful review.

W1. Output distribution of SCD + BV, naive SCD + BV, and autoregressive sampling

To answer this, we built a small synthetic experiment. The target distribution generates 3 tokens, each 0 or 1, with the probabilities below. SCD + BV and naive SCD + BV both use 2 draft tokens.

TokenProbability that the token is 1
First0.9
Second0.4 if the first token is 1; 0.1 if it is 0
Third0.6 if the first token is 1; 0.4 if it is 0 (whatever the second token is)

Naive SCD + BV uses Block Verification but keeps the rest of SCD unchanged, as in the "Naive update" of Section 4.1 of the paper. The probabilities below are exact and not Monte Carlo estimates. We compute them by enumerating every outcome of each method's random draws. Bold entries differ from autoregressive sampling.

OutputAutoregressiveSCD + BV (ours)Naive SCD + BV
0005.40%5.40%5.33%
0013.60%3.60%3.67%
0100.60%0.60%0.60%
0110.40%0.40%0.40%
10021.60%21.60%21.60%
10132.40%32.40%32.40%
11014.40%14.40%18.00%
11121.60%21.60%18.00%

SCD + BV matches autoregressive sampling on every output, and naive SCD + BV does not. For example, after the tokens 1, 1, it outputs 1 as the third token with probability 0.5 instead of 0.6.

W2. Comparison with SJD++ and SJD-PAC

We have now run the official SJD-PAC code, which supports only Lumina-mGPT, and the full table is in our shared response to all reviewers. At window 64, the default in the SJD-PAC paper, SCD + BV needs 13.8% fewer target-model calls and 15.7% less time than SJD-PAC. Our method is lossless. We believe that the fair comparison is with other lossless methods, because a method that gives up this guarantee can trade accuracy for speed. SJD++ is not lossless, and to our knowledge its authors have not published code to reproduce it, so we do not compare with it.

W3. Applicability to text

The method applies to text, but its gain on text is almost zero. The algorithm and its proof hold for any autoregressive model over discrete tokens.

The draft consists of the tokens sampled in the previous iteration, before the tokens in front of them changed. In text, the next token depends strongly on the exact tokens before it. When one word changes, the old draft is rarely accepted. In an image, a different token often means only a slightly different texture or color. It hardly changes the distributions of the other tokens, so the later draft tokens are often accepted.

SJD-PAC measured this on Lumina-mGPT (its Section 4.2 and Figure 4). It reports that image tokens are less sensitive than text to changes in earlier tokens. For language models, CLLM (Kou et al., 2024) notes that Jacobi decoding "shows only marginal speedup over AR decoding in practice, e.g., an average of 1.05x speedup".

We also ran SJD, SCD, and SCD + BV on Qwen3.5-2B and Llama-2-7B at temperature 1. All three produce about 1.1 tokens per decoding step, compared with 1 for autoregressive generation and more than 5 for SCD + BV on Lumina-mGPT.

W4. Speedup and model size

To address this, we ran a new experiment. Janus-Pro comes in two sizes, so we ran both Janus-Pro-1B and Janus-Pro-7B on the same 1,000 prompts and the same window sizes.

Calls are target-model calls per image, with the change relative to SCD in parentheses. Speedups are over autoregressive generation. Bold marks the fastest window.

Window1B calls: SCD / SCD + BV7B calls: SCD / SCD + BV1B speedup: SCD / SCD + BV7B speedup: SCD / SCD + BV
16204.3 / 197.1 (-3.5%)182.9 / 176.9 (-3.2%)2.25x / 2.33x2.47x / 2.58x
32172.4 / 162.9 (-5.5%)146.7 / 138.7 (-5.4%)2.48x / 2.64x3.07x / 3.14x
64162.6 / 149.9 (-7.8%)135.4 / 124.7 (-7.9%)2.62x / 2.84x3.31x / 3.50x
128164.3 / 153.2 (-6.8%)136.0 / 123.5 (-9.2%)2.46x / 2.69x3.17x / 3.41x
256178.1 / 162.4 (-8.8%)136.6 / 122.3 (-10.5%)2.45x / 2.62x2.77x / 3.01x
512162.2 / 146.8 (-9.5%)134.8 / 119.3 (-11.5%)2.58x / 2.83x2.33x / 2.59x

SCD + BV needs fewer calls and reaches a higher speedup than SCD at every window for both model sizes. Its saving in calls relative to SCD does not shrink as the model grows. It is about equal for the two sizes up to window 64, and it is larger on the 7B model at windows above 64. The speedup over autoregressive generation is larger for the 7B model at every window up to 256.

Note that the gain from Block Verification comes for free: the method changes only a few lines of SCD, is provably lossless, and adds negligible overhead, so we see no reason not to use it whenever SCD is used.


We will add all of the above to the paper. We believe this addresses all four points, and we would be glad to answer any concern that remains.

Comments on the paper

2 comments

We thank all reviewers for their time and feedback. All three ask for a comparison with SJD-PAC and SJD++.

SJD-PAC

We ran the official SJD-PAC code, unmodified, with the configuration of its main results (K=4K=4, D=3D=3). The SJD-PAC paper evaluates Lumina-mGPT and Emu3, but its released code supports only Lumina-mGPT. This is one of the models in our paper, so we compare there. We ran all methods in the table again for this comparison. Prompts (1,000 from MS-COCO), sampling settings, software, and hardware are identical for all methods.

WindowMethodTarget-model callsTokens per callTime (s)Speedup
-Autoregressive2357.01.0071.41 ±\pm 0.441.00x
32SJD1072.8 ±\pm 3.12.20 ±\pm 0.0135.69 ±\pm 0.162.00x ±\pm 0.02
32SCD637.1 ±\pm 3.43.70 ±\pm 0.0222.25 ±\pm 0.143.21x ±\pm 0.02
32SJD-PAC798.1 ±\pm 2.42.95 ±\pm 0.0128.69 ±\pm 0.122.49x ±\pm 0.02
32SCD + BV (ours)571.2 ±\pm 3.14.13 ±\pm 0.0220.47 ±\pm 0.123.49x ±\pm 0.03
64SJD1071.0 ±\pm 3.02.20 ±\pm 0.0135.55 ±\pm 0.232.01x ±\pm 0.01
64SCD550.2 ±\pm 4.24.28 ±\pm 0.0319.43 ±\pm 0.163.67x ±\pm 0.03
64SJD-PAC534.8 ±\pm 3.54.41 ±\pm 0.0319.73 ±\pm 0.123.62x ±\pm 0.03
64SCD + BV (ours)461.2 ±\pm 3.55.11 ±\pm 0.0416.64 ±\pm 0.134.29x ±\pm 0.04
128SJD1073.3 ±\pm 3.22.20 ±\pm 0.0136.63 ±\pm 0.231.95x ±\pm 0.01
128SCD535.0 ±\pm 4.64.41 ±\pm 0.0419.06 ±\pm 0.163.75x ±\pm 0.03
128SJD-PAC498.6 ±\pm 4.34.73 ±\pm 0.0422.03 ±\pm 0.183.24x ±\pm 0.03
128SCD + BV (ours)435.1 ±\pm 4.25.42 ±\pm 0.0515.94 ±\pm 0.144.48x ±\pm 0.04

The ±\pm values are half-widths of 95% bootstrap confidence intervals. Speedup is relative to autoregressive generation.

Compared with SJD-PAC, SCD + BV needs 13.8% fewer calls and 15.7% less time at window 64, which is the default in the SJD-PAC paper and also its fastest window in our runs. When each method uses its fastest window (ours 128, SJD-PAC 64), SCD + BV needs 18.6% fewer calls and 19.2% less time.

SJD++

SJD++ is not lossless. We believe that the fair comparison is with other lossless methods, because a method that gives up exactness can trade accuracy for speed. To our knowledge, the authors of SJD++ have also not published code to reproduce it.


We will add this comparison and discussion to the paper.

Thank you for your detailed and constructive review.

W1. Size of the gain

We agree that the correctness analysis is a very important part of the paper. We will make this explicit in the abstract and the introduction.

The speedup of SCD + BV over SCD is modest on some models and larger on others. However, we want to emphasize that it is a "free lunch". Moving from SCD to SCD + BV changes only a few lines of code. It needs no training, no draft model, and no extra target-model passes, and the output distribution stays exactly the same. It was faster than SCD in all 18 combinations of model and window. Inference is expensive, so saving even a few percent of it matters. We see no reason to use SCD instead of SCD + BV.

A user would choose the window with the highest speedup over autoregressive generation, which is 128 for Lumina-mGPT, 64 for Janus-Pro, and 256 for Cosmos. We will change the ranges in the abstract to cover only these windows. They become 6.2% to 18.7% fewer calls and 3.3% to 16.3% less time than SCD. The current low end, 2.1%, is on Cosmos at window 16. It is not within the noise, because its 95% bootstrap confidence interval is 2.0% to 2.3%.

W2. SJD-PAC, SJD++, and tables

We have now run the official SJD-PAC code, which supports only Lumina-mGPT. The table is in our shared response to all reviewers. At window 64, the default in the SJD-PAC paper, SCD + BV needs 13.8% fewer target-model calls and 15.7% less time than SJD-PAC. We do not compare with SJD++ because it is not lossless and, to our knowledge, its authors have not published code to reproduce it. We will add tables with all the numbers behind Figure 2.

Q1. Speedup including the overhead

All times reported in the paper and in this response are end-to-end wall-clock times, so both the speedups below and the 1.6% to 19.9% range include the normalization overhead. The table gives the speedup over autoregressive generation at each model's fastest window and the time that SCD + BV saves relative to SCD.

Model (window)SCDSCD + BVTime saved vs. SCD
Lumina-mGPT (128)3.49x4.17x16.3%
Janus-Pro (64)3.41x3.52x3.3%
Cosmos (256)8.78x9.29x5.5%

Q2. SJD-PAC, SJD++, and the correction

Please find our answer on SJD-PAC and SJD++ in W2 and in our shared response to all reviewers.

Without its tree of alternative draft tokens, SJD-PAC is virtually the same as SCD. Like SCD, it verifies token by token, so the defect does not arise. With Block Verification and our correction, SJD-PAC without the tree would be our method. With the tree, Block Verification and our correction would first need to be extended to trees. This is future work, so we cannot say whether it would beat our method.

SJD++ has a similar defect. Its reused tokens do not follow the distribution that they are verified against, which is why SJD++ is not lossless. Its reuse rule is different from SCD's update, so our correction does not apply to it directly.

Q3. Distribution across inputs

The gain is spread over most inputs and is not driven by a few of them. At each model's fastest window, SCD + BV needs fewer calls than SCD on 94.6% of the inputs on Lumina-mGPT, 69.6% on Janus-Pro, and 82.6% on Cosmos.

Q4. Independence step in Theorem 1

Thank you for this suggestion. The proof of Theorem 1 states this step only briefly. We agree that a lemma will make it clearer, and we will add it to the revised paper.

Q5. CFG

All reported call reductions are measured on the CFG-batched pass. One call is one forward pass with both CFG branches. Every method is counted this way, including autoregressive generation. Verification uses the distribution after CFG, which is the one that autoregressive sampling draws from.


Thank you again for your careful review. We will add all the results and clarifications above to the revised paper. We believe that we have addressed all of your concerns. If any remains, we would be glad to discuss it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.