- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper combines block verification with SCD and corrects the distribution of draft tokens reused across rounds. It speeds up image and video generation while preserving the target distribution, without training or extra target-model calls. Experiments cover three models and several window sizes.
Critical strengths
-
The paper identifies an easily overlooked issue in combining block verification with draft reuse and provides a correction supported by theory.
-
The method is simple and effective, with little added cost and speed improvements across all tested configurations.
Critical weaknesses
-
A comparison with SJD-PAC and a discussion of SJD++ would help show the advantage over related methods.
-
The benefit of reusing the draft is unclear without a comparison with resampling.
-
The gains are small on two models, and the differences across models are not well explained. Tests are limited to one GPU and batch size one.
-
The background is long, and the workflow figure could make the proposed changes easier to see.
Questions for authors
-
How does the method compare with SJD-PAC and SJD++ under similar settings?
-
How much does corrected reuse help compared with resampling the remaining draft?
-
Why do the gains vary across models, and do they hold with other batch sizes or sampling settings?
3 comments
Thank you for the additional SJD-PAC comparison, suffix-resampling ablation, and cross-round analysis. These experiments address my concerns about the missing baseline and clarify the value of corrected suffix reuse. However, the practical improvement remains modest on two of the three models, and the reasons for the model-dependent gains remain unexplained. The claim that throughput benefits extend to larger batches is also not established by per-sequence acceptance gains alone. While the response strengthens the evidence, the practical significance and scope of the improvement remain insufficiently established. I therefore maintain my original score.
Thank you for the reply, and for stating that the new experiments address your concerns about the missing baseline and clarify the value of corrected suffix reuse.
On the remaining concerns, we want to stress one point. SCD + BV is a "free lunch" over SCD. It is a small code change with no training and no extra target-model passes, and it comes with a proof of losslessness. It was faster than SCD in all 24 combinations of model and window that we tested, and every difference is statistically significant. For a model that is deployed at scale, a saving of a few percent of the inference cost is large in absolute terms, and it can make a real difference to profit margins. We argue that this merits publication at ICLR.
Thank you for your thoughtful review and questions.
W1 / Q1. SJD-PAC and SJD++
We have now run the official SJD-PAC code, which supports only Lumina-mGPT, under the same settings as our method. The full table is in our shared response to all reviewers. At window 64, which is the default in the SJD-PAC paper and also its fastest window in our runs, SCD + BV needs 13.8% fewer target-model calls and 15.7% less time than SJD-PAC.
Our method is lossless. We believe that the fair comparison is with other lossless methods, because a method that gives up this guarantee can trade accuracy for speed. SJD++ is not lossless, and to our knowledge its authors have not published code to reproduce it, so we do not compare with it.
W2. Cross-round effects
All results in the paper are measured over complete generations. The effect of a higher acceptance on the suffix that is reused in later rounds is therefore already included in every number that we report.
Nevertheless, we measured this effect for this rebuttal. We ran SCD on Lumina-mGPT at window 64 (1,000 prompts). At every call, we also computed how many tokens Block Verification would generate on average from the same draft tokens. This is the gain of Block Verification in a single round, without its effect on later rounds. In a single round, Block Verification generates 50% more tokens per call than the token-wise verification of SCD (about 6.4 compared with 4.28). Over complete generations, SCD + BV generates 19% more tokens per call than SCD (5.11 compared with 4.28). The gain over complete generations is smaller than the gain in a single round. The reason is the cross-round effect that you describe. After a round of SCD + BV, the draft tokens that remain for the next round are accepted less often than the draft tokens that remain after a round of SCD.
Q2. Reuse compared with resampling
We understand this question as a comparison of SCD + BV with SJD that uses Block Verification (SJD + BV). SJD resamples the draft tokens that are not accepted (the suffix), so SJD + BV is lossless without our correction. We ran this comparison for this rebuttal. The result is that SCD + BV needs 48% fewer calls than SJD + BV. The table gives the number of target-model calls per image on Lumina-mGPT at window 64, over 1,000 prompts, with the speedup over autoregressive generation in parentheses.
Reusing the suffix gives the larger gain. It removes about half of the calls with either verifier. Block Verification removes 16% to 17% of the calls, whether the suffix is resampled or reused. Our correction is what allows the two to be combined and remain lossless.
W3 / Q3. Variation of the gain
The speedup of SCD + BV over SCD is modest on some models and larger on others. However, it is present in every setting, and it is free. The 95% confidence interval of the reduction relative to SCD is above zero in all 18 combinations of model and window, for both calls and time. The method changes a few lines of SCD, remains provably lossless, and adds negligible overhead. Inference is expensive, so saving even a few percent of it matters.
We do not have an explanation for why the gain differs between the models, and this is an interesting question for future work. It does not change the contributions of the paper. We show how to combine Block Verification with SCD, we prove that the combination is lossless, and we show that it is faster than SCD on all three models.
The number of target-model calls does not depend on the hardware, so the reductions in calls carry over to any GPU.
We generate one image or video at a time, as in the SJD and SCD papers. The throughput of a batch is the average number of tokens generated per call divided by the time of a call. A larger batch can change the time of a call, with or without Block Verification. It does not change the gain of Block Verification in tokens per call, because verification acts on each sequence separately. Hence, the gain in throughput carries over to larger batches and to other hardware.
We use the sampling settings of the SCD paper, which differ between the models. SCD + BV improves on SCD under all three.
W4. Presentation
Thank you for these suggestions. We will make the contributions clearer and state them earlier in the introduction. We will also change Figure 1(b) so that it shows clearly which steps are the same as in SCD and which step is new in our method. We would like to keep the background, because many readers do not know both SCD and Block Verification.
In the revised paper, we will include the new experiments and the clarifications from this response. We believe that our answers resolve the concerns that you raised. We would be happy to answer any further question.