- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes SDVG, a speculatative decoding method for accelrating the autoregressive video diffusion. It uses 1.3B small model to serve as a drafter and a 14B large model as a main generator block. Also the paper uses the ImageReward to score the drafter candidate to mimic the LLM style probability score. On 1,003 MovieGenVideoBench prompts, this paper reports a 1.59× speedup at 98.1% of the target’s VisionReward, and 1.96× at 96.9% for the SDVG-hybrid.
Critical strengths
Strengths:
-
The speculatative method does accelarate the generating speed without lossing too much quality, which does solve the inference bottleneck, what's more the method is traninig free, which is even better.
-
The paper shows a comprehensive experiment work, also with honest and carefull designed ablation.
-
The figure and paper writing are clean and friendly to read.
Critical weaknesses
Weakness:
-
I am not sure if the reward router actually doing the work, since table 2 suggests most of the gain come from the block0 instead of reward-guided verification. With the anchor in place, random routing reaches nearly the same quality as SDVG. So it would be better to show that anchor + random baseline swept.
-
I am confused the defination of speculative decoding in video part, it seems like SDVG can not gurantee lossless and also the speedup comes from the target verifying several drafts in parallel.
-
To me, the quality evaluation is overstated. SDVG recovers 78–90% on VR, not 98%. Warping error recovers only 28–47%, which is consistent with a per-frame router that can’t see motion.
Please answer the weakness carefully since my score may vary depend on how the core claim look like
Questions for authors
-
It would be better to see some comparison just simply running the target for fewer denoising steps, with T-Stitch or SRDiffusion applied to the whole video at matched speedup, i am wonder how will the results looks like.
-
Section 3.1 says block 0 bypasses the drafter entirely, while Section 5 says drafter compute on block 0 is wasted. Which is correct?
-
Contribution (1) calls the threshold “calibration-free,” while Section 3.1 says it is “calibrated offline.” Which is it?
2 comments
Thank you so much for the rebuttal, my concern is addressed here, so there will be needed for modification in the final paper.
I maintain my score and good luck.
Dear Reviewer kZYi,
Thank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.
R1: Thank you for your insightful comment. Anchoring block 0 is part of SDVG's design. Beyond it, we swept anchored random routing over matched target budgets and measured closeness to the 14B target's own video (frame-wise PSNR and LPIPS, same prompt and seed; neither uses a reward model):
Table R1. Anchored random routing vs. SDVG, 1,003 prompts.
At every budget SDVG is closer to the target (paired 95% CIs on PSNR exclude zero), and the margin grows with the budget (+0.4, +0.9, +1.3 dB). The router concentrates the target where it matters: at τ = −0.7 it regenerates 4.0 blocks on the 35% of prompts it flags (random: 1.4), bringing them 5.6 dB closer to the target than random routing. We will add this sweep to the final version.
R2: You are right that SDVG is not lossless and does not verify in parallel. It follows reward-guided speculative decoding (RSD) [1], where a reward model, not the target's likelihood, decides whether each drafted step is kept. The speedup comes from skipping the target on accepted blocks: a draft block costs about one sixth of a target block to denoise. We will make this explicit in the final version.
R3: Thank you for the careful reading. We will report the recovered share of the draft-to-target gap instead of the ratio to target-only: SDVG recovers 79–90% of the VR gap across τ (VR scoring every block). Warp error follows the same trend: at τ = −0.7 SDVG recovers 46% of its gap, nearly twice anchor-only's 26%. Although the router scores frames, each decision covers a 12-frame block, and the frame-wise fidelity in Table R1 is sensitive to motion, since frames diverge from the target's wherever the motion does. The revised core claims are:
R1: Following your suggestion, we compared methods at about SDVG's speed (one GPU, 1,003 prompts):
Table R2. Matched-speed baselines, fidelity to the target-only video.
At matched speed, SDVG stays closest to the target's own output. Fewer steps also lose quality: the 2-step target scores VR 0.0682 vs 0.0753 for SDVG. Because the target is distilled to 4 steps, stitching cannot go beyond k = 3 (1.37×), whereas SDVG's τ extends the trade-off.
R2: Thank you for spotting this. In the paper's runs, the drafter still denoised block 0 and the result was discarded without scoring or decoding, so Section 5 is right (about 0.5 s per video on one GPU), and the reported timings are conservative. Skipping it leaves all outputs unchanged, and all new runs do so. We will correct both sections.
R3: τ is a single constant set once offline and never adapted per prompt or video; this is what "calibration-free" was meant to convey. We will reword it as "a fixed threshold set once offline".
[1] B. Liao et al. Reward-Guided Speculative Decoding for Efficient LLM Reasoning. ICML 2025.