acceptodds
Under review as a conference paper at ICLR 2027

Speculative Decoding for Autoregressive Video Generation

Abstract

Autoregressive video diffusion has emerged as a promising paradigm for streaming video synthesis, yet accelerating inference remains a critical challenge. While speculative decoding is the dominant acceleration strategy for large language models (LLMs), adapting it to video generation remains non-trivial, as continuous spatiotemporal video tensors lack discrete token probability distributions for exact rejection sampling. To bridge this gap, we introduce SDVG, a training-free framework that brings speculative decoding to autoregressive video diffusion via reward-guided block verification. Instead of discrete logits, SDVG employs a lightweight image-quality router with worst-frame score aggregation to evaluate candidate blocks proposed by a compact draft model. Accepted drafts are directly committed to the target model's key-value (KV) cache, while rejected blocks trigger target regeneration. By enforcing mandatory target-based generation on the initial anchor block to fix global scene composition, a single calibrated threshold provides a simple, continuous control knob over the quality–speed trade-off. We further introduce SDVG-hybrid, which integrates step-level trajectory stitching on rejected blocks to reuse draft compute and incorporates an on-demand temporal cascade to minimize verification overhead. Evaluated on 1,003 MovieGenVideoBench prompts (832 × 480), standard SDVG achieves a 1.59× speedup while preserving 98.1% of target visual quality (VisionReward), reaching up to 2.05× acceleration at 95.9% retention (+17.4% over draft-only generation). Furthermore, SDVG-hybrid advances the Pareto frontier to a 1.88× speedup at 98.0% quality retention and 1.96× at 96.9% retention. Requiring no retraining or architectural modifications, SDVG serves as a practical, plug-and-play accelerator for modern autoregressive video pipelines.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Speculative Decoding, Video Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kZYiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SDVG, a speculatative decoding method for accelrating the autoregressive video diffusion. It uses 1.3B small model to serve as a drafter and a 14B large model as a main generator block. Also the paper uses the ImageReward to score the drafter candidate to mimic the LLM style probability score. On 1,003 MovieGenVideoBench prompts, this paper reports a 1.59× speedup at 98.1% of the target’s VisionReward, and 1.96× at 96.9% for the SDVG-hybrid.

Critical strengths

Strengths:

  1. The speculatative method does accelarate the generating speed without lossing too much quality, which does solve the inference bottleneck, what's more the method is traninig free, which is even better.

  2. The paper shows a comprehensive experiment work, also with honest and carefull designed ablation.

  3. The figure and paper writing are clean and friendly to read.

Critical weaknesses

Weakness:

  1. I am not sure if the reward router actually doing the work, since table 2 suggests most of the gain come from the block0 instead of reward-guided verification. With the anchor in place, random routing reaches nearly the same quality as SDVG. So it would be better to show that anchor + random baseline swept.

  2. I am confused the defination of speculative decoding in video part, it seems like SDVG can not gurantee lossless and also the speedup comes from the target verifying several drafts in parallel.

  3. To me, the quality evaluation is overstated. SDVG recovers 78–90% on VR, not 98%. Warping error recovers only 28–47%, which is consistent with a per-frame router that can’t see motion.

Please answer the weakness carefully since my score may vary depend on how the core claim look like

Questions for authors
  1. It would be better to see some comparison just simply running the target for fewer denoising steps, with T-Stitch or SRDiffusion applied to the whole video at matched speedup, i am wonder how will the results looks like.

  2. Section 3.1 says block 0 bypasses the drafter entirely, while Section 5 says drafter compute on block 0 is wasted. Which is correct?

  3. Contribution (1) calls the threshold “calibration-free,” while Section 3.1 says it is “calibrated offline.” Which is it?

2 comments

Dear Reviewer kZYi,

Thank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.

W1: Is the reward router doing the work? Table 2 suggests most of the gain comes from block 0; please sweep an anchored random baseline.

R1: Thank you for your insightful comment. Anchoring block 0 is part of SDVG's design. Beyond it, we swept anchored random routing over matched target budgets and measured closeness to the 14B target's own video (frame-wise PSNR and LPIPS, same prompt and seed; neither uses a reward model):

Table R1. Anchored random routing vs. SDVG, 1,003 prompts.

MethodTarget blocksPSNR (dB) ↑LPIPS ↓
Anchor-only114.20.460
Random, p = 0.941.5014.50.449
SDVG (Ours), τ = −1.51.4914.90.445
Random, p = 0.881.9814.90.439
SDVG (Ours), τ = −1.01.9815.80.429
Random, p = 0.822.4415.30.428
SDVG (Ours), τ = −0.72.4216.60.412

At every budget SDVG is closer to the target (paired 95% CIs on PSNR exclude zero), and the margin grows with the budget (+0.4, +0.9, +1.3 dB). The router concentrates the target where it matters: at τ = −0.7 it regenerates 4.0 blocks on the 35% of prompts it flags (random: 1.4), bringing them 5.6 dB closer to the target than random routing. We will add this sweep to the final version.


W2: SDVG neither guarantees lossless generation nor verifies drafts in parallel; how is it speculative decoding?

R2: You are right that SDVG is not lossless and does not verify in parallel. It follows reward-guided speculative decoding (RSD) [1], where a reward model, not the target's likelihood, decides whether each drafted step is kept. The speedup comes from skipping the target on accepted blocks: a draft block costs about one sixth of a target block to denoise. We will make this explicit in the final version.


W3: The quality claim is overstated: SDVG recovers 78–90% of the VR gap, not 98%, and only 28–47% of the warp-error gap.

R3: Thank you for the careful reading. We will report the recovered share of the draft-to-target gap instead of the ratio to target-only: SDVG recovers 79–90% of the VR gap across τ (VR scoring every block). Warp error follows the same trend: at τ = −0.7 SDVG recovers 46% of its gap, nearly twice anchor-only's 26%. Although the router scores frames, each decision covers a 12-frame block, and the frame-wise fidelity in Table R1 is sensitive to motion, since frames diverge from the target's wherever the motion does. The revised core claims are:

  1. Block-level speculative generation for AR video diffusion, with the first block anchored on the target.
  2. Reward-guided routing that concentrates target compute where the draft deviates, keeping outputs closer to the target than content-blind routing at equal compute (Table R1).
  3. Gap-based evaluation: the recovered share of the draft-to-target gap, plus fidelity to the target's own video.

Q1: How do fewer target steps, or T-Stitch / SRDiffusion on the whole video, compare at matched speedup?

R1: Following your suggestion, we compared methods at about SDVG's speed (one GPU, 1,003 prompts):

Table R2. Matched-speed baselines, fidelity to the target-only video.

MethodTime / videoPSNR (dB) ↑LPIPS ↓
Target, 2 steps24.1 s12.30.534
Stitching on all blocks (T-Stitch, k = 3)25.0 s14.70.444
SDVG (Ours), τ = −0.724.1 s16.60.412

At matched speed, SDVG stays closest to the target's own output. Fewer steps also lose quality: the 2-step target scores VR 0.0682 vs 0.0753 for SDVG. Because the target is distilled to 4 steps, stitching cannot go beyond k = 3 (1.37×), whereas SDVG's τ extends the trade-off.


Q2: Section 3.1 says block 0 bypasses the drafter, but Section 5 says drafter compute on block 0 is wasted. Which is correct?

R2: Thank you for spotting this. In the paper's runs, the drafter still denoised block 0 and the result was discarded without scoring or decoding, so Section 5 is right (about 0.5 s per video on one GPU), and the reported timings are conservative. Skipping it leaves all outputs unchanged, and all new runs do so. We will correct both sections.


Q3: Is the threshold "calibration-free" or "calibrated offline"?

R3: τ is a single constant set once offline and never adapted per prompt or video; this is what "calibration-free" was meant to convey. We will reword it as "a fixed threshold set once offline".

[1] B. Liao et al. Reward-Guided Speculative Decoding for Efficient LLM Reasoning. ICML 2025.

Thank you so much for the rebuttal, my concern is addressed here, so there will be needed for modification in the final paper.

I maintain my score and good luck.

Reviewer_q7mjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses inference acceleration for autoregressive (AR) video diffusion models. These models generate a video block by block with a diffusion model and keep the previously generated blocks in a KV cache. Inspired by speculative decoding in LLMs, the authors let a cheap draft model generate each block and score it with an ImageReward-based scorer, taking the minimum score over the block's frames. If the score exceeds a threshold, the draft block is committed to the KV cache as is. Otherwise, the target model regenerates the block. The first block is always generated by the target to fix the scene composition. An extension, SDVG-hybrid, adds two changes. On rejected blocks, the target resumes from the draft's intermediate denoising result and runs only the remaining steps. It also reduces verification cost by scoring a subset of frames first.

Experiments use Krea Realtime 14B as the target and Wan2.1-1.3B as the draft on MovieGenVideoBench. SDVG retains 95–98% of the target's quality at roughly 1.6–2× speedup over target-only generation. SDVG-hybrid pushes the speedup further at similar quality. The authors also show that the threshold serves as a flexible knob over the speed–quality trade-off.

Critical strengths
  1. A practical, training-free acceleration method. The method requires no retraining or architectural changes to either model and can be plugged into existing AR video pipelines as is. A single threshold controls the speed–quality trade-off. The speedups are measured in wall-clock time on 1,003 prompts, which shows the effect in a realistic setting.

  2. An interesting finding about the first block. Generating only the first block with the target and filling the rest with the draft already recovers most of the quality gap between draft-only and target-only generation (VisionReward 0.0644 → 0.0757, vs. 0.0788 for the target). Conversely, applying stitching to the first block degrades quality significantly. This suggests that the quality of AR video depends heavily on the first block through the KV cache, and it could motivate follow-up work that concentrates compute on early blocks.

Critical weaknesses
  1. It is hard to call this speculative decoding. The key benefits of speculative decoding in LLMs are lossless acceleration, which preserves the target distribution exactly, and parallel verification, where the target checks several draft tokens in a single forward pass. SDVG has neither. An external reward model does the verification, and nothing guarantees the output distribution. A rejected block costs more than running the target alone, because it goes through both models. The method is therefore a relaxed cascade that trades quality for speed. In that case, simpler ways of trading quality for speed should serve as baselines. The most direct one is reducing the number of denoising steps of the target (e.g., from 4 to 3 or 2), but the paper provides no such comparison.

  2. Reward-guided routing performs no better than random routing.

    • When the target generates the first block in both cases, random routing (70.3% acceptance) reaches 0.0771 VisionReward and SDVG (τ = −0.7, 73.1% acceptance) reaches 0.0773. The two are essentially the same, and random routing is even faster (1.67× vs. 1.59×).
    • Generating only the first block with the target and accepting all other drafts without any verification (0.0757 at 2.21×) is faster than SDVG at τ = −1.5 (0.0757 at 1.88×) and τ = −2.0 (0.0756 at 2.05×), with equal or better quality.
    • The paper concludes that reward-guided routing is indispensable by comparing it against a random baseline that also randomizes the first block (0.0706). Most of that gap comes from the first block, not from the router.

    For the same reason, it is unclear whether the gains of SDVG-hybrid come from routing or from stitching. Separating the two would require baselines such as "target first block + random routing + stitching" and "target first block + stitching on all blocks." The paper reports neither.

Questions for authors

Q1. With the target generating the first block in both cases, is there evidence that SDVG is clearly better than random routing? For example, can the authors report paired per-prompt comparisons or results over multiple seeds at a matched acceptance rate or matched speed?

Q2. How exactly are the KV caches of the draft and the target kept in sync? When a draft block is accepted, the target presumably needs an extra forward pass over that block to build its own KV entries. When the target regenerates a block, the draft's cache must also be updated. How is this implemented, and is its cost included in the reported speedups?

4 comments

Dear Reviewer q7mj,

Thank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.

W1: Hard to call this speculative decoding; simpler baselines (e.g., fewer target steps) are missing.

R1: Thank you for your insightful comment. We agree that SDVG is not lossless and does not verify in parallel; it is closer to reward-guided speculative decoding (RSD) [1], where a reward model, not the target's likelihood, decides whether each drafted step is kept. We will make this explicit in the final version.

Following your suggestion, we compared with fewer target steps (one GPU, 1,003 prompts; VR scores every block here, so values are below the paper's):

Table R1. Fewer target steps vs. SDVG.

MethodTime / videoVR ↑
Target, 4 steps34.2 s0.0766
Target, 3 steps29.2 s0.0768
Target, 2 steps24.1 s0.0682
SDVG (Ours), τ = −0.724.1 s0.0753

Fewer steps cannot match SDVG's speed and quality at once: 3 steps keeps quality at only 1.17×, and 2 steps matches SDVG's time but loses 0.0071 VR.


W2: Reward-guided routing is no better than random routing once block 0 is anchored.

R2: With block 0 anchored in both, we compare SDVG and random routing at equal target compute by closeness to the 14B target's own video (frame-wise PSNR and LPIPS, same prompt and seed).

Table R2. Fidelity to the target-only video, 1,003 prompts.

MethodTarget blocksPSNR (dB) ↑LPIPS ↓
Anchor-only114.20.460
SDVG (Ours), τ = −1.51.4914.90.445
Random, p = 0.822.4415.30.428
SDVG (Ours), τ = −0.72.4216.60.412

SDVG is clearly closer to the target than random routing at equal compute, and at τ = −1.5 also closer than anchor-only (paired 95% CIs exclude zero). The router concentrates the target where it matters: on the 35% of prompts it flags, it regenerates 4.0 blocks (random: 1.4), bringing them 5.6 dB closer to the target than random routing. We will add it to the final version.


W3: It is unclear whether SDVG-hybrid's gains come from routing or from stitching.

R3: We ran both baselines you proposed (block 0 on the target, 1,003 prompts, k = 3):

Table R3. Target steps: 4 per fully generated block, 1 per stitched block.

MethodTarget stepsVR ↑PSNR (dB) ↑
Random routing + stitching, p = 0.825.440.074214.3
SDVG-hybrid (Ours), τ = −0.75.450.074614.3
SDVG (Ours), τ = −0.79.680.075316.6
Stitching on all blocks (T-Stitch)120.077114.7
  1. Routing vs. stitching. SDVG-hybrid's gain over SDVG comes mainly from stitching; at the same stitching budget, SDVG-hybrid scores slightly higher VR than random routing (0.0746 vs 0.0742).
  2. Why routing is still needed. T-Stitch [2] was designed for models sampled with many steps (e.g., 100 DDIM steps), allowing fine-grained switching. Our target is distilled to 4 steps, so k = 3 is already the most the draft can take, capping stitching at 1.37× on one GPU. SDVG-hybrid adds routing on top, so τ trades quality for speed beyond this cap (1.78× at τ = −1.0).
  3. Fidelity. SDVG stays closest to the target with fewer target steps than stitching every block.

We will add both baselines to the final version.


Q1: Is SDVG clearly better than random routing (paired per-prompt or multi-seed, at matched acceptance or speed)?

R1: Yes. Paired per-prompt differences (SDVG − random) in fidelity to the target, with 95% bootstrap CIs over 1,003 prompts:

Table R4. Paired comparisons with random routing.

SDVG vs randomMatchedΔPSNR (dB) ↑ΔLPIPS ↓
τ = −0.7 vs p = 0.82compute+1.3 [+0.9, +1.8]−0.016 [−0.025, −0.008]
τ = −1.0 vs p = 0.88compute+0.9 [+0.5, +1.2]−0.010 [−0.017, −0.003]
τ = −0.7 vs p ≈ 0.79speed (1.59× vs 1.67×)+1.0 [+0.6, +1.5]−0.0089 [−0.0179, −0.0004]

Every interval excludes zero: SDVG is consistently closer to the target's output at matched compute and speed, even when random routing uses more target blocks (last row).


Q2: How are the draft and target KV caches kept in sync, and is the cost included?

R2: Before each block, both models rebuild their caches with one t = 0 pass over the committed context (first latent frame plus the two latest), whichever model produced it. This costs 4.30 s (target) and 0.68 s (draft) per video on one GPU and is included in all reported timings.

[1] B. Liao et al. Reward-Guided Speculative Decoding for Efficient LLM Reasoning. ICML 2025.

[2] Z. Pan et al. T-Stitch: Accelerating Sampling in Pre-Trained Diffusion Models with Trajectory Stitching. ICLR 2025.

Dear Reviewer q7mj,

We hope that our response and revision have adequately addressed your concerns. If our efforts have indeed addressed your concerns, we would be very grateful if you could reconsider our work and possibly adjust the score accordingly. If you have any additional questions or suggestions, we would be happy to have further discussions.

Best regards,

The Authors

Thank you for the response. Most of my concerns are addressed, but W2 remains. Could you report the VR comparison between SDVG and anchored random routing at matched compute? I think the PSNR results are hard to interpret because, with a shared seed, PSNR is likely to favor concentrating target blocks within the same prompts.

Dear Reviewer q7mj,

Thank you for your follow-up and for the insightful point on PSNR. Following your suggestion, we compare VR at matched target compute, with block 0 anchored in both (1,003 prompts; VR scores every block, as in Table R1).

Table R4. VR of SDVG vs. anchored random routing at matched compute.

SDVG / RandomTarget blocksSDVG (Ours) VR ↑Random VR ↑
τ = −0.7 / p = 0.822.42 / 2.440.07530.0759
τ = −1.0 / p = 0.881.98 / 1.980.07440.0745
τ = −1.5 / p = 0.941.49 / 1.500.07390.0738
  1. VR. At matched compute, SDVG shows no significant advantage over anchored random routing; the differences are close to zero.
  2. PSNR. We agree that, with a shared seed, a prompt whose blocks all go to the target reproduces the target video almost exactly, so PSNR favors concentrating target blocks in a few prompts, and the PSNR gap comes from this concentration. This concentration is also well aimed in VR terms: on the prompts SDVG flags, target-only exceeds anchor-only by 0.0083 VR, versus only 0.0005 on the prompts it leaves to the draft. Thus, we believe PSNR, with such similar pattern compared to VR, reflects a real advantage of SDVG. Reviewer JuGY also stressed this as closeness to what the 14B target would have produced. When most of a prompt's blocks go to the target, its video stays closer to the target's trajectory, whereas a few scattered target blocks cannot undo the drift from earlier drafted blocks. At matched compute, SDVG therefore stays closer to the target (PSNR 16.6 vs. 15.3 dB, LPIPS 0.412 vs. 0.428; Table R2).
  3. Why the VR gap is hard to widen. SDVG needs a small draft and a large target that share the latent space and the block-causal KV cache; few public models qualify, and Krea Realtime 14B was the only large target we could find for the 1.3B draft. In this pair, the 4-step distilled 14B target leaves little VR headroom over the draft beyond block 0 (anchor-only 0.0734 vs. target-only 0.0766), so routing between the two has only a 0.0032 band to work with, which makes policies hard to separate. Nevertheless, we have explored the design space extensively (thresholds, score aggregation, anchoring, stitching), and the existing signs are encouraging: SDVG keeps 98% of the target's VR at 1.42× speed on one GPU, which fewer target steps cannot match (Table R1).

We will add Table R4 to the final version, state that SDVG and anchored random routing are on par in VR, and clarify that SDVG's fidelity gain comes from concentrating the target. We hope these results help you re-evaluate our work and its potential.

Reviewer_JuGYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues block-wise autoregressive video diffusion admits speculative decoding, since each block is a self-contained spatiotemporal unit before entering the KV cache. Because video blocks are continuous and have no logits, exact rejection sampling is replaced by a learned router: a 1.3B Self-Forcing draft proposes a block in four denoising steps, the block is decoded and scored by ImageReward with min-over-frames aggregation, and it is accepted into the target's KV cache if the score exceeds τ, otherwise regenerated by a 14B Krea Realtime target. Block 0 always goes to the target to fix scene composition. SDVG-hybrid reuses the first k=3 draft steps on rejection (trajectory stitching, after T-Stitch) and adds a cascade that scores 4 of 12 frames first, falling back to all 12 only inside (τ−δ, τ+δ). Evaluation is on 1,003 MovieGenVideoBench prompts at 832×480, nine blocks per video, with threshold sweeps, random-routing controls, aggregation comparisons and anchor-block ablations.

Critical strengths

The framing fits the architecture, and the paper is honest about what it gives up. The block boundary is the natural verification unit for causal video diffusion, and the limitation that the target distribution is not preserved exactly is stated plainly. The distinction from step-level stitching (T-Stitch, SRDiffusion) and request-level routing (MoDM) is drawn correctly: those operate on timesteps or whole requests and ignore causal structure.

The controls are the right ones. Random routing at matched acceptance rate (0.0706 vs 0.0773) rules out the possibility that the gain merely reflects regenerating a fixed fraction of blocks, and the anchor-block ablation (0.0766 → 0.0677 at k=3 when block 0 is stitched) isolates a real and non-obvious failure mode. That the authors went looking for these controls counts in the paper's favour.

Critical weaknesses

The quality metric shares its measurement basis with the router. Retention is VisionReward relative to target-only, where VisionReward is a learned preference reward model and the router uses ImageReward — same family, overlapping preference data. A router that admits blocks when a preference model likes them will produce outputs a second preference model also likes, independently of whether they approach what the 14B target would have produced. The paper's own limitation concedes ImageReward scores frames independently and cannot see temporal consistency, so the retained quantity is precisely what the router is blind to. There is no human study, and EvalCrafter does not settle it: at τ = −0.7, VQA_T moves 93.33 → 93.11 while warp error goes 0.0066 → 0.0085 — motion quality measurably degrades while the preference score barely moves.

Only one draft/target pair is tested, and the pair is unusually favourable. The target (Krea Realtime 14B) is Self-Forcing-distilled from Wan2.1-T2V-14B and the draft is the 1.3B Self-Forcing model: same backbone family, same distillation recipe, same schedule. High acceptance (73–87%) is expected almost regardless of routing design, and the "force-reject, no verification" ablation reaching 0.0757 at 2.21× confirms most of the gain comes from the pair being interchangeable. The claim of a general plug-and-play accelerator is therefore unsupported by a single pair, and the practically relevant case — draft and target from different families — is untested. Two further points: the 97.0 s baseline and SDVG timings come from a two-GPU setup with an asynchronous reward stream on GPU 1, so pipeline engineering is folded into the comparison unseparated; and the headroom is narrow (draft-only is 3.77×), so the method captures a modest slice.

Questions for authors
  1. Can you give a human preference study, or a metric outside the reward-model family (FVD against target-only outputs, motion/flow consistency)? The EvalCrafter warp error suggests the retained quantity is not the whole of quality.
  2. How much acceptance survives when the draft is not a Self-Forcing distillation of the same backbone? Have you tried a Wan draft with a CogVideoX- or Hunyuan-based target?
  3. What is the wall-clock breakdown of the 97.0 s baseline versus the SDVG runs, including reward passes, VAE decodes and cross-GPU transfers? Is the speedup present on a single GPU without the asynchronous stream?
  4. "Force-reject only, no verification" reaches 0.0757 at 2.21×. What does the full method add over that policy at matched latency (tuned to 1.59×)?
  5. On rejected blocks the target runs one step of a four-step schedule. Given that k=4 collapses to 0.0752, how stable is this across prompts, and was k selected per-model or is it transferable?
2 comments

Dear Reviewer JuGY,

Thank you for recognizing the potential and effectiveness of our work and for providing detailed constructive comments. Below, we address each point raised.

W1 & Q1: VisionReward shares its basis with the router; is there a metric outside the reward-model family?

R1: Thank you for your insightful comment. We test exactly your criterion, closeness to what the 14B target itself generates: frame-wise PSNR and LPIPS against the target's own video for the same prompt and seed, neither using a reward model. Random routing also gives block 0 to the target and accepts each later draft with probability p.

Table R1. Fidelity to the target-only video, 1,003 prompts.

MethodTarget blocksPSNR (dB) ↑LPIPS ↓
Draft-only09.30.709
Anchor-only (force-reject only)114.20.460
Random, p = 0.822.4415.30.428
SDVG (Ours), τ = −0.72.4216.60.412

At equal target compute, SDVG is closer to the target than content-blind routing on both metrics (paired 95% CIs exclude zero). A human study is not feasible in the discussion period; we will report both metrics and list it as a limitation.


W2 & Q2: Only one, unusually favourable draft/target pair is tested.

R2: You are right: this pair is unusually favourable. We will drop the "plug-and-play" claim and scope SDVG to pairs that share a latent space, so draft blocks can enter the target's KV cache; a Wan draft with a CogVideoX or HunyuanVideo target does not. LLM speculative decoding has the same constraint (same family, shared tokenizer).


W3 & Q3: Wall-clock breakdown; speedup on one GPU without the asynchronous stream?

R3: Yes. We timed each stage on one RTX PRO 6000 (96 GB), synchronizing after every stage (no overlap), and compared with the paper's two-A6000 layout with the stream (same code, 29 prompts).

Table R2. Seconds per video (KV = KV-cache refresh, decode = VAE decode).

MethodTarget KVDraft KVDraft denoiseDraft decodeRewardTarget denoiseFinal decodeOtherTotal, 1 GPUTotal, 2 GPUs
Target-only4.3––––25.53.80.634.297.1
SDVG, τ = −0.74.30.73.83.52.87.21.00.724.1 (1.42×)67.6 (1.44×)
SDVG-hybrid, τ = −1.04.30.73.83.51.14.20.90.719.2 (1.78×)52.6 (1.85×)

These results demonstrate that the speedup does not rely on the stream. These prompts are harder than average (SDVG regenerates 1.62 blocks vs 1.42 overall), hence 1.44× rather than 1.59×.


W4 & Q4: Narrow headroom (draft-only 3.77×); what does SDVG add over force-reject only?

R4: 1.59× out of 3.77× understates SDVG's gain: 3.77× is the speed of giving up quality entirely, and fewer target steps is much worse at equal time (a 2-step target takes SDVG's single-GPU time, 24.1 s, but scores VR 0.0682 vs 0.0753). SDVG saves 51–70% of the 71.3 s that draft-only saves, while recovering 79–90% of the VR gap (VR here scores every block, so values are below the paper's).

Force-reject only offers no way to improve quality further, and its videos are not well aligned with the target. Even with random routing added to roughly match SDVG's speed, its alignment still lags SDVG:

Table R3. Paper timings, 1,003 prompts.

MethodTime / videoTarget blocksVR ↑PSNR (dB) ↑LPIPS ↓
Anchor-only43.8 s (2.21×)10.073414.20.460
Random, p ≈ 0.7958.2 s (1.67×)2.670.074715.60.420
SDVG (Ours), τ = −0.760.9 s (1.59×)2.420.075316.60.412

At matched latency, SDVG uses fewer target blocks (2.42 vs 2.67), yet its output is closer to the target's own result (paired 95% CIs on PSNR and LPIPS exclude zero).


Q5: How stable is the one-step correction across prompts, and was k selected per model?

R5: k matters only on the 355 prompts (35%) with a rejected block; on the other 648, every k gives the same video. On those 355 prompts (τ = −0.7; VR on the paper's grid; time over all prompts):

Table R4. Target steps on a rejected block.

MethodTime / videoVR ↑PSNR (dB) ↑LPIPS ↓Closer than k = 4
k = 4 (0 steps, ≈ anchor-only)52.5 s0.063513.910.482—
k = 3 (1 step)54.3 s0.067714.160.47594%
k = 2 (2 steps)56.2 s0.066414.300.47097%
k = 0 (4 steps, SDVG)60.9 s0.069420.610.34594%

The correction is stable: one target step moves 94% of these prompts closer to the target (PSNR) and raises VR. Fidelity rises monotonically with target steps, so k is a speed–fidelity knob, not a fragile optimum. k = 3 was chosen on this pair; its transfer is untested.

Reviewer JuGY,

We hope that our response and revision have adequately addressed your concerns. If our efforts have indeed addressed your concerns, we would be very grateful if you could reconsider our work and possibly adjust the score accordingly. If you have any additional questions or suggestions, we would be happy to have further discussions.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.