acceptodds
Under review as a conference paper at ICLR 2027

BlockDelta: Block Residual Reconstruction for Frame-Sparse Few-Step Video Diffusion

Abstract

Few-step distillation accelerates video diffusion models, yet each remaining step is still computationally expensive, and on 4-step models, cross-step feature reuse yields limited benefits. Frame-sparse inference alleviates this cost by computing only selected frames (anchors) within each transformer block and reconstructing the skipped frames from them. Its effectiveness hinges on two design choices: (1) what to reconstruct for skipped frames, and (2) how to estimate this target from anchors. We propose BlockDelta, a training-free frame-sparse method that systematically addresses both. For (1), we reconstruct the block residual, the difference between a block's output and input, rather than the full output. Reconstructing the full output implicitly re-estimates the already-known input, introducing unnecessary approximation error. Theoretical analysis and empirical evaluation confirm that residual reconstruction reduces block-wise error and boosts generation quality. For (2), we find that lower local reconstruction error does not necessarily yield higher fidelity to dense inference, i.e., sampling without acceleration. Averaging all anchors has the lowest local error but over-smooths inter-frame differences and leads to larger deviation from dense inference, whereas linear interpolation between the two nearest enclosing anchors preserves temporal structure more faithfully, and BlockDelta uses it. On distilled Wan2.2 and HunyuanVideo-1.5, BlockDelta reduces per-step computation by nearly half and keeps the VBench score within 1.1 points of dense inference, while part of the motion is lost, most on Wan2.2 T2V. It also applies to recent audio–video diffusion models.

Primary area:
generative models
Keywords:
diffusion model, video generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zmp1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces an inference-time acceleration method that interpolates the residual (changes) after the DiT block. Specifically, after block i, the delta of the anchor feature (net changes from the input of block i) is used to interpolate the non-anchor (skipped frames) delta. The paper analyzes when residual reconstruction reduces local error and why minimizing that error alone can suppress frame-specific variation. Experiments across several video diffusion models evaluate generation fidelity, efficiency, reconstruction choices, and implementation safeguards, including comparisons with step reduction and caching.

Critical strengths
  1. Simple yet effective and well-motivated novel method. Preserving an already available input and reconstructing only the delta introduced by model block is a clear design choice. Table 1 isolates this change under the same FIS-DiT anchor, dense-block, and dense-step settings, with identical PFLOPs and nearly identical latency. The proposed method has minimal performance degradation compared with dense inference and strong speedup. It is also compatible with SVG to further speed up with similar, sometimes better, performance

  2. Solid ablations and analysis. Equations and propositions provide solid analysis, and the experiments are thorough and complete in analyzing the effect of each component.

Critical weaknesses
  1. Weak motion may favor the evaluation benchmark. The issue is (and the author also acknowledged) evaluation benchmark can possibly tend to be better when motion is weaker. Table 7 and some visual examples show that the motion is sometimes weaker than the dense baseline. I would like to see evaluation metrics where the author can somehow cap the motion (filtering based on motion). This can rule out the effect of motion strength.
Questions for authors

See weakness. I would like to see an evaluation under a motion filter (generate a lot of videos, filter, and regenerate until a fixed number of videos, etc). I will preserve the WA rating if the evaluation is provided.

3 comments
AuthorsOpenReview ↗

Response to Reviewer zmp1

W1: We thank the reviewer for this suggestion. We ran the evaluation under a motion filter on all VBench dimensions of the three models, comparing videos of the same prompt with nearly equal RAFT motion. At matched motion, residual reconstruction scores higher than FIS-DiT on all ten dimensions of both I2V models, and imaging and aesthetic quality on Wan2.2 show no drop whose 95% interval excludes 0, nor do the eight semantic and style dimensions of Wan2.2 T2V. The consistency gains of BlockDelta shrink, suggesting that part of them comes from weaker motion, as the reviewer suspected.

  • Motion filter. VBench generates several videos per prompt. Within each prompt, we pair BlockDelta and dense videos with nearly equal RAFT motion and drop the unpaired videos, so each pair shares a prompt and a motion level, without regenerating videos (Appendix I.2).

The table below gives the VBench scores of BlockDelta minus dense inference, over all videos / over the RAFT-matched pairs. The matched pairs come from a subset of the prompts, for example 82%, 54% and 77% of them for dynamic degree on the three models. * marks a 95% interval that excludes 0, and imaging quality is on a 0–100 scale.

DimensionWan2.2 T2VWan2.2 I2VHunyuanVideo-1.5 I2V
Subject consistency+0.0194* / +0.0109+0.0115* / +0.0048*+0.0059* / −0.0017
Background consistency+0.0163* / +0.0112*+0.0050* / +0.0007+0.0016* / −0.0027*
Motion smoothness+0.0103* / +0.0053*+0.0034* / +0.0017*−0.0008* / −0.0016*
Temporal flickering+0.0143* / +0.0070*+0.0064* / +0.0018*+0.0055* / +0.0010*
Imaging quality+0.09 / −0.03−0.04 / +0.03−0.51* / −0.73*
Aesthetic quality+0.0051* / +0.0037+0.0006 / −0.0013−0.0038* / −0.0058*
I2V subject–+0.0031* / +0.0002−0.0006 / −0.0028*
I2V background–+0.0008 / −0.0008+0.0007* / −0.0004*
Camera motion–−0.1125* / −0.0484*+0.0107* / +0.0053
  • Against FIS-DiT at matched motion. BlockDelta (FIS-DiT settings) scores higher than FIS-DiT on all ten dimensions of both I2V models (Table 14 in the revised paper). On Wan2.2 T2V, 12 of 15 intervals include 0, and the other three favor FIS-DiT by at most 0.0077; the advantage of residual reconstruction on this model is in FID and FVD (9.83 vs. 22.22 and 174.6 vs. 470.1; Table 1).
  • Quality at matched motion. Besides the results on Wan2.2 given above, imaging quality on HunyuanVideo-1.5 I2V is 0.73 points lower, and aesthetic quality and I2V subject are slightly lower.
  • Does weak motion favor VBench? In part. At matched motion, the consistency gains of BlockDelta shrink on all three models, and on HunyuanVideo-1.5 I2V some turn into small losses.
  • Robustness and coverage. Matching by frame difference instead of RAFT leads to the same conclusions (Table 13 in the revised paper), and applied to dense inference with different seeds, the filter finds no significant difference (all 35 intervals include 0). The kept pairs come mostly from low- and middle-motion prompts, so the results describe the motion range that both methods reach. Regenerating videos until each motion level has enough pairs, as the reviewer suggests, would extend this range, and the revised paper lists such a protocol as future work.

Appendix I.2 of the revised paper adds these results for both motion measures and three tolerances, with the check on different seeds.

Thanks the author for response. My concerns are mostly addressed (though I believe it is worth investigating on solving the motion degradation issue) and I will maintain the score of 3.

Thank you for your thoughtful feedback and continued support for our work!

We fully agree that mitigating motion loss warrants further investigation, and we have highlighted it as the primary open problem in the revised manuscript (Appendix K, Future Work). Specifically, the lost motion resides in the frame-specific component of the block residual, which cannot be recovered through interpolation or simply by minimizing squared error. We therefore plan to treat skipped-frame reconstruction as conditional generation rather than regression.

Reviewer_RNfGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BlockDelta accelerates few-step video diffusion by computing selected anchor frames and reconstructing the skipped frames. Unlike FIS-DiT, which interpolates the full output, BlockDelta retains the known inputs of each skipped frame and interpolates only the block residuals. Specifically, this approach combines enclosing-anchor linear interpolation with dense head/tail blocks, noise-aligned dense late steps, and VAE-aligned anchors.

Full VBench evaluations on Wan2.2 T2V/I2V and HunyuanVideo-1.5 I2V show speedups of 1.74x / 1.66x / 1.55x compared to dense model, with VBench score changes of +0.13 / -0.89 / -1.06 points, respectively. Detailed ablation studies and theoretical analyses demonstrate the effectiveness of each component.

Critical strengths

Clear Motivation: The paper provides a clear and well-grounded motivation by showing that interpolating full outputs discards known input information.

Practical Method: The proposed method is simple, lightweight, and training-free, achieving substantial end-to-end speedups.

Comprehensive Ablations: Well-controlled ablation studies, systematically isolating the impact of reconstruction targets, interpolation schemes, and architectural placement across multiple model architectures.

Critical weaknesses

(1) Model-Dependent Temporal/Fidelity Trade-off The efficiency gains incur noticeable temporal degradation, particularly motion loss on Wan2.2 T2V (pixel-change dynamic proxy: 53.6/80.3/93.3). Furthermore, matched baselines—such as two-step dense Wan2.2 I2V and step-reduction/reuse on HunyuanVideo-1.5—yield better FID/FVD fidelity at similar or higher speeds, tempering the practical edge indicated by VBench alone.

(2) Overstated Sparse-Anchor Neutrality Claiming that anchor perturbations affect both residual and full-output targets identically overextends the algebraic proof. Because the expected squared-error gap retains the cross-term −2E[⟨eX,h⟩]-2\mathbb{E}[\langle e_X, h\rangle], sharing perturbation hh does not inherently guarantee dense-anchor ordering, and the observed 1.024× residual error shift fails to capture this gap.

(3) Poor Presentation The paper's presentation could be significantly improved; the conceptual clarity of the proposed framework is frequently overshadowed by heavy notation and visually congested figures. For example, Figure 3 packs too many dense mathematical variables, indices, and flow labels into a single diagram, obscuring the primary pipeline intuition at a glance. Streamlining such figures and prioritizing conceptual visual abstraction over exhaustive notation would greatly enhance readability.

Questions for authors

As illustrated in Weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer RNfG (1/2)

W1: We thank the reviewer for this comparison. In our view, BlockDelta trades mainly motion, not per-frame quality, for speed, and it complements fewer steps and step reuse rather than only competing with them. Because BlockDelta reduces computation within each step and these methods across steps, their combinations are the fastest configurations we measured on all three models. On Wan2.2 T2V, BlockDelta + 3 steps is also faster than dense inference with 2 steps and closer to dense inference. On the two I2V models, as the reviewer notes, fewer steps or step reuse alone is closer to dense inference than BlockDelta alone.

The table below adds BlockDelta and its combinations to the comparison of Appendix G, on 100 prompts per model. LPIPS to the dense video is divided by that between two dense videos with different seeds, and Dyn. is relative to dense inference (= 100).

ModelMethodE2E speedup ↑FID ↓FVD ↓LPIPS / seed-change LPIPS ↓Dyn.VBench ↑
Wan2.2 T2VDense inference, 2 steps1.83×102.81494.80.9378.778.02
BlockDelta1.75×89.71174.30.9153.680.24
BlockDelta + 3 steps2.05×97.21208.70.9049.681.53
Wan2.2 I2VDense inference, 2 steps1.74×16.7188.90.5692.287.66
BlockDelta1.67×21.2553.31.0980.387.41
BlockDelta + 3 steps1.92×20.5343.01.1082.987.74
HunyuanVideo-1.5 I2VDense inference, 4 steps1.73×11.9104.10.5099.190.03
TeaCache (4)1.72×9.371.80.1798.690.04
BlockDelta1.56×19.9283.50.7293.389.54
BlockDelta + TeaCache (2)1.81×19.8281.30.7093.989.31
  • Combined with matched baselines. BlockDelta + 3 steps is the fastest configuration on both Wan2.2 models (2.05× and 1.92×). On Wan2.2 T2V, it is also closer to dense inference than 2 steps (FVD 1208.7 vs. 1494.8), but not on Wan2.2 I2V. On HunyuanVideo-1.5 I2V, adding TeaCache (2) raises the speedup of BlockDelta from 1.56× to 1.81× with almost no change in FID and FVD.
  • Against matched baselines. On Wan2.2 T2V, BlockDelta alone is closer to dense inference than 2 steps (FVD 1174.3 vs. 1494.8) at a slightly lower speedup. On the two I2V models, fewer steps are faster than BlockDelta alone and closer to dense inference, and so is TeaCache (4) on HunyuanVideo-1.5 I2V, so we suggest using them first there and adding BlockDelta for more speed.
  • Temporal degradation. The cost is mainly in motion rather than in per-frame quality: on pairs of videos with matched motion, imaging and aesthetic quality on Wan2.2 show no drop whose 95% interval excludes 0, and imaging quality on HunyuanVideo-1.5 I2V is 0.73 points lower on a 0–100 scale (Table 13 in the revised paper). The motion loss comes from interpolation, which cannot recover the frame-specific part of the block residual (Section 4.2), and recovering this part is the main direction of our future work.
  • Practical edge beyond VBench. Within frame-sparse methods, residual reconstruction is a clear gain in fidelity: with the settings of FIS-DiT, it lowers FID on full VBench from 22.22, 167.48 and 168.58 to 9.83, 5.37 and 7.13 at the same cost, and FVD on all three models (Table 1). In VBench, as the reviewer suggests, the consistency gains of BlockDelta shrink at matched motion.

Section 5.1 and Appendix K of the revised paper state where fewer steps or step reuse is the better choice, and Appendix G (Table 7) adds these results.

AuthorsOpenReview ↗

Response to Reviewer RNfG (2/2)

W2: We thank the reviewer for checking the proof closely. We agree that the claim overextends the algebraic result, and we have removed the claim "The perturbation therefore favors neither target" from Appendix A.2. Measured with frame-sparse anchors, the ordering of the two targets still holds: the ratio of their errors changes only from 4.08 to 3.99.

  • Cross term. With frame-sparse anchors, the difference between the two local errors gains the cross term −2 E[⟨eX,h⟩]-2\,\mathbb E[\langle e_X,h\rangle], where hh is the anchor perturbation interpolated to the skipped frame. Because the sign of this term is not fixed, sharing hh does not guarantee the dense-anchor ordering, which therefore has to be measured.
  • Dense-anchor ordering. We measured both targets at the same block inputs on Wan2.2 T2V with the BlockDelta anchors, over all sparse blocks and steps of 24 prompts (Appendix A.2). The ratio of squared errors L(Y)/L(Δ)\mathcal L(Y)/\mathcal L(\Delta) is 4.08 with dense anchors and 3.99 with frame-sparse anchors. The output target has the lower error in only 2.4% of the measurements, all on block 21, where Appendix A.2 already reports a reversal with dense anchors. In generated videos, Table 1 runs both targets with the same frame-sparse schedule, and residual reconstruction gives the lower FID and FVD on all three models.
  • The 1.024× value. As the reviewer notes, this value covers only the residual error. Measured directly, the ratio of the two errors drops by about 2%, and the cross term is only −0.56% of L(Y)−L(Δ)\mathcal L(Y)-\mathcal L(\Delta), so most of this drop comes from the 1.024× rise of the residual error.

Appendix A.2 of the revised paper gives the identity with the cross term and this measurement, and Section 4.1 points to it.

W3: We thank the reviewer for this suggestion. We redrew Figure 3 and reduced the notation of Sections 3 and 4.

  • Figure 3. The new figure shows the pipeline conceptually, organized around the three design choices of the method: what to reconstruct, how to interpolate, and which blocks and steps stay dense. It has 13 mathematical labels instead of 43.
  • Notation. Symbols used only once are replaced by words or moved to the appendix, including the explicit formulas of Proposition 1. Sections 3 and 4 now have 15 fewer inline math expressions and one fewer numbered equation. A notation table at the start of the appendix (Table 3) lists the main symbols, and Section 3 points to it.
AuthorsOpenReview ↗

Response to Reviewer RNfG

Thank you again for your time and for your review. As the discussion period is ending soon, we would like to ask whether our responses have addressed your concerns. If any point remains unclear, we would be glad to clarify it before the discussion closes.

We would greatly appreciate any further feedback you may have.

Best regards,

The Authors

Reviewer_ei6POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

BlockDelta is a training-free method that accelerates few-step video diffusion by computing selected latent frames within transformer blocks and approximating the rest. It preserves each skipped frame’s input and interpolates only the block residual, reducing errors compared with full-output interpolation. Experiments show 1.55–1.74× end-to-end speedups with similar aggregate VBench scores, although some motion is lost.

Critical strengths
  • Well-motivated residual reconstruction. Preserving known inputs and interpolating only block residuals improves upon full-output interpolation, supported by controlled comparisons under the same sparsity settings.

  • Insightful analysis of approximation trade-offs. The paper shows that lower local reconstruction error does not necessarily improve final video fidelity, and candidly examines motion loss and the mixed results of a learned corrector.

Critical weaknesses
  • Unresolved quality–efficiency trade-off. Despite similar aggregate VBench scores, substantial motion loss and content changes occur in some settings. The practical acceptability of these degradations is not sufficiently established.

  • Insufficiently characterized reliability boundaries. The fixed, content-independent sparsity rules are not systematically evaluated across motion difficulty or dynamic textures. Although the paper analyzes local approximation errors, it remains unclear when accumulated feature-distribution shifts lead to systematic losses in generation quality.

Questions for authors
  1. Can you analyze whether degradation increases with motion complexity, particularly for dynamic textures? A focused breakdown of existing results would help establish where the fixed sparsity strategy remains reliable.
  2. How should the local-error analysis and corrector results be interpreted regarding accumulated feature-distribution shifts? Please clarify what they establish about preserving generation quality and what remains unresolved. I find the approach interesting and would consider raising my recommendation if these points establish a convincing quality–efficiency trade-off and clearer applicability boundaries.
8 comments
AuthorsOpenReview ↗

Response to Reviewer ei6P (1/3)

W1: We thank the reviewer for this question. To establish the practical acceptability of these changes, we added three analyses. At matched motion, per-frame quality stays close to dense inference: no significant drop in imaging or aesthetic quality on Wan2.2, and about 1% lower imaging quality on HunyuanVideo-1.5 I2V. In every prompt category, the content changes of BlockDelta are smaller than, or about as large as, those of a seed change. The main cost is the motion loss, which depends on the model. For the quality–efficiency trade-off, BlockDelta alone gives a 1.56× to 1.75× speedup, and at a similar speedup, it is closer to dense inference than fewer steps on Wan2.2 T2V. On the two I2V models, fewer steps or step reuse is closer, and BlockDelta adds speed on top of them: the combinations are the fastest configurations we measured on all three models.

The table below compares BlockDelta with fewer steps and step reuse on 100 prompts per model (Table 7 in the revised paper). LPIPS to the dense video is divided by that between two dense videos with different seeds, and Dyn. is relative to dense inference (= 100).

ModelMethodE2E speedup ↑FID ↓FVD ↓LPIPS / seed-change LPIPS ↓Dyn.VBench ↑
Wan2.2 T2VDense inference, 2 steps1.83×102.81494.80.9378.778.02
BlockDelta1.75×89.71174.30.9153.680.24
BlockDelta + 3 steps2.05×97.21208.70.9049.681.53
Wan2.2 I2VDense inference, 2 steps1.74×16.7188.90.5692.287.66
BlockDelta1.67×21.2553.31.0980.387.41
BlockDelta + 3 steps1.92×20.5343.01.1082.987.74
HunyuanVideo-1.5 I2VDense inference, 4 steps1.73×11.9104.10.5099.190.03
TeaCache (4)1.72×9.371.80.1798.690.04
BlockDelta1.56×19.9283.50.7293.389.54
BlockDelta + TeaCache (2)1.81×19.8281.30.7093.989.31
  • Quality at matched motion. On pairs of videos of the same prompt with matched RAFT motion, imaging and aesthetic quality on Wan2.2 show no drop whose 95% interval excludes 0, and neither do the eight semantic and style dimensions of Wan2.2 T2V. On HunyuanVideo-1.5 I2V, imaging quality is 0.73 points lower on a 0–100 scale (Table 13 in the revised paper).
  • Content changes. On average in every prompt category of full VBench, BlockDelta videos deviate from the same-seed dense video less than a seed change does on Wan2.2 T2V and HunyuanVideo-1.5 I2V, and by at most 0.024 LPIPS more on Wan2.2 I2V (Figure 8 in the revised paper). On the 100 prompts, the LPIPS of BlockDelta relative to a seed change is 0.72 to 1.09 (table).
  • Quality–efficiency trade-off. On Wan2.2 T2V, BlockDelta is closer to dense inference than dense inference with 2 steps (FVD 1174.3 vs. 1494.8) and scores higher on VBench, at a slightly lower speedup. On the two I2V models, fewer steps or step reuse alone is closer to dense inference at a similar speedup, because it costs less there: relative to a seed change, the LPIPS of halving the steps is about 0.5 on the I2V models, against 0.93 on Wan2.2 T2V. Because BlockDelta reduces computation within each step, it can be added to these methods, and the combinations are the fastest configurations we tested, with FVD at most 3% above that of BlockDelta alone.
  • Motion loss. The motion loss, measured by Dyn. in the table, is largest on Wan2.2 T2V and smallest on HunyuanVideo-1.5 I2V. It comes from interpolation, which cannot recover the frame-specific part of the block residual (Section 4.2). Where more motion is needed, a smaller stride on high-motion prompts recovers part of it at little cost (our answer to W2-1 / Q1).

Section 5.1 and Appendix K of the revised paper state where fewer steps or step reuse is the better choice, and Appendix I adds the analyses of content changes and of VBench quality at matched motion.

AuthorsOpenReview ↗

Response to Reviewer ei6P (2/3)

W2-1 / Q1: We thank the reviewer for this suggestion. We broke down the full-VBench results by prompt category and by the motion of the dense video, and compared both motion and imaging quality. The share of motion lost does not generally increase with motion: only on Wan2.2 I2V do high-motion prompts lose a clearly larger share. Dynamic textures are not the category with the largest motion loss on any model, and on Wan2.2, their imaging quality shows no significant drop; on HunyuanVideo-1.5 I2V, it is about 1 point lower. A per-prompt stride gains little speed over the fixed rule, but it can recover part of the motion on high-motion prompts. We summarize the resulting applicability boundaries at the end.

The table below breaks down the full-VBench results by prompt category. Dyn. is relative to the same-seed dense video (= 100), and imaging quality is the score of BlockDelta, with its difference from that of reseed in parentheses.

ModelMeasureStatic or subtle actionSubject motionCamera motionDynamic textureMultiple subjects or actionsUnspecifiedDense inference with a different seed (reseed), range over categories
Wan2.2 T2VDyn. (dense = 100)46.852.761.959.554.347.098.1–103.2
Imaging quality (0–100)68.42 (−0.11)68.86 (+0.09)— (too few prompts)68.63 (+0.95)67.99 (−0.65)70.88 (+0.80)67.68–70.09
Wan2.2 I2VDyn. (dense = 100)80.881.480.483.781.879.299.6–100.5
Imaging quality (0–100)69.91 (+0.08)70.40 (−0.09)— (too few prompts)68.38 (−0.25)71.31 (−0.02)73.60 (+0.03)68.63–73.57
HunyuanVideo-1.5 I2VDyn. (dense = 100)83.783.3100.795.598.399.699.3–100.5
Imaging quality (0–100)69.95 (−0.41)68.89 (−0.75)— (too few prompts)67.15 (−0.95)70.17 (−0.74)73.14 (−0.34)68.10–73.48
  • Dynamic textures. On every model, dynamic textures keep more motion than the category with the lowest Dyn., and on Wan2.2, no category shows a drop in imaging quality whose interval excludes 0. On HunyuanVideo-1.5 I2V, imaging quality on dynamic textures is 0.95 points below that of reseed, 0.54 points more than on static prompts (95% interval 0.08 to 0.99; Appendix I.1).
  • Motion complexity. We split the prompts into three groups of equal size by the motion of the dense video. Only on Wan2.2 I2V does the high-motion group keep clearly less motion than the low-motion group (Dyn. −15.0, 95% interval −18.4 to −11.3; Figure 9 in the revised paper); on the other two models, the sign of the difference changes with the motion measure. On both Wan2.2 models, the high-motion group also deviates more from dense inference in LPIPS.
  • Fixed, content-independent rule. On the 100-prompt subset, we also chose the stride for each prompt from its category or from the motion of its dense video (Table 12 in the revised paper). Even an oracle that picks, for each prompt, the largest stride that does not increase LPIPS to dense is only 5.3%, 0.9% and 0.7% faster than stride 3, so the fixed rule gives up little speed. Where more motion is wanted, strides 2, 3 and 4 on the high-, middle- and low-motion groups recover 32% to 40% of the motion gap on the high-motion group and are only 2.1% to 3.2% slower than stride 3. These groups use the motion of the dense video, which a deployable rule has to estimate during generation.

Applicability boundaries. Taken together, the fixed rule deviates from the dense video less than a seed change in every prompt category on Wan2.2 T2V and HunyuanVideo-1.5 I2V, and keeps imaging quality in every category on Wan2.2. The motion loss is larger on high-motion prompts of Wan2.2 I2V, and imaging quality drops more on dynamic textures of HunyuanVideo-1.5 I2V. Across steps, the last denoising steps must stay dense: making one more of them sparse raises FVD to 1.35 to 2.24 times its value (Table 11; our answer to W2-2 / Q2).

Appendices H and I.1 of the revised paper add these analyses, and choosing the stride from an early motion estimate is listed as future work.

AuthorsOpenReview ↗

Response to Reviewer ei6P (3/3)

W2-2 / Q2: We thank the reviewer for this question. The local-error analysis selects the reconstruction target and the interpolation weights, and generated videos confirm both choices. It uses features of dense inference, which carry no accumulated shift, so it does not bound the shift (Appendix A.2), and the corrector results show that a lower local error alone is not enough. On Wan2.2 T2V, the shift appears mainly as lower motion rather than lower per-frame sharpness, and it leads to a much larger quality loss when the last steps are also sparse, which BlockDelta avoids by keeping them dense.

  • What the local-error analysis establishes. Residual reconstruction has the lower local error on 39 of 40, 54 of 54 and 47 of 48 blocks of Wan2.2, HunyuanVideo-1.5 and LTX-2.3 (Section 4.1), and, as the reviewer notes, it gives lower FID and FVD in every controlled comparison (Tables 1, 2 and 8; Appendix E.2). For the weights, Proposition 1 shows that every reduction of the local error removes the same amount of frame-specific energy, so neither end of the weight concentration is best. Figure 5 confirms this on Wan2.2 T2V and HunyuanVideo-1.5 I2V: the FVD of the mean of all anchors, which has the lowest local error, is 1.32 and 1.53 times that of linear interpolation, and the FVD of copying the nearest anchor, which retains the most energy, is 4.46 and 7.37 times.
  • What the corrector results show. The corrector is trained on states of the sparse trajectory, which carry the shift, with the squared error to the dense velocity, whose minimizer is the mean of that velocity given the input. On HunyuanVideo-1.5, it lowers the held-out error from 0.153 to 0.119 but worsens FID and FVD. On Wan2.2 T2V, it lowers the per-frame sharpness from 1.00 to 0.71 and raises FVD from 1174 to 1422 (Appendix F). A lower held-out error therefore does not necessarily give a video closer to dense inference, and reconstruction choices have to be judged on generated videos.
  • When the shift leads to systematic losses. On Wan2.2 T2V, BlockDelta keeps the per-frame sharpness of dense inference (1.00) but has a Dyn. of 53.6, so the shift appears as lower motion rather than blur. This is consistent with the analysis: under Assumption 1, every sparse block retains less frame-specific variation than dense inference, so the loss is one-sided and does not average out. The shift becomes a clear quality loss when the last steps are also sparse: making one more step sparse raises FVD from 1174, 553 and 284 to 1580, 1050 and 634 (Table 11), so BlockDelta keeps the last steps dense.

What remains unresolved. The main open problem is recovering the frame-specific residual of skipped frames. By Lemma 1, in any class of estimators closed under scaling, the least-squares estimate retains less frame-specific energy than dense inference, by exactly its error. Because the anchors do not determine this part, the error cannot reach 0, and the corrector, which Lemma 1 does not cover, behaves the same way on Wan2.2 T2V. This suggests treating skipped-frame reconstruction as conditional generation rather than regression, in line with the perception–distortion trade-off (Blau & Michaeli, 2018). A solution would reduce the main cost of frame-sparse inference.

The revised paper adds these explanations to Section 4.1, Appendix F and Appendix K and states the open problem as future work.

Blau, Yochai, and Tomer Michaeli. "The perception-distortion tradeoff." 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2018.

Comments on the paper

2 comments
Area_Chair_hRunOpenReview ↗

Discussion Reminder

Dear Reviewers,

I hope this message finds you well. The rebuttal is now available in the discussion forum. I would like to kindly remind you to check the rebuttal and leave comments for the authors, which would be very helpful.

Thank you very much for your time and effort. Please let me know if you have any questions.

Best

AuthorsOpenReview ↗

Summary of Discussion and Acknowledgement for Reviewers

Dear Program Chairs, Senior Area Chairs, Area Chairs and Reviewers,

We thank the reviewers for their valuable comments and constructive feedback. We appreciate the reviewers' recognition of the following strengths of our work:

  1. Well-motivated residual reconstruction (zmp1, RNfG, ei6P). Keeping the known input of each skipped frame and reconstructing only the block residual was recognized as a clear and well-motivated design.
  2. Controlled comparisons (zmp1, ei6P). Table 1 isolates residual reconstruction under the same FIS-DiT settings and PFLOPs. Under these settings, it lowers FID from 22.22, 167.48 and 168.58 to 9.83, 5.37 and 7.13 on Wan2.2 T2V, Wan2.2 I2V and HunyuanVideo-1.5 I2V.
  3. Thorough ablations and analysis (zmp1, RNfG, ei6P). The ablations isolate the reconstruction target, the interpolation scheme and the block placement, and the analysis of approximation trade-offs explains why a lower local error does not always give a better video.
  4. Practical method (RNfG). BlockDelta is simple, lightweight and training-free, and gives substantial end-to-end speedups.

During the rebuttal phase, we addressed the concerns with additional experiments, analyses and clarifications. The major concerns, our responses and the paper updates are listed below; all revisions are marked in blue.

  1. Evaluation under weak motion (zmp1). We evaluated all VBench dimensions at matched motion (Appendix I.2). zmp1 reported that their concerns were mostly addressed.
  2. Sparse-anchor claim and presentation (RNfG). We removed the overstated claim from Appendix A.2 and showed with frame-sparse anchors that the ordering of the two targets still holds. We also redrew the method figure (now Figure 4) and reduced the notation of Sections 3 and 4. RNfG confirmed that the perturbation concern was resolved and that the figure was clearer.
  3. Reliability boundaries and error accumulation (ei6P). We broke down the results by prompt category and by motion (Appendix I.1), tested content-dependent strides (Appendix H), and clarified what the local-error analysis establishes and what remains open (Appendix K). ei6P confirmed that the breakdowns addressed their concern and appreciated the clarification on error accumulation across denoising steps.
  4. Comparison with step reduction and step reuse (RNfG, ei6P). Appendix G compares BlockDelta with these methods and with their combinations on three models (Table 7).

Remaining concern (RNfG, ei6P). The remaining concern is the motion loss of BlockDelta and the advantage of simpler alternatives in the two I2V settings. Video generation needs both meaningful motion and clean frames, and step reduction and BlockDelta lose different ones: with 2 steps, the frames show ghosting and a dotted texture (Figure 1), and imaging quality drops by 0.84 points on Wan2.2 I2V, whereas BlockDelta keeps the frames clean but loses part of the motion (Appendix L). Because BlockDelta reduces computation within each step, it complements these methods, and their combinations are the fastest configurations we tested (Table 7). We regard recovering the lost motion as the main open problem of this work, and leave it as future work (Appendix K).

We are grateful to all three reviewers for their active engagement during the discussion. We thank all the reviewers, ACs, SACs and PCs for their time and dedication to the review process.

Best regards,

Authors of Paper Submission 2905

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.