acceptodds
Under review as a conference paper at ICLR 2027

Coded Computation: Perceptual-Semantic Acceleration for Video Diffusion

Abstract

Efficient video diffusion is commonly approached by making generative computation cheaper through fewer denoising steps, quantized operators, feature reuse, or selective computation. We explore a different perspective—can computation itself be coded and subsequently reconstructed? Inspired by coded acquisition in computational imaging, we introduce coded computation, a new formulation that jointly designs which generative computations are explicitly performed and how the omitted information is recovered. We instantiate this principle with a perceptual-semantic computational encoder and a lightweight code-aware generative decoder. The encoder combines bottom-up visual cues with top-down semantic relevance from the text prompt to construct a time-varying compute code, allocating richer computation to perceptually and semantically important content while encoding less critical content more efficiently. The decoder then leverages the compute code and strong spatiotemporal and diffusion priors to reconstruct the omitted generative information. Extensive experiments demonstrate that coded computation substantially accelerates video diffusion inference while preserving visual quality, temporal consistency, and prompt fidelity.

Primary area:
generative models
Keywords:
Coded Computation, Video Diffusion, Acceleration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_X9ETOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper accelerates video diffusion by combining perceptual-semantic token scheduling with learned reconstruction and aligned K/V context. It reports up to 2.64x acceleration on Wan and 2.48x on LTX while largely preserving native output quality.

Critical strengths
  1. The work is substantial, with experiments across two backbones, baseline comparisons, component ablations, and runtime analysis. Although some targeted ablations are missing, the overall technical development and experimental evaluation are thorough.

  2. The method demonstrates practical acceleration on both backbones, reaching up to 2.64x on Wan and 2.48x on LTX with limited changes in aggregate VBench scores. Measured wall-clock latency and multiple operating points provide concrete evidence of its speed-fidelity trade-off.

Critical weaknesses
  1. Overstated conceptual framing. The method is essentially feedback-driven token scheduling coupled with learned reconstruction and noise-aligned K/V context: new DiT observations guide subsequent refresh decisions (Eqs. (7), (A7), and (A8)). Equation (2) defines the "compute code" as an execution-mask sequence. While technically substantial, the work does not clearly establish why this constitutes a new computational paradigm beyond relabeling the scheduling and reconstruction loop. The authors should clarify the additional design principle provided by the coding perspective or moderate the paradigm-level claims.

  2. Reliability of early importance estimates. In the evaluated configuration, visual and semantic cues are extracted once after step 10 and used throughout the remaining 40 steps. Final-output results support the overall approach, but there is no targeted analysis of whether these initial importance estimates remain reliable as finer details emerge. Dynamic refresh decisions alone do not isolate the contribution of the fixed cues. An analysis across denoising stages or a comparison with periodic cue extraction would strengthen this design choice. (Section 3.4; Algorithm A2; Appendices A and G.)

  3. Missing warm-up length ablation. The 10-step dense warm-up affects the reliability of the entrance estimates and the remaining opportunity for acceleration, but the paper does not compare alternative switching points. The sensitivity study varies tau and beta while keeping warm-up fixed. An ablation at a fixed total sampling budget should report final quality and end-to-end latency, clarifying whether the auxiliary networks are adapted to each entrance state. This would establish how sensitive the reported quality-latency trade-off is to the switching point. (Section 4.1; Appendices B.4-B.5, C.1, and I.)

  4. Boundary consistency. Full-context attention and neighborhood safeguards may help maintain consistency, and Table 3 supports the decoder's overall effectiveness. However, no targeted evaluation checks for spatial or temporal artifacts at interfaces between refreshed, unrefreshed, and directly reconstructed regions. A brief analysis or close-up video comparison around these interfaces would be useful. (Table 3; Appendix B.2-B.5.)

  5. Missed candidates. The method includes offline risk calibration and neighborhood safeguards, but the candidate set remains fixed after warm-up. The router cannot subsequently admit tokens excluded due to underestimated risk, and their states follow the direct reconstruction path. It would be useful to discuss the potential impact of these errors and whether candidate membership could be reconsidered. (Appendix B.2, B.4, and B.5.)

  6. Attribution within the complete router. Table 2 supports visual and semantic guidance under coverage-based selection, but several factors change together when comparing those strategies with the complete router. Removing visual guidance, semantic guidance, reconstruction risk, and innovation feedback individually under matched execution budgets would clarify their respective contributions. (Table 2; Equation (A7); Appendix G.)

  7. Training and resource details. The paper describes teacher-trajectory generation, but does not clearly state the training prompt sources or the total numbers of prompts and trajectories. The provenance of Wan's pretrained reconstruction checkpoint also needs clarification. Reporting training compute, including teacher-data generation, auxiliary parameter counts, and additional peak GPU memory would complement the existing inference-timing results and improve reproducibility. (Appendices B-C and J.)

Questions for authors

See weakness.

5 comments
AuthorsOpenReview ↗

R3 · Reviewer X9ET (Part1)

R3-W1: Design principle and conceptual framing of coded computation

Thank you for the comment. Coded computation uses the reconstructability of omitted generative information to guide selective execution, incorporating reconstruction reliability directly into computation allocation. We have replaced “paradigm” with “framework” in the introduction's first contribution statement and explicitly emphasized a reconstruction-aware formulation: jointly deciding which generative computations to execute and how to recover omitted information throughout the diffusion trajectory.

  1. Our design incorporates learned reconstruction reliability into computation allocation. TeaCache estimates temporal output changes to guide caching; ProfilingDiT uses foreground/background importance for adaptive feature reuse; and SenCache makes caching decisions based on output sensitivity. We jointly consider perceptual-semantic importance, reconstruction risk, and observation history. Computation is therefore allocated according to both content importance or change and the value of explicit evaluation relative to what the decoder can recover.

  2. Computation and reconstruction are coupled throughout generation. The compute code in Eq. (2) specifies the observation pattern along the diffusion trajectory. Learned dense prediction recovers omitted information and constructs K/V references aligned with the diffusion state for subsequent sparse DiT computation. Fresh observations then update routing decisions. Reconstruction thus participates in the ongoing generation process rather than operating as independent post-processing.

  3. Experiments with matched budgets and execution masks support this design. Fig. 2(c) shows that different computation schedules produce different reconstruction errors at the same budget. Table 2 verifies the benefits of adaptive computation allocation. Table 3 shows that learned reconstruction and diffusion-state alignment substantially improve fidelity even with identical execution masks. Appendices E–F provide the full experimental settings and analysis.

R3-W2: Reliability of early importance estimates during later denoising

Thank you for raising the question of early importance reliability. In Appendix E.2, we added an analysis of stability across denoising stages and controlled refresh experiments using 100 VBench prompts with the 41-frame Wan configuration. The new Table 10 reports the refresh results and paired confidence intervals.

  1. The step-10 cues remain highly consistent with later estimates. Our newly added stage-wise statistics in Appendix E.2 show that fused step-10 importance has mean Spearman correlations of 0.980, 0.974, and 0.970 with estimates at steps 20, 30, and 40, respectively, and an 88.05% recall of the top-20% importance tokens at step 40. These results support the continued usefulness of the initial cues during later denoising.

  2. Periodic extraction does not improve average fidelity or latency in the compared configurations. The configurations in our new refresh experiment in Table 10 share the encoder, decoder, reconstruction, risk map, candidate set, and dynamic feedback mechanism. One-time extraction achieves LPIPS 0.04794 at 29.19 s. One or two additional refreshes yield LPIPS 0.04810/0.04822 at 30.55/31.98 s, respectively. One-time extraction achieves the best average fidelity and lowest latency among these configurations; the additional refreshes increase end-to-end cost by 4.65%/9.55%.

  3. Control experiments separate cue updates from additional probing. In Appendix E.2, we added a control that performs the probes while retaining the original importance map, producing outputs identical to those obtained with fixed cues. We also added offline Oracle refreshes to Table 10 using later Native cues at matched per-step sparse budgets, further supporting the effectiveness of step-10 guidance. Early visual-semantic importance provides a stable content prior, while subsequent observations and dynamic feedback adjust actual token updates.

AuthorsOpenReview ↗

R3 · Reviewer X9ET (Part2)

R3-W3: Dense warm-up length and adaptation to the entrance state

Thank you for suggesting a comparison of warm-up lengths. In Table 14 of Appendix F.1, we added a controlled experiment with K = 5, 10, 20, and 30 on 100 VBench prompts using the 41-frame Wan configuration, with the total sampling budget fixed at 50 steps.

  1. We train a separate decoder for each entrance state. All four decoders, including K = 10, use identical initialization, training trajectories, sample order, and 900 optimizer updates. Each decoder takes the clean-state prediction at its corresponding entrance as input and learns to recover the final Native latent. The backbone, VAE, visual and subject auxiliary branches, and original step-10 calibration are shared and frozen. This design compares entrance adaptation under the same training budget; Appendix F.1 specifies the training and freezing settings for our new experiment.

  2. Increasing warm-up from 5 to 10 steps substantially improves the basis for reconstruction. The results of our new warm-up experiment in Table 14 show a 55.3% reduction in LPIPS and a 5.30 dB increase in PSNR, with 5.09 s of additional latency. We also added an analysis of direct latent-prediction error in Appendix F.1: the decoder's prediction MSE decreases by 54.3%, supporting the benefit of a more informative entrance state for reconstruction.

  3. Longer warm-up further improves fidelity while increasing latency. At K = 10, LPIPS is 0.0527 and latency is 29.18 s, retaining a 2.55× speedup over Native. At K = 20/30, LPIPS decreases to 0.0164/0.0086 and latency increases to 40.82/52.99 s. Our newly added Table 14 reports final fidelity, end-to-end latency, and actual DiT execution for each configuration.

  4. Ten warm-up steps provide a practical default under the evaluated conditions. Compared with an earlier switch, this setting obtains substantial reconstruction gains while preserving 40 steps for efficient coded computation. The results characterize the fidelity–latency trade-off associated with the switching point.

R3-W4: Boundary consistency between computation regions

Thank you for suggesting a targeted evaluation of boundary consistency. In Appendix F.2, we added an analysis of three types of region interfaces across 100 Wan videos. Our new Table 15 and Fig. 12 report the boundary statistics, and we added aligned visualizations in Fig. 13 covering five spatial-error quantiles.

  1. Shared reconstruction and context maintain coupling between regions. Dense reconstruction, noise-aligned context references, and neighborhood-aware risk estimates keep selectively evaluated tokens connected to the complete video representation and incorporate reconstruction needs near boundaries into computation allocation.

  2. Matched controls support spatial compatibility. We match Native gradients and entrance prediction risk, using candidate-region interiors as controls for refreshed/unrefreshed interfaces. Our new spatial analysis in Table 15 and Fig. 12(a) reports mean spatial excess errors of −0.657 × 10⁻³ for refreshed/directly reconstructed interfaces, −0.205 × 10⁻³ for unrefreshed/directly reconstructed interfaces, and −0.064 × 10⁻³ for refreshed/unrefreshed interfaces. The last has a confidence interval of [−0.138, +0.010] × 10⁻³ and a mean close to zero. These statistics agree with the preserved local structures and color transitions in our added visualizations in Fig. 13, supporting spatial compatibility between reconstructed content and selectively computed content.

  3. The temporal statistics characterize errors at the interfaces. Our new temporal analysis in Table 15 and Fig. 12(b) also reports positive temporal excess errors of 0.276 × 10⁻³ and 0.824 × 10⁻³ at refreshed/unrefreshed and unrefreshed/directly reconstructed interfaces, respectively, indicating measurable temporal excess error near unrefreshed regions. Together, the spatial and temporal statistics characterize reconstruction behavior at the three types of interfaces.

AuthorsOpenReview ↗

R3 · Reviewer X9ET (Part3)

R3-W5: Whether fixed candidates miss tokens that later require computation

Thank you for the comment. In Appendix E.3, we added candidate-coverage analysis and matched-budget experiments on membership updates, examining the roles of entrance selection and subsequent dynamic routing.

  1. The entrance candidate set covers the main reconstruction demands. We added coverage statistics over 100 videos in Table 11: the candidate set covers 99.96% of the top-10% endpoint-error tokens and 99.70% of the top-10% final-output-error tokens, while retaining approximately 97.6% coverage of the top-20% importance tokens at later steps. This provides a candidate pool with broad coverage, within which actual computation can adapt throughout denoising.

  2. We compare membership updates at the same actual execution budget. On 20 paired videos with nonempty excluded regions, all policies retain the same candidate capacity, per-step refresh quotas, and actual DiT token execution. Our new membership-update experiment in Table 12 shows that the fixed candidate set achieves global LPIPS 0.045902, reducing LPIPS by 6.06%, 4.75%, and 14.00% relative to score-based, conservative, and random membership updates, respectively. Our added paired statistics in Table 13 show that all three improvements remain significant after Holm correction. The conservative policy also protects scheduled refreshes and applies an admission margin to test more selective membership changes.

  3. Regional results explain the global trade-off. The regional metrics from our new experiment in Table 12 show that membership updates improve reconstruction in the originally excluded regions but increase error in the original candidate regions. Entrance selection and dynamic routing therefore serve complementary roles: the former maintains coverage of the main reconstruction demands, while the latter adjusts the limited refresh budget within this stable set to preserve overall video fidelity.

R3-W6: Individual contributions within the complete router

Thank you for the suggestion. In Appendix E.1, we added matched-budget ablations on 100 Wan prompts, individually removing visual guidance, semantic guidance, reconstruction risk, and innovation feedback. Our new Tables 8–9 report the final-output and shared-state update results, respectively.

  1. Each ablation changes only the corresponding signal's contribution to routing. All variants in our new ablations in Table 8 share the reconstruction network, initial candidate count, per-step refresh counts, and actual DiT token execution, directly comparing the signal's effect on routing decisions at the same computation budget.

  2. Visual and semantic guidance improve update effectiveness in detail and subject regions, respectively. In our newly added shared-state experiment in Table 9, actual updates start from identical states, candidate locations, and observation histories at matched budgets. Visual guidance increases detail-error removal from 23.21% to 29.11%; semantic guidance increases subject-error removal from 22.61% to 27.93%, with subject evaluation using the 60 prompts with independent valid masks. Our new ablations in Table 8 further report changes in the final videos: removing visual guidance increases detail MAE from 0.03678 to 0.03728; removing semantic guidance increases subject LPIPS from 0.0486 to 0.0495 while reducing global LPIPS from 0.0479 to 0.0454. This trade-off reflects the allocation of limited computation toward subject regions by semantic guidance.

  3. Risk and innovation feedback affect different error metrics. The other two new ablations in Table 8 show that removing risk guidance reduces PSNR from 29.910 dB to 29.690 dB and increases detail MAE from 0.03678 to 0.03788. Removing innovation feedback increases subject LPIPS from 0.0486 to 0.0497 and temporal-difference MAE from 0.01204 to 0.01211.

Reviewer_2AsZOpenReview ↗
Rating
3
Confidence
2(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes "coded computation," a computational-imaging-inspired paradigm for accelerating video diffusion. Motivated by coded acquisition (measure a structured subset, reconstruct the rest), the authors reframe efficient generative inference from an execution-centric problem (fewer steps, caching, sparsification) into a joint computation-encoding and generative-reconstruction problem. After a short dense warm-up establishes a coarse spatiotemporal scaffold, a perceptual-semantic encoder combines bottom-up visual cues (motion, texture, structure) with top-down semantic relevance from the text prompt to construct a time-varying "compute code" — a per-step mask deciding which DiT tokens receive explicit evaluation. A code-aware generative decoder then learns a dense clean-state prediction R together with a per-token reconstruction risk, and reconstructs the omitted tokens via noise-level-aligned key/value references (diffusion-state alignment) and a compute-code router.

Critical strengths
  1. The conceptual framework is novel. The analogy from coded-acquisition to coded-computation is insightful, clearly distinguishing this paper from the mainstream execution-centric acceleration literature. Positioning inference as "jointly deciding what to compute + how to reconstruct what was not computed" gives the paper a self-consistent, non-incremental motivation.

  2. The experiments are solid and well-controlled. The evaluation covers two backbones, four strong caching/selective execution baselines (TeaCache, ProfilingDiT, SenCache, NaviCache), the full set of VBench dimensions, matched-budget encoder ablation, factorial decoder ablation, routing sensitivity, and token-update distribution.

3.The fidelity advantage is compelling. Ours-Low achieves an LPIPS of 0.006 on Wan (the strongest baseline is 0.025–0.028) and 0.003 on LTX, with a PSNR improvement of approximately 6.8 dB over TeaCache under similar latency. This demonstrates that learning-based reconstruction indeed preserves the native output more faithfully than reuse-based methods.

Critical weaknesses
  1. No theoretical support; the method is entirely heuristic. Despite the appealing "coding" framing, there is no theorem, proposition, or formal guarantee anywhere in the paper. Every central quantity — the importance map Ix = 1+3S+0.5h (Eq. A2), the tolerances and risk (Eqs. A4–A5), the routing demand Dt (Eq. A7), the alignment coefficient αt (Eq. A6) — is a hand-designed formula with dense, unexplained magic coefficients (0.003, 0.75, 0.25, 0.10, 0.15, 0.20, etc.), and none of these coefficients (except τ and β) is subjected to any sensitivity analysis.

2.The speedup claim is weaker than the title suggests. On Wan, Ours-High’s 2.64× is only about 3.5% higher than SenCache’s 2.55×, while Ours-Mid (2.38×) is actually slower than SenCache. The real selling point is “higher fidelity at comparable speed,” not “faster”—the abstract’s "accelerates ... by 1.42–2.64×" overstates the method's differentiation from existing baselines.

Questions for authors
  1. Justify the coding formulation's substance.Beyond the analogy, what is the formal benefit of "coding" over plain selective execution + reconstruction? Can you state any guarantee (e.g., a reconstruction-error bound conditioned on the compute code) that distinguishes coded computation from token-wise sparsification with learned prediction? If no such result is intended, please temper the paradigm-level claims accordingly.

  2. Clarify the headline claim. Given that Ours-High is only 3.5% faster than SenCache on Wan and Ours-Mid is slower, would you agree the primary contribution is fidelity-at-fixed-latency rather than raw speedup? Consider reframing the abstract to lead with the fidelity advantage.

  3. Discuss VBench per-dimension regressions. Color (−3.43) and Spatial relationship (−3.41) drop notably at High on Wan, and the semantic aggregate falls ~1 point. Are these within VBench noise, or do they reflect systematic degradation? Please add error bars / multi-seed variance to Table 4.

4 comments
AuthorsOpenReview ↗

R2 · Reviewer 2AsZ (Part1)

R2-W1/Q1: Substance of the coding formulation, framework positioning, and coefficient sensitivity

Thank you for raising this question. We have replaced “paradigm” with “framework” in the introduction's contribution statement, explained the formulation through reconstruction-aware computation allocation, and added a sensitivity study of the fixed coefficients.

  1. The formulation expresses a trade-off between computation cost and reconstruction distortion. Coded computation can be written as min⁡C,DE[d(X,D(YC,C,P))]+λCost⁡(C)\min_{\mathcal C,\mathcal D}\mathbb E[d(X,\mathcal D(Y_{\mathcal C},\mathcal C,P))]+\lambda\operatorname{Cost}(\mathcal C).

    Here, the compute code C determines which DiT computations are explicitly acquired, and the decoder D uses these observations and generative priors to recover omitted information. Beyond selecting computations by importance alone, this formulation considers both the perceptual cost of omission and the reliability of recovering the omitted information. We approximate these factors through perceptual-semantic importance, reconstruction risk, and observation history.

  2. Reconstruction participates in subsequent generation. The learned prediction supplies both estimates of omitted states and dense K/V references aligned with the diffusion state for subsequent sparse DiT computation. Observation feedback further adjusts computation allocation during denoising. Fig. 2(c) and Tables 2–3 support the roles of computation scheduling, adaptive allocation, and code-aware reconstruction, respectively. Appendices E–F provide the corresponding experimental settings and analyses.

  3. We added a sensitivity study of the fixed coefficients. Fig. 14 in Appendix G.1 reports our new controlled study of four coefficient groups across 17 configurations, using the same prompt and seed as Fig. 5 and listing all tested coefficients and results. Across scale factors of 0.8 and 1.2, the maximum absolute LPIPS changes relative to the default are 0.00073 and 0.00066 for semantic/visual weighting and local tolerance, respectively. The corresponding changes for the risk-related and history-normalization groups are 0.00330 and 0.00351, indicating a more pronounced effect.

  4. We position the contribution as a framework. The current routing strategy is a heuristic implementation of reconstruction-aware computation allocation, evaluated through matched-budget ablations and coefficient sensitivity experiments. A theoretical reconstruction-error bound has not been established. We have replaced “paradigm” with “framework” in the introduction's contribution statement to describe the methodological contribution accurately.

R2-W2/Q2: Emphasizing fidelity gains at comparable latency

Thank you for the suggestion. We have removed the speedup range from the abstract and now emphasize better preservation of the Native output's visual quality, temporal consistency, and prompt fidelity at comparable inference speeds. We have also revised the introduction, main results, and conclusion to clarify the principal benefits and fidelity–latency trade-offs.

  1. We distinguish the gains on the two backbones. The quantitative comparison in Table 1 and our newly added quality–speed curves in Fig. 9 of Appendix D.3 show the benefits on each backbone. On Wan, Mid adds 2.74 s of latency relative to SenCache while improving PSNR by 0.89 dB. On LTX, Mid reduces latency and LPIPS by approximately 4.9% and 50%, respectively, improving both speed and fidelity. The qualitative comparisons in Fig. 4 and Figs. 10–11 of Appendix D.5 also show more accurate recovery of local details.

  2. We removed the speedup range from the abstract. We deleted the “1.42–2.64×” speedup statement and now summarize the results as better preservation of visual quality, temporal consistency, and prompt fidelity than existing acceleration methods at comparable inference speeds. The introduction explains the gains through direct baseline comparisons and notes that the same trained weights support different inference budgets. The main results distinguish fidelity gains on Wan from simultaneous improvements in speed and fidelity on LTX. They also explicitly state that Wan High is approximately 3.5% faster than SenCache but has higher average LPIPS. The conclusion has been revised accordingly to reflect these trade-offs and their scope.

  3. We retain the additional applications of the reconstructed representation. Appendix I describes subject-preserving background completion and replacement using the reconstructed representation, with controllable editing results in Figs. 7 and 16. We present this additional use separately from the acceleration comparisons.

AuthorsOpenReview ↗

R2 · Reviewer 2AsZ (Part2)

R2-Q3: VBench per-dimension regressions and variation across seeds

Thank you for pointing out the VBench per-dimension regressions. We have reanalyzed all three matched generation seeds and have added the mean ± sample standard deviation across seeds for every dimension and both normalized aggregate scores in Table 7 of Appendix D.4, with the evaluation procedure and operating configurations unchanged.

  1. The Spatial relationship changes are not negative for every seed. Wan Native scores 66.69 ± 9.48, compared with 63.28 ± 6.48 for Ours-High. The paired differences for individual seeds are +0.38, −5.87, and −4.74 percentage points, with a mean of −3.41 ± 3.33 percentage points. The decline is therefore not consistent across all seeds.

  2. The Color decline cannot be fully attributed to random noise. Native scores 91.73 ± 0.45 and Ours-High scores 88.30 ± 0.49, with paired differences of −3.69, −3.31, and −3.27 percentage points. This is a repeatable metric-level trade-off for the High configuration in these experiments.

  3. Other acceleration methods also exhibit trade-offs across dimensions. Our expanded Table 7 also reports the baseline results for each dimension and their variation across seeds. On Wan, SenCache reduces Color, Spatial relationship, and the semantic aggregate by 1.71, 2.90, and 0.97 percentage points, respectively. ProfilingDiT reduces both Color and the semantic aggregate by 0.78 percentage points, while TeaCache reduces the semantic aggregate by 0.34 percentage points. These results characterize how different acceleration methods affect individual evaluation dimensions.

  4. Overall quality preservation and changes in individual dimensions should be interpreted separately. Wan Ours-High achieves a quality aggregate of 83.17 ± 0.34, compared with 83.17 ± 0.28 for Native. At the 2.64× speedup reported in Table 1, the overall VBench score decreases by 0.21 percentage points. LTX does not exhibit the same Color/Spatial pattern: Ours-High changes these dimensions by +1.48 and +0.08 percentage points, respectively, and the semantic aggregate by −0.21 percentage points. These results show that overall quality preservation can coexist with changes in specific semantic dimensions.

  5. The LTX baseline comparison supports the overall efficiency–quality gains. In the LTX results of our expanded Table 7, SenCache's quality/semantic aggregates are 82.71 ± 1.14 / 56.62 ± 2.73, compared with 83.03 ± 0.96 / 56.85 ± 2.85 for Ours-High. The overall comparison in Table 1 further shows that Ours-High achieves a higher VBench score (77.80 versus 77.49) at a larger speedup (2.48× versus 2.05×).

Thank you for the detailed response. I maintain my original score and have increased my confidence in this work.

Thank you for following up and for taking the time to consider our responses and revisions. We appreciate your constructive feedback and your continued engagement with our work.

Reviewer_jFhmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper accelerates frozen video diffusion transformers through selective token refreshes and a learned clean-state prediction. The prediction supplies both skipped-state reconstruction and dense attention references, aligned across noise levels and corrected by fresh observations. Routing uses visual and text-relevance cues, predicted reconstruction risk, and observation history. Experiments on Wan2.1-1.3B and LTX-Video-2B report up to 2.64$\times$ and 2.48$\times$ end-to-end acceleration, with fixed-budget encoder ablations and a factorial decoder analysis.

Critical strengths
  1. The decoder addresses a real coupling problem: sparse token evaluation still requires coherent dense context. Table 3 and Appendix H test all eight combinations of learned state recovery, learned contextual content, and noise-dependent alignment under identical execution masks. Removing state recovery raises LPIPS from 0.0330 to 0.1456. Removing alignment raises it to 0.1094. This is meaningful evidence for the integrated mechanism, beyond an architectural diagram or selected examples.

  2. The evaluation includes two backbones, three operating points, and encoder ablations over 944 prompts and three seeds. LTX Mid improves both latency and fidelity over the reported SenCache configuration (23.31 versus 25.90 s; LPIPS 0.019 versus 0.032). Appendix J counts reference construction, routing, decoding, and video writing in end-to-end timings. Evaluation clips also exceed the teacher-training lengths, providing some evidence of temporal transfer.

Critical weaknesses
  1. The broad novelty and comparative claims exceed the evidence. ToCa addresses token selection [1], TaylorSeer and LESA predict omitted features [2,3], and CG-Taylor couples forecasting with prediction reliability [5]. These differ from the proposed endpoint/context design, but constrain its new-paradigm claim. Table 1 omits forecasting baselines despite an official TaylorSeer Wan2.1 implementation [4]. Appendix H instead uses extrapolation fixed from the final two warm-up observations. This is a valid internal control for the integrated mechanism, but does not establish an advantage over competitive online prediction. The demonstrated technical contribution should be distinguished from broader claims of originality and superiority.

  2. The comparative gains have important boundaries. Wan High is only about 3.5% faster than SenCache, with worse LPIPS (0.033 versus 0.027); LTX provides stronger evidence, but its SenCache adapter reuses sensitivity data from version 0.9.1 on 0.9.6 (Appendix E.1). Neither proves unfairness, yet matched trade-off curves would clarify the advance. Table 2 supports individual cues within history-based selection but changes the cue, risk, and feedback rules together when comparing the full router. Appendix C.1 also omits training-prompt sources/counts, checkpoint provenance, and explicit separation from evaluation prompts; this is a disclosure gap, not evidence of leakage.

[1] Zou et al. (2025). Accelerating Diffusion Transformers with Token-wise Feature Caching.

[2] Liu et al. (2025). From Reusing to Forecasting: Accelerating Diffusion Models with TaylorSeers.

[3] Cai et al. (2026). LESA: Learnable Stage-Aware Predictors for Diffusion Model Acceleration.

[4] TaylorSeer authors (2025). Official TaylorSeer repository and Wan2.1 implementation.

[5] Guan et al. (2025). Forecasting When to Forecast: Accelerating Diffusion Models with Confidence-Gated Taylor.

Questions for authors
  1. Does the endpoint/context design improve the latency-fidelity trade-off over TaylorSeer or a competitive adaptive forecaster on the same Wan implementation? Please distinguish this comparison from the fixed warm-up extrapolation ablation.

  2. Within the final router, what changes when visual and semantic cues are removed separately at fixed execution budgets? Does the LTX advantage persist using sensitivity calibrated for the evaluated model?

  3. Please identify the training, validation, and calibration prompt sources/counts and their separation from evaluation, the initialization checkpoint, and the offline training cost and peak inference memory.

6 comments
AuthorsOpenReview ↗

R1 · Reviewer jFhm (Part1)

R1-W1/Q1: Distinction from online feature forecasting and direct comparison

Thank you for highlighting the relevant work and the missing comparison. We have added a full evaluation of the official TaylorSeer implementation and clarified the roles of comparisons with external methods and internal mechanism ablations.

  1. Our method jointly recovers omitted states and the complete context required by subsequent computation. ToCa reuses historical features according to token-wise caching sensitivity; TaylorSeer forecasts intermediate features through Taylor expansion; LESA learns stage-specific feature predictors; and CG-Taylor uses prediction error to choose between forecasting and full computation. In our method, a shared learned clean endpoint R supports both state reconstruction and dense K/V construction, together with noise-level alignment and correction from fresh observations. Explicitly evaluated tokens therefore receive aligned full context, while skipped regions receive a consistent reference for state recovery. Together, these mechanisms preserve the Native video associated with the same prompt and seed.

  2. We added a direct comparison with online forecasting. Table 6 in Appendix D.2 reports our new evaluation of three configurations of the official TaylorSeer Wan2.1 implementation, evaluated over 944 prompts and three seeds against their corresponding Native outputs. At comparable reported latency, Ours-High achieves 37.55 s and LPIPS 0.0330, compared with 36.67 s and LPIPS 0.4478 for TaylorSeer (fresh = 3). This reduces LPIPS by 92.6% and improves PSNR from 15.98 dB to 31.27 dB. The added matched eight-frame comparison in Fig. 8 shows that all three Ours configurations preserve the Native car's appearance, orientation, and motion progression, whereas the three TaylorSeer configurations exhibit changes in viewpoint, vehicle scale, and trajectory. These results support better preservation of the original content and its temporal evolution at comparable runtime. We have also added quality–speed curves across configurations in Fig. 9 of Appendix D.3.

  3. The fixed-extrapolation ablation identifies the roles of the integrated components. The internal ablation in Table 3 replays identical per-step execution masks and constructs a fixed extrapolated endpoint from the final two warm-up observations. With the other two components retained, replacing learned state recovery with fixed extrapolation, replacing learned contextual content with fixed extrapolation, or disabling noise-level alignment increases LPIPS from 0.0330 to 0.1456, 0.1046, and 0.1094, respectively. These controls identify each component's contribution under the same explicit-computation schedule. Appendix F provides the experimental settings and analysis.

AuthorsOpenReview ↗

R1 · Reviewer jFhm (Part2-1)

R1-W2/Q2/Q3: Fair comparisons, routing attribution, training data, and resource costs

Thank you for carefully examining the experimental settings. We have recalibrated and re-evaluated the LTX baseline and added routing ablations at matched budgets, a prompt-reuse control experiment, and resource measurements.

  1. We corrected the LTX baseline and completed a full re-evaluation. We replaced SenCache's outdated LTX 0.9.1 sensitivity data with sensitivity calibrated on the evaluated LTX 0.9.6 model and re-evaluated 944 prompts across three seeds, totaling 2,832 videos. The updated Table 1 shows that, compared with the corrected SenCache baseline, Ours-Mid/High reduce latency by 4.85%/17.22% and LPIPS by 50.26%/24.08%, respectively, improving both speed and fidelity to Native outputs. Appendix D.1 describes the updated calibration settings, and Fig. 9 in Appendix D.3 presents our newly added quality–speed curves that also include three TaylorSeer configurations.

  2. We clarify the speed–fidelity trade-offs on Wan. Table 1 reports the different operating points: compared with SenCache, Mid adds 2.74 s of latency while reducing LPIPS by 8.68% and improving PSNR by 0.89 dB; Low reduces LPIPS by 78.49% and improves PSNR by 6.82 dB while retaining a 1.42× speedup over Native; High reduces latency by 3.45%, with LPIPS 0.0330. Fig. 4 and Figs. 10–11 in Appendix D.5 illustrate better preservation of Native contours and details. Appendix I also explains how the reconstructed dense clean state supports subject-preserving background completion and replacement through segmentation and foreground compositing. Figs. 7 and 16 show the corresponding controllable editing results.

  3. We isolate the effects of guidance at the same execution budget. In Appendix E.1, we added a matched-budget experiment on 100 Wan prompts, holding the reconstruction network, per-step refresh counts, and actual DiT computation fixed. Our new final-output comparison in Table 8 shows that, relative to history-based coverage, the full router improves PSNR by 1.166 dB and reduces subject LPIPS and detail error by 19.54% and 13.55%, respectively. In our newly added shared-state experiment in Table 9, actual updates start from identical states and candidate locations. Visual guidance increases detail-error removal from 23.21% to 29.11% and reduces post-update detail error by 8.91%; semantic guidance increases subject-error removal from 22.61% to 27.93%. Subject metrics use the 60 samples with independent, nonempty subject masks, while the remaining metrics cover all 100 samples. On the final videos, compared with the variant without semantic guidance, the full router reduces subject LPIPS from 0.0495 to 0.0486 while increasing global LPIPS from 0.0454 to 0.0479, reflecting the prioritization of subject fidelity under a fixed computation budget.

  4. We disclose the training sources, counts, and initialization. We have expanded the training settings in Appendices C.1 and C.3, and the new Table 4 summarizes prompt IDs, distinct texts, and trajectories at each stage. Wan teacher data contain 946 VBench records with 944 distinct texts, yielding 2,838 trajectories across three seeds. Final adaptation uses 1,024 training trajectories from 680 prompt IDs (678 distinct texts), with 37 validation and 12 calibration trajectories. LTX's final training set contains 958 distinct texts and 960 trajectories, with eight additional validation trajectories. The released calibration uses 121-frame trajectories from three of these validation texts. The main teacher trajectories for Wan and LTX contain 41 frames at 480×832 and 17 frames at 704×1216, respectively, generated by 50 sampling steps of the frozen Native models.

    The Wan reconstruction backbone is adapted from GoPro-pretrained NAFNet weights. LTX transfers shape-compatible Wan weights and adapts its input and output interfaces. Wan's final adaptation training, validation, and calibration sets are disjoint by both prompt ID and text; the 12 calibration texts belong to the earlier validation split. LTX's training and validation texts are disjoint, and calibration uses a subset of the validation texts. Training and the main evaluation reuse VBench texts, with teacher seeds disjoint from the evaluation seeds 0, 42, and 114514.

AuthorsOpenReview ↗

R1 · Reviewer jFhm (Part2-2)

  1. We use VBench for its broad content coverage and directly test the effect of prompt reuse. We choose VBench prompts to construct teacher trajectories because their broad content coverage supports learning reconstruction across diverse generated content. To test whether reusing texts provides an additional benefit, we added a 100-prompt control experiment in Table 5 of Appendix C.3 that directly reconstructs the terminal state from the step-10 clean prediction. Two new decoders start from identical random initialization and are trained on 100 VBench or 100 EvalCrafter prompts, respectively, each with 300 trajectories and 900 optimizer updates. They share frozen auxiliary networks trained in the preceding stages. Evaluation uses 100 VBench prompts: 50 were seen by the VBench decoder during training, and 50 are absent from both new decoders' training sets.

    The LPIPS advantage of VBench over EvalCrafter training is 0.01189 on seen texts and 0.01096 on unseen texts. Their difference is 0.00093, with a 95% CI of [−0.00287, 0.00481]. On unseen texts, reconstruction also reduces LPIPS from the input baseline's 0.21435 to 0.20560. These results provide empirical support for this data choice: the perceptual reconstruction gains from VBench training transfer to texts unseen by the new decoder, and this experiment detects no additional advantage on seen texts.

  2. We report auxiliary parameter counts, offline costs, and peak inference memory. We added auxiliary parameter counts in Appendix H.1: 73.096M for Wan and 73.662M for LTX, counting the reconstruction and guidance networks retained at inference. In the new Table 17, we summarize recorded teacher generation, label preparation, inherited training, and calibration costs on a single A800: at least 84.59 h for Wan and at least 3.36 h of additional work for LTX after the shared Wan weights. We added peak GPU memory measurements for complete generation, including VAE decoding, in Table 18. For one 81-frame Wan sample, Native uses 24.02 GiB and all three Ours configurations use approximately 58.78 GiB. Across four 121-frame LTX samples, the corresponding measurements are 26.01 GiB and 37.76 GiB.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Appreciation and Summary of Common Review Concerns

We thank the Area Chair and reviewers for their careful evaluation and constructive discussion. Our core contribution is a reconstruction-aware coded computation framework that jointly determines which generative computations to execute and how to recover the omitted information. Perceptual-semantic computation allocation and learned reconstruction work together to recover missing states and provide noise-aligned dense context, preserving native outputs more faithfully at comparable latency. The reconstructed dense representation also supports subject-preserving background completion and replacement as additional applications.

Strengths noted in the reviews include the framework’s novelty, its treatment of the coupling between sparse computation and dense context, the controlled ablations and evaluation across two backbones, and its fidelity advantages. Our revision further addresses the following concerns:

  1. Contribution positioning and design justification (jFhm, 2AsZ, X9ET). We clarified how reconstruction reliability guides computation allocation and replaced “paradigm” with “framework” in the contribution statement. We added sensitivity experiments covering four coefficient groups and 17 configurations to provide empirical support for the design choices (Fig. 14).

  2. Comparison fairness and performance advantages (jFhm, 2AsZ). We clarified that the principal benefit is higher fidelity at comparable latency and removed the speedup range from the abstract. We added an official TaylorSeer comparison, where Ours-High reduces LPIPS by 92.6% at comparable latency. After recalibrating LTX SenCache, Ours-Mid still reduces LPIPS by 50.26% while lowering latency by 4.85% (Tables 1 and 6; Fig. 9).

  3. Component attribution and validation of key design choices (jFhm, X9ET). We expanded the explanations of routing factors and coupling across computation regions. We added matched-budget ablations of the complete router, shared-state update experiments, and targeted evaluations of importance refresh, warm-up length, candidate membership updates, and boundary consistency (Tables 8–15).

  4. Evaluation variability and per-dimension changes (2AsZ). We distinguished overall quality preservation from changes in individual semantic dimensions. We added means and standard deviations across all three seeds, together with paired-seed differences, to directly characterize the VBench results and their consistency (Table 7).

  5. Training data and reproducibility (jFhm, X9ET). We documented data splits, initialization, parameter counts, and offline costs, and added peak-memory measurements. Our new 100-prompt text-reuse control shows that gains from VBench training transfer to texts unseen by the new decoder, with no additional advantage detected on seen texts. This provides empirical support for choosing VBench for its broad content coverage (Tables 4–5 and 17–18).

We sincerely appreciate the Area Chair’s and reviewers’ time and constructive suggestions. We hope the revised manuscript and additional experiments present our contributions more clearly and provide a solid basis for your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.