acceptodds
Withdrawn from ICLR 2027

One Anchor is Not Enough: Trajectory Trace Caching for Video Diffusion Acceleration

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video diffusion transformers spend almost all of their inference cost on repeated forward passes of a large denoiser, and training-free caching reduces that cost by reusing network outputs across denoising steps. Existing caches keep only the latest output residual and reuse it as a frozen point estimate, so they ignore that this residual rotates, decays, and oscillates along the sampling path, and they trigger reuse from open-loop heuristics that are never checked against the error the reuse actually caused. The cost is largest in continuous video editing, where successive instructions are applied to one clip, every round reruns the full sampling loop, and the accelerated output of one round becomes the input of the next, so cache error is amplified rather than merely repeated. We observe that the conditional and unconditional residuals share a low-dimensional subspace whose reduced coefficients evolve almost linearly, so a skipped step becomes advancing a small state and projecting it back rather than reconstructing a high-dimensional output. We introduce **AlignVACE**, a training-free cache that maintains an online mode basis and a stability-projected reduced generator, reconstructs a whole run of skipped steps with one matrix exponential over the entire gap, and routes skips by a risk budget built from out-of-subspace energy and reduced prediction error, calibrated online against the realized error of the guided output. Because the stored trace is compressed, the operating point we use holds *less* device memory than a single uncompressed anchor. Across four editing categories, four-round edit chains, two editing backbones, and text-to-video and text-to-image generation, **AlignVACE** reaches speedup at dB on object editing and stays dB above the strongest baseline at the fourth edit round, while using GB of cache against to GB for prior caches. Our code is available at <https://anonymous.4open.science/r/AlignVACE-86BA/>.

Primary area:
generative models
Keywords:
Video Diffusion Acceleration, Training-Free Feature Caching, Storage-Compute Trade-off, Continuous Video Editing, Dynamic Mode Decomposition

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DUzVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a training-free video DiT caching method ALIGNVACE: utilizing streaming SVD and continuous-time dynamical operators to predict CFG residuals within a low-rank subspace, combined with closed-loop error calibration dynamic stepping, achieving leading acceleration ratios and PSNR/LPIPS in multi-round video editing and generation tasks.

Critical strengths

The proposed solution is interesting: Formulating diffusion residual caching as dynamical prediction within a low-rank subspace, combined with closed-loop error calibration for step-skipping, offers an interesting perspective while keeping the cache memory footprint low.

Critical weaknesses
  1. Lack of Comprehensive Comparison:

    a. The prediction-based methods under "Forecasting rather than reusing" (Section 2), such as TaylorSeer, are not included in the baselines;

    b. The paper chooses a video model (Wan2.1 1.3B) for Text-to-Image comparison instead of dedicated T2I models like FLUX or Hunyuan-Image.

  2. Limited Acceleration Gains and Insufficient Qualitative Comparison:

    The speedup improvement over the strongest baseline (EasyCache) is relatively small (mostly ~1.8×–1.9× vs. ~1.6×–1.8×), and qualitative comparisons are only conducted on the Wan backbone, lacking visual results on other tasks and models (e.g., LTX-Video and T2I).

  3. The method may be over-engineered, and the actual effectiveness of each module is difficult to determine:

    In Table 3, "Without adaptive rank" achieves better quality and lower memory than the "Full" method, "Without cross-round transfer" is faster than "Full", and "Without predictive increment" has a slightly higher PSNR than "Full".

Questions for authors
  1. How does ALIGNVACE compare against prediction-based caching methods such as TaylorSeer under the same settings?

  2. Can the authors provide Text-to-Image evaluations on dedicated T2I models (e.g., FLUX or Hunyuan-Image)?

  3. Can the authors provide qualitative visual comparisons on LTX-Video and T2I tasks, as well as results under higher acceleration ratios (e.g., >2.5×)?

  4. Given that removing individual modules in Table 3 improves either quality, memory, or speed over the "Full" model, how do the authors justify the necessity and complexity of each proposed module?

Reviewer_fAKXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AlignVACE, a training-free caching method for accelerating video diffusion transformers, particularly in multi-round video editing. AlignVACE models residual evolution use an online low-rank basis and streaming Dynamic Mode Decomposition (DMD) to predict skipped steps. It also introduces an adaptive risk budget to control approximation errors and reuses the learned basis across editing rounds. Experiments on VACE Wan2.1, LTX-Video, and Wan2.1 show that AlignVACE generally achieves higher speedups while maintaining competitive reconstruction quality compared to existing caching methods.

Critical strengths

1. A principled reformulation of feature caching.

The paper replaces the existing anchor-and-trigger design with a dynamical model: residuals are tracked in a low-rank subspace with a linear generator, skipped steps are predicted rather than copied, and recomputation is triggered by a risk signal calibrated against the realized error. This design contributes to the ability to capture rotation and decay that a single anchor cannot.

2. Thorough and candid evaluation.

The method is evaluated across editing categories, multi-round chains, two editing backbones, and text-to-video and text-to-image generation. The authors also report where the method does not outperform, including temporal consistency and the limited quality contribution of two components. This report makes the positive results more credible.

Critical weaknesses

The closest baselines are not compared.

The paper itself proposes forecasting-based caches such as TaylorSeer and SpeCa as the nearest in spirit, but all baselines are reuse-based. Without this comparison, it is unclear how much of the gain comes from modeling the residual dynamics specifically, rather than from predicting instead of reusing. I strongly recommend authors to report those comparision experiment results.

Questions for authors

Clarification requests

  1. Why were forecasting-based caches (TaylorSeer, SpeCa, confidence-gated Taylor) excluded from the baselines, given that the paper identifies them as the closest work in spirit? Was this a matter of implementation availability or a deliberate scoping choice?
  2. Methods are compared at different speedups, with the proposed method usually the fastest. How do the baselines compare when their triggers are tuned to match AlignVACE's speedup, or vice versa?
  3. How many clips and edit chains underlie Tables 1, 2, and 5, and what is the variance across clips and seeds?

Suggestions for future work

  1. Add TaylorSeer and SpeCa as baselines on the main settings (single-round editing, four-round chains, and text-to-video), under the same backbone and sampling configuration. This is the experiment the central claim depends on.
  2. Isolate the contribution of the dynamics model itself by comparing, within the AlignVACE framework, three predictors at matched speedup: zeroth-order reuse, first-order extrapolation, and the full generator with complex spectrum.
  3. Pair the above with the same trigger for all predictors, so that the gain from the closed-loop calibration can be separated from the gain from the predictor.
Reviewer_ZYu9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. AlignVACE predicts conditional and unconditional residuals in a shared low-rank space using fitted continuous-time dynamics. An online error calibrator controls skipped denoising steps, and cached states can transfer across editing rounds. Evaluation covers four editing categories, four-round chains, a second editing backbone, and image and video generation.
Critical strengths
  1. Combining a shared residual basis with online prediction-error feedback is a useful approach to controlling the cost and accuracy of diffusion caching. Table 2 reports 1.92x speedup and 19.30 dB at round four, compared with EasyCache at 1.80x and 17.81 dB.
  2. The evaluation includes component ablations and per-clip results. The paper explicitly reports the weaker temporal warp error and the quality-speed trade-offs of adaptive rank and cross-round transfer, which help delimit the contribution.
Critical weaknesses
  1. The supplied code leaves the predictor and cache representation unclear. The entry point installs alignvace_fast_patch.py, which retains a full FP32 residual and predicts z_exact + U(a_hat - a_exact), whereas Equation (2) gives U a_hat. DenseBasisStore also stores a dense basis, without the compression described in Appendix A.5. Equation (4) omits the anchor, although the code's persistent_bytes includes it. The paper needs to identify which implementation produced the tables and reconcile its cache state with the stated memory model.
  2. Section 2 identifies TaylorSeer as the closest forecasting baseline, but the experiments do not compare against it. Table 3 shows the benefit of the predictive increment within AlignVACE; it leaves the gain over existing temporal forecasting unresolved. The reported speed and quality gains are promising, but I lean toward rejection until the implementation discrepancy and this comparison are addressed.
Questions for authors
  1. Which predictor and basis-compression scheme produced Tables 1 to 5? Please provide the corresponding configuration and account for the full resident cache, including anchors and basis storage.
  2. How does AlignVACE compare with TaylorSeer at comparable latency on the same backbone and edit chains? If adaptation is impractical, please explain the constraints and which comparative claims the existing results support.
  3. Appendix A.6 describes held-out tuning. Please provide the evaluation counts, hardware, precision, and frozen configurations used for the main tables; the supplied synthetic example does not establish these settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.