acceptodds
Withdrawn from ICLR 2027

Trajectory-Aware Timestep Scheduling for Video Diffusion Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Video diffusion models require expensive model evaluations, making timestep allocation critical under limited inference budgets. Existing schedules allocate steps in fixed proportions and apply the same grid to all samples. We observe that trajectory acceleration magnitudes are initially high and decay rapidly, with substantial variation across samples concentrated in early denoising. Motivated by these findings, we propose a scheduling framework combining Double-Shift mapping with acceleration-driven online scheduling. Double-Shift reserves an absolute early-stage step budget and applies a separate nonlinear mapping. The online scheduler adapts early-stage timesteps to each sample using acceleration estimated from cached velocity predictions and a step-size rule derived from Euler local truncation error analysis. Experiments across models, benchmarks, and sampling budgets demonstrate improved generation quality, robustness to scheduling parameters, and negligible scheduling overhead. Both components are effective independently and yield further gains when combined, while remaining compatible with model variants and inference-time enhancement techniques.

Primary area:
generative models
Keywords:
Video diffusion models, Timestep scheduling, Efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xv2oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a training-free timestep scheduling framework for video diffusion models, combining Double-Shift mapping with acceleration-driven online scheduling. It reserves an early-stage step budget and adapts timestep locations using cached velocity predictions. Experiments across models, benchmarks, and sampling budgets report improved generation quality, complementary benefits from both components, and negligible scheduling overhead.

Critical strengths
  1. Practical applicability. The scheduler requires no additional training and reuses cached velocity predictions without extra model evaluations. Its reported negligible scheduling overhead makes it a practical approach to improving generation quality under fixed sampling budgets.
  2. Empirical support. Experiments cover multiple models, benchmarks, and sampling budgets, while ablations demonstrate complementary benefits from both components. These results support the method’s practical utility beyond a single model or evaluation setting.
Critical weaknesses
  1. Limited methodological novelty. Double-Shift extends an existing mapping, while the adaptive rule follows standard Euler error analysis. The stage boundary and budget allocation remain empirically chosen, limiting the conceptual advance beyond combining established scheduling ideas.
  2. Limited validation of sample-specific adaptation. The ablations demonstrate benefits from online scheduling but do not isolate the value of sample-specific timesteps against a shared, calibrated grid. This leaves the claimed need for per-sample adaptation insufficiently established.
Questions for authors

Please refer to the Critical Weaknesses.

Reviewer_GbxVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new timestep scheduling framework for video diffusion model. The authors observe that the trajactory accleration drops rapidly at early stages and remain at a relatively low level in later stages. The trajactory accleration also differs given different prompts. Based on these obervations, the authors propose a online scheduling double-shift methods. The experiments shos that the method achieves better generation quality.

Critical strengths
  1. The observation is convincing and the method is clear and easy to follow.
  2. The methods show improments among different video models, which shows its generality.
Critical weaknesses

My main concern lies in the limited conceptual and methodological novelty. The observation that early denoising exhibits larger trajectory acceleration is broadly consistent with the expected progression from noise toward coarse global structure, followed by refinement. Its empirical characterization is useful, but the paper does not establish a sufficiently deeper insight beyond this qualitative intuition. On the algorithmic side, Double-Shift reuses the existing Shift mapping in two intervals, with its principal modification being an absolute early-stage step budget. The online scheduler is the more interesting component, but its candidate step-size rule follows directly from the elementary Euler local-error expansion, with acceleration approximated through finite differences. Its practical implementation further relies on an empirical decay factor, budget clipping, and a prescribed boundary-closing step. The manuscript does not convincingly explain what substantive methodological advance emerges from this combination beyond an effective scheduling heuristic. The limited demonstrated novelty is my primary reason for recommending rejection.

Questions for authors
  1. Clarification of the claimed contribution: What is the substantive algorithmic insight beyond combining a two-stage Shift schedule with an Euler-error-motivated step-size controller? In particular, which design element follows specifically from the newly identified video-trajectory properties, rather than from a general preference for finer discretization where the velocity changes rapidly? Please distinguish the empirical contribution of characterizing video trajectories from the methodological novelty of the scheduler.

  2. Clarification of the implemented algorithms and budgets: What are the actual NFEs for every variant in Table 5, including their distribution for standalone AD? For the four-step AnyFlow-14B experiment in Table 3, what are the early-stage budget and exact update sequence? Under Algorithm 1, two warm-up steps, the boundary-closing step, and at least one late-stage step leave no acceleration-controlled update within four steps. Please clarify which components are active in this experiment.

Reviewer_uszZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses how to allocate a limited sampling budget in video diffusion models. Starting from two observations that the acceleration of the denoising trajectory drops sharply early on, and that this pattern varies with prompt complexity, it proposes Double-Shift, which assigns an absolute step budget and a separate nonlinear mapping to the early interval, and an online scheduler that adjusts the step size by estimating acceleration from finite differences of cached velocity predictions. No additional training is required, the overhead is small, and improvements over existing schedules are reported across several models and benchmarks.

Critical strengths
  1. It attaches directly to existing samplers without training or a separate scheduler. The scheduling cost is 0.028% of total inference, which is effectively negligible, so the practicality is high.
  2. In Table 2, with hyperparameters held fixed, it outperforms Shift tuned per budget across 20/30/50 steps. This is a clear practical advantage over existing approaches that require re-tuning whenever the budget changes.
  3. Covering several model families (Wan, CogVideoX, AnyFlow, Cosmos3) and three benchmarks supports the generality claim.
Critical weaknesses
  1. The central premise is not established. The paper's argument rests entirely on "larger acceleration implies larger discretization error." However, Eq. (7) is only an upper bound, not an equality, and the paper subsequently treats ∣a∣|a| as if it were the error itself. That a bound holds and that acceleration is the dominant component of the error are different claims. In fact the relative error of a single step depends on the magnitude of the velocity, whereas the proposed scheduler looks only at ∣a∣|a|. In regions where the velocity is overwhelmingly large, the relative contribution of the error can be small even when the acceleration is large, and the converse is also possible. An analysis of the share that acceleration accounts for in the error is needed.
  2. The correlation evidence in Fig. 2(c) is uncertain. According to the appendix, the reference for the local error is computed with two Euler half-steps. Since the quantity being compared and the reference share the same O(h2)O(h^2) error structure, it is possible that terms arising from acceleration were selectively amplified in the measurement. The authors themselves acknowledge in the appendix that this reference is not the exact solution, but the measurement should at least be repeated against a reference obtained with a sufficiently high-order solver such as RK45.
  3. In addition, the correlation analysis is pooled across steps. Each point in the scatter plot is a (sample, step) pair, and the correlation coefficient is a single value computed over all pairs in the interval t∈[0.9,1]t\in[0.9,1]. This interval contains several step positions, and if acceleration and local error decrease together as the steps proceed — as the authors find — then the points form clusters by step and the cluster centers line up along a single line. In that case the pooled correlation can come out high even if the correlation across samples within each step is close to zero. Meanwhile, what is needed as evidence for the proposed method is the acceleration values that differ across samples at the same step.
  4. The step from Eq. (6) to Eq. (7) is hard to follow. eie_i is not defined, and the O(hi3)O(h_i^3) term present in Eq. (6) disappears in Eq. (7), which takes the form of an inequality. With the intermediate step omitted, it is difficult to judge in what sense Eq. (7) is an upper bound.
  5. There is no evidence that the method actually reduced the discretization error. If discretization error is taken as the starting point of the argument, the paper should show the per-sampling-step LTE and acceleration curves of existing schedules (Shift, Beta, Linear-Quad) and whether these decrease after applying the proposed method. Benchmark scores alone do not tell us whether the mechanism worked as claimed.
  6. The scope of the sensitivity analysis is narrow. The η\eta sweep (Appendix F) is performed only in a single-model, single-benchmark setting. te=0.9t_e=0.9 and ρ=0.8\rho=0.8 are also presented as globally fixed values, with no evidence that they hold for other model families or datasets.
Questions for authors
  1. How is eie_i in Eq. (7) defined? Please also clarify how the O(hi3)O(h_i^3) term in Eq. (6) is handled in Eq. (7).
  2. Can the share that acceleration accounts for in the actual local error be quantified directly? Could you decompose the error measured against a high-order solver reference into the acceleration contribution and the remainder?
  3. Could you report scatter plots and correlation coefficients separated by step?
  4. Could you show directly that the proposed method reduces the local error relative to existing schedules?
  5. Do tet_e and ρ\rho work without re-tuning on other model families or resolutions? Are there results repeating the η\eta sensitivity experiment in other settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.