Trajectory-Aware Timestep Scheduling for Video Diffusion Models

Video diffusion models require expensive model evaluations, making timestep allocation critical under limited inference budgets. Existing schedules allocate steps in fixed proportions and apply the same grid to all samples. We observe that trajectory acceleration magnitudes are initially high and decay rapidly, with substantial variation across samples concentrated in early denoising. Motivated by these findings, we propose a scheduling framework combining Double-Shift mapping with acceleration-driven online scheduling. Double-Shift reserves an absolute early-stage step budget and applies a separate nonlinear mapping. The online scheduler adapts early-stage timesteps to each sample using acceleration estimated from cached velocity predictions and a step-size rule derived from Euler local truncation error analysis. Experiments across models, benchmarks, and sampling budgets demonstrate improved generation quality, robustness to scheduling parameters, and negligible scheduling overhead. Both components are effective independently and yield further gains when combined, while remaining compatible with model variants and inference-time enhancement techniques.