Heavy-First, Light-Late: Reversing Capacity Scheduling for Efficient Video Diffusion

Diffusion transformers achieve state-of-the-art video generation quality but incur prohibitive inference latency due to repeated full-capacity denoiser evaluations across sampling steps. Many multi-capacity methods follow a coarse-to-fine paradigm, allocating higher computational budgets to late-stage refinement. In contrast, we show that capacity demands peak during early denoising: $\text{early steps determine global spatiotemporal trajectories, while late steps perform localized texture synthesis.}$ We formalize this property as $\text{trajectory anchoring}$, showing that executing a brief prefix of early steps with a large model locks the generative path, after which a compact model can complete the remaining steps with minimal fidelity loss. Motivated by this observation, we propose $\text{ANCHOR}$ ($\text{A}$daptive $\text{N}$etwork $\text{C}$apacity $\text{H}$andoff for $\text{O}$ptimal $\text{R}$outing), a training-free cross-scale inference framework that dynamically routes computation between heterogeneous diffusion transformers. ANCHOR introduces a coordinate alignment mechanism to reconcile latent representations across model scales without fine-tuning, alongside an adaptive handoff predictor that determines sample-specific transition points from early trajectory dynamics. Comprehensive evaluations on Wan2.1, CogVideoX, and SkyReels-V2 demonstrate up to $3.7\times$ speedup while preserving high frame-aligned similarity to the full-Large reference. Furthermore, our adaptive routing exposes superior fidelity-speed Pareto operating points by dynamically assigning longer prefixes to complex video samples.