acceptodds
Under review as a conference paper at ICLR 2027

Heavy-First, Light-Late: Reversing Capacity Scheduling for Efficient Video Diffusion

Abstract

Diffusion transformers achieve state-of-the-art video generation quality but incur prohibitive inference latency due to repeated full-capacity denoiser evaluations across sampling steps. Many multi-capacity methods follow a coarse-to-fine paradigm, allocating higher computational budgets to late-stage refinement. In contrast, we show that capacity demands peak during early denoising: We formalize this property as , showing that executing a brief prefix of early steps with a large model locks the generative path, after which a compact model can complete the remaining steps with minimal fidelity loss. Motivated by this observation, we propose (daptive etwork apacity andoff for ptimal outing), a training-free cross-scale inference framework that dynamically routes computation between heterogeneous diffusion transformers. ANCHOR introduces a coordinate alignment mechanism to reconcile latent representations across model scales without fine-tuning, alongside an adaptive handoff predictor that determines sample-specific transition points from early trajectory dynamics. Comprehensive evaluations on Wan2.1, CogVideoX, and SkyReels-V2 demonstrate up to speedup while preserving high frame-aligned similarity to the full-Large reference. Furthermore, our adaptive routing exposes superior fidelity-speed Pareto operating points by dynamically assigning longer prefixes to complex video samples.

Primary area:
generative models
Keywords:
video diffusion transformers, efficient inference, model-capacity scheduling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3FxiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ANCHOR, a training-free cross-scale inference framework that accelerates video diffusion transformers. It runs a large model for an early prefix of denoising steps and then hands the trajectory off to a smaller model from the same family for the remaining steps. The authors argue that early steps determine the global spatiotemporal structure, a property they call trajectory anchoring. They support this with matched-budget comparisons showing that a contiguous early large-model prefix preserves fidelity to the full-Large output better than late or periodic placement. ANCHOR has two components. The first is an effective-SNR coordinate alignment that reconciles noise levels and prediction parameterizations across models. The second is a difficulty-aware predictor that selects a sample-specific handoff point among 10%, 20%, and 30% of the steps, based on early trajectory curvature and spectral energy. Experiments on Wan2.1, CogVideoX, and SkyReels-V2 report up to 3.7× end-to-end speedup with high frame-aligned similarity to the full-Large reference. Combined with caching methods such as TeaCache and MagCache, the speedup exceeds 3.8×.

Critical strengths
  1. The method is simple, practical, and orthogonal to existing acceleration techniques. It requires only two pretrained models from the same family and no additional training. Table 4(a) shows that it can be combined with TeaCache and MagCache to reach more than 3.8× speedup. The claim that cross-step capacity scheduling operates on a different axis from intra-step caching is reasonable.

  2. The system-level measurements are transparent. The paper reports end-to-end latency including weight-swapping costs, per-model swap overhead, and peak GPU memory identical to the full-Large baseline.

  3. The alignment ablation is informative. The cumulative ablation in Table 7 shows that SNR matching and latent rescaling alone actually degrade fidelity (0.682), and that fidelity recovers to 0.901 only with prediction remapping. This clearly identifies which component matters for a successful handoff.

Critical weaknesses
  1. Insufficient novelty. Users who can already afford to run the large model have little reason to take on the risk of mixing two models during inference unless the gain is substantial. The most meaningful contribution of this paper would be the heavy-first framework itself. However, this was first proposed by SRDiffusion (Cheng et al., 2025), which applies the same large-to-small structure to the same model pairs (Wan2.1 14B→1.3B and CogVideoX 5B→2B) and reports comparable speedups. The authors acknowledge this in the related work section, yet the abstract and introduction present heavy-first scheduling as a new finding that contradicts the coarse-to-fine consensus. The paper mainly adds analysis (prefix vs. suffix placement and contiguous vs. periodic allocation), which only strengthens the credibility of the heavy-first design. The methodological components are also limited. Therefore, I do not think these contributions are sufficiently novel.

  2. Inefficient memory management. The small model (e.g., Wan2.1-1.3B, about 2.6 GB in BF16) is an order of magnitude smaller than the large model, so keeping both models resident on the GPU adds only marginal memory. On the H20 GPUs used in the paper (96 GB), the full-Large peak memory of Wan2.1 is 64.81 GiB, which leaves enough room to load both models. The rationale for repeatedly offloading and reloading the models is therefore unclear. If the goal is to claim that peak memory never exceeds that of the large model, this is not a meaningful advantage, because running the large model already dominates memory usage.

  3. Missing baselines. The main table compares only against the large model, so it shows only how much faster the method is and how well quality is preserved relative to the large model. I would also like to see a baseline that uses only the small model. If the small model alone shows little quality degradation, there is little reason to combine the large and small models in this way to accelerate the large model. The full-Large VBench scores and a reduced-step large-model baseline are also missing.

  4. The design rationale of the predictor is weak and its gain is small. The assumption that higher curvature indicates lower capacity demand is counterintuitive, yet it is justified by only one sentence. The normalization statistics are computed from only 30 trajectories, and the label criterion (CLIP ≥ 0.95) is arbitrary. The improvement over a fixed prefix under the same budget is also marginal (CLIP 0.965 → 0.967 on Wan2.1, Table 8).

Questions for authors
  1. Could you provide a quality comparison with SRDiffusion at matched speedups? Since SRDiffusion reports a direct handoff without alignment on the same CogVideoX 5B→2B pair, could you clarify how its setting differs from the direct handoff in Table 5?
  2. Could you add the latency and VBench of the small-model-only baseline (Sᴺ), the full-Large baseline (Lᴺ), and a reduced-step large-model baseline to Table 4?
  3. Could you provide a sensitivity analysis of the label threshold (CLIP ≥ 0.95) and explain why higher curvature indicates lower capacity demand?
Reviewer_J7hUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how model capacity should be allocated across denoising steps for efficient video diffusion. Contrary to the conventional small-first, large-late scheduling paradigm, the authors observe that allocating high-capacity computation to early denoising steps better preserves fidelity to full-Large model outputs. Based on this observation, they propose ANCHOR, a training-free cross-scale inference framework that executes a Large model during an initial denoising prefix and switches to a Small model for the remaining steps. The framework introduces an effective-SNR-based coordinate alignment mechanism to facilitate cross-model handoff and an adaptive routing strategy that determines the handoff point using trajectory curvature and spatiotemporal spectral energy. Experiments on Wan2.1, CogVideoX, and SkyReels-V2 demonstrate substantial inference speedups while maintaining relatively high fidelity to full-Large reference videos.

Critical strengths
  • The paper provides consistent empirical evidence that allocating Large-model computation to early denoising steps preserves full-Large reference fidelity more effectively than allocating the same computational budget to late steps. The comparisons across Wan2.1, CogVideoX, and SkyReels-V2, together with the contiguous-versus-periodic allocation experiments, support the practical value of heavy-first scheduling. This observation is consequential because it provides a useful principle for designing computationally efficient inference schedules without retraining the underlying diffusion models.
  • ANCHOR combines cross-model handoff, effective-SNR coordinate alignment, and adaptive capacity allocation into a practical inference framework. The coordinate alignment mechanism is particularly useful, as the ablation results demonstrate substantial improvements over direct latent handoff. The method also achieves meaningful end-to-end inference speedups across different video diffusion architectures, with model-swapping overhead included in the latency measurements.
  • These results demonstrate the practical potential of cross-scale capacity scheduling as an additional dimension for accelerating video generation.
Critical weaknesses
  • The central mechanistic claim is not sufficiently supported by direct experimental evidence.

A central claim of the paper is that early denoising steps determine global spatiotemporal trajectories, whereas late steps primarily perform localised texture synthesis. However, the presented evidence mainly demonstrates that early Large-model computation better preserves similarity to full-Large reference outputs. In particular, the primary metric, frame-aligned CLIP similarity, measures semantic similarity between corresponding frames but does not directly characterise motion trajectories, global spatial structure, or local texture refinement. Therefore, the observed advantage of early Large-model execution does not necessarily establish the proposed functional distinction between early and late denoising stages. This distinction is important because the trajectory-anchoring interpretation serves as a central motivation for the proposed framework. More direct analysis of spatial structure, temporal motion, or frequency-dependent changes across denoising stages would be necessary to substantiate this claim.

  • The adaptive routing criteria lack sufficient justification, and their benefits over simpler alternatives remain limited.

The proposed adaptive predictor combines trajectory curvature and spatiotemporal spectral energy into a difficulty score. However, the relationship between these metrics and the required Large-model capacity is not sufficiently established. In particular, the assumption that higher trajectory curvature indicates lower capacity demand is not adequately justified. Moreover, Table 6 suggests that the spectral criterion accounts for most of the improvement: compared with spectral-only routing, the combined predictor achieves the same under-allocation rate (14.9%) and reduces over-allocation only modestly (39.9% to 37.2%). The independent contribution of the curvature criterion therefore remains unclear. The matched-average-budget comparison in Appendix C shows relatively modest improvements over fixed-prefix scheduling, especially on Wan2.1, where CLIP-to-L improves only from 0.965 to 0.967. These results raise questions about whether the proposed adaptive routing mechanism provides sufficiently substantial benefits over a simpler fixed-prefix strategy. The evaluation also lacks a comprehensive end-to-end comparison with directly related cross-scale scheduling methods, particularly SRDiffusion, and does not establish a consistent advantage over existing acceleration methods under matched latency or computational budgets. This limits the strength of the claimed efficiency–fidelity superiority.

Questions for authors
  1. The experiments demonstrate that early Large-model allocation improves similarity to full-Large reference outputs. However, how does this observation directly support the stronger claim that early steps determine global spatiotemporal trajectories while late steps mainly synthesise localised textures? Could the authors provide a more direct analysis of spatial structure, motion trajectories, or low-/high-frequency information across denoising stages? Such evidence would help distinguish the proposed mechanism from the more general observation that early model errors have a larger influence on final outputs.
  2. What empirical evidence supports the assumption that higher solver curvature corresponds to lower capacity demand, while higher spatiotemporal spectral energy corresponds to higher capacity demand? Could the authors report the correlations between these two metrics and the oracle-required Large-prefix length, together with the corresponding difficulty prediction accuracy or confusion matrix?
Reviewer_pfBBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ANCHOR is a training-free framework that accelerates video diffusion by using a large denoiser in early sampling steps and a smaller model thereafter. It combines coordinate alignment with adaptive handoff to balance generation fidelity and inference cost.

Critical strengths
  1. The paper is well written and easy to follow, with clear motivation and an accessible presentation of the proposed method.
  2. The proposed method is simple yet effective. Allocating the large model to early denoising steps and using a smaller model for the remaining steps provides practical acceleration without retraining the denoisers.
Critical weaknesses
  1. The novelty appears limited. The main idea is to use a larger model in early denoising steps and a smaller model in later steps, which seems primarily an engineering heuristic.
  2. Table 4 lacks standalone generation-quality results, such as VBench scores, for the full-Large and full-Small baselines in each model family (e.g., L50L_{50} and S50S_{50}). This makes it difficult to directly assess the potential quality degradation introduced by the proposed schedule.
  3. The experiments lack results on few-step models. Can the method be applied to few-step (e.g., 4-step) generators, and how does its generation quality compare with existing few-step models?
  4. Beyond the caching baselines, comparisons with other training-free acceleration approaches are missing, including methods that change resolution across denoising stages, such as Bottleneck Sampling [1], and sparse attention methods, such as Sparse VideoGen [2].
  5. The evaluation also lacks human preference assessments, which would help better substantiate the claimed generation quality.

[1] Ye Tian et al. Training-free Diffusion Acceleration with Bottleneck Sampling. arXiv preprint arXiv:2503.18940, 2025.

[2] Haocheng Xi et al. Sparse Video-Gen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity. Proceedings of the 42nd International Conference on Machine Learning, 2025.

Questions for authors

see Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.