- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes ANCHOR, a training-free cross-scale inference framework that accelerates video diffusion transformers. It runs a large model for an early prefix of denoising steps and then hands the trajectory off to a smaller model from the same family for the remaining steps. The authors argue that early steps determine the global spatiotemporal structure, a property they call trajectory anchoring. They support this with matched-budget comparisons showing that a contiguous early large-model prefix preserves fidelity to the full-Large output better than late or periodic placement. ANCHOR has two components. The first is an effective-SNR coordinate alignment that reconciles noise levels and prediction parameterizations across models. The second is a difficulty-aware predictor that selects a sample-specific handoff point among 10%, 20%, and 30% of the steps, based on early trajectory curvature and spectral energy. Experiments on Wan2.1, CogVideoX, and SkyReels-V2 report up to 3.7× end-to-end speedup with high frame-aligned similarity to the full-Large reference. Combined with caching methods such as TeaCache and MagCache, the speedup exceeds 3.8×.
Critical strengths
-
The method is simple, practical, and orthogonal to existing acceleration techniques. It requires only two pretrained models from the same family and no additional training. Table 4(a) shows that it can be combined with TeaCache and MagCache to reach more than 3.8× speedup. The claim that cross-step capacity scheduling operates on a different axis from intra-step caching is reasonable.
-
The system-level measurements are transparent. The paper reports end-to-end latency including weight-swapping costs, per-model swap overhead, and peak GPU memory identical to the full-Large baseline.
-
The alignment ablation is informative. The cumulative ablation in Table 7 shows that SNR matching and latent rescaling alone actually degrade fidelity (0.682), and that fidelity recovers to 0.901 only with prediction remapping. This clearly identifies which component matters for a successful handoff.
Critical weaknesses
-
Insufficient novelty. Users who can already afford to run the large model have little reason to take on the risk of mixing two models during inference unless the gain is substantial. The most meaningful contribution of this paper would be the heavy-first framework itself. However, this was first proposed by SRDiffusion (Cheng et al., 2025), which applies the same large-to-small structure to the same model pairs (Wan2.1 14B→1.3B and CogVideoX 5B→2B) and reports comparable speedups. The authors acknowledge this in the related work section, yet the abstract and introduction present heavy-first scheduling as a new finding that contradicts the coarse-to-fine consensus. The paper mainly adds analysis (prefix vs. suffix placement and contiguous vs. periodic allocation), which only strengthens the credibility of the heavy-first design. The methodological components are also limited. Therefore, I do not think these contributions are sufficiently novel.
-
Inefficient memory management. The small model (e.g., Wan2.1-1.3B, about 2.6 GB in BF16) is an order of magnitude smaller than the large model, so keeping both models resident on the GPU adds only marginal memory. On the H20 GPUs used in the paper (96 GB), the full-Large peak memory of Wan2.1 is 64.81 GiB, which leaves enough room to load both models. The rationale for repeatedly offloading and reloading the models is therefore unclear. If the goal is to claim that peak memory never exceeds that of the large model, this is not a meaningful advantage, because running the large model already dominates memory usage.
-
Missing baselines. The main table compares only against the large model, so it shows only how much faster the method is and how well quality is preserved relative to the large model. I would also like to see a baseline that uses only the small model. If the small model alone shows little quality degradation, there is little reason to combine the large and small models in this way to accelerate the large model. The full-Large VBench scores and a reduced-step large-model baseline are also missing.
-
The design rationale of the predictor is weak and its gain is small. The assumption that higher curvature indicates lower capacity demand is counterintuitive, yet it is justified by only one sentence. The normalization statistics are computed from only 30 trajectories, and the label criterion (CLIP ≥ 0.95) is arbitrary. The improvement over a fixed prefix under the same budget is also marginal (CLIP 0.965 → 0.967 on Wan2.1, Table 8).
Questions for authors
- Could you provide a quality comparison with SRDiffusion at matched speedups? Since SRDiffusion reports a direct handoff without alignment on the same CogVideoX 5B→2B pair, could you clarify how its setting differs from the direct handoff in Table 5?
- Could you add the latency and VBench of the small-model-only baseline (Sᴺ), the full-Large baseline (Lᴺ), and a reduced-step large-model baseline to Table 4?
- Could you provide a sensitivity analysis of the label threshold (CLIP ≥ 0.95) and explain why higher curvature indicates lower capacity demand?