acceptodds
Under review as a conference paper at ICLR 2027

EcoVideo++: Characterizing and Exploiting Tri-dimensional Capability Boundaries for Collaborative Video Diffusion

Abstract

Diffusion Transformers (DiTs) are widely used for video generation, but their high computational cost limits deployment. Existing collaborative methods switch between large and small models at different denoising timesteps. However, this timestep-level strategy overlooks the multi-dimensional nature of the capability gap. We show that the gap varies across timesteps, concentrates on difficult frames, and is further localized to a small number of tokens. Thus, applying the large model to an entire timestep is unnecessarily coarse-grained. Based on these observations, we propose EcoVideo-β, a collaborative framework that coordinates large and small models at the timestep, frame, and token levels. The small model generates the full video to maintain global consistency, while the large model selectively refines challenging spatiotemporal regions. A pipelined edge–cloud execution strategy further reduces collaboration overhead. EcoVideo-β enables fine-grained capability allocation for efficient, high-quality video generation.

Primary area:
generative models
Keywords:
video generation, Large-small model, Collaborative generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_T3W1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EcoVideo++, a training-free large–small model collaboration framework for video DiTs that allocates large-model computation across denoising stages, temporal segments, and spatial regions using cheap online proxies, namely the clean-latent change for switching, neighboring-frame latent differences for segment selection, and the small model's CFG gap for spatial cores, combined with context-expanded crops, global 3D RoPE offsets, and smooth fusion, and reports 2.29–2.93× speedups on Wan2.1, Wan2.2, and CogVideoX with VBench within 0.0027 of large-only inference.

Critical strengths
  1. Extending large–small collaboration from step-level switching to spatiotemporal regions is well-motivated and practically relevant for expensive video DiTs.
  2. The online routing signals add negligible overhead, and the CFG-gap signal reuses computations already required for classifier-free guidance.
  3. The joint-denoising design, including context expansion, global RoPE alignment, and smooth fusion, is carefully engineered and each component is ablated.
  4. The evaluation is broad, covering three model pairs, the full VBench prompt set with five seeds, signal-replacement ablations, sensitivity sweeps, and compatibility with TeaCache.
  5. The paper is clearly written, with an explicit algorithm and detailed implementation settings for the Wan models.
Critical weaknesses
  1. EcoVideo++ outperforms large-only inference on many dimensions such as aesthetic quality, imaging quality, overall consistency, multiple objects, and scene, even though most post-switch computation comes from the small model, and no explanation is given.
  2. The main claims rest on VBench gaps of at most 0.0027 but are not supported by confidence intervals, human studies, or reference-based fidelity metrics against large-only outputs.
  3. The capability analysis uses only 40 prompts and discusses the proxy–gap correlations only qualitatively.
  4. Potential crop-induced boundary artifacts and global-coherence failures, especially for high-motion or multi-object videos, are not analyzed.
  5. Lack of Failure Case Analysis. For instance, how does the method handle scenes with rapid, complex motion spanning the entire frame (where spatial selection might miss crucial background dynamics)? How does it handle long videos where the small model's temporal consistency inherently degrades?
Questions for authors
  1. How does EcoVideo++ surpass large-only quality on multiple dimensions, and can you provide confidence intervals and reference-based fidelity against large-only outputs?
  2. What switch step does the step-only variant in Table 3 use, and is it the same as in the full method?
  3. What are the average switch step, the actual fraction of tokens processed by the large model after context expansion, and the per-component latency breakdown?
  4. How were the hyperparameters chosen for Wan2.2 and CogVideoX, and were they tuned on the evaluation prompts?
  5. How does EcoVideo++ compare with region-adaptive sampling or sparse-attention methods at matched speedup?
  6. Can you provide human preference results and a failure-case analysis, for example when spatial cores miss relevant objects?
7 comments
AuthorsOpenReview ↗

Response to Reviewer T3W1 (1/5)

We sincerely thank you for the valuable feedback and for recognizing our work. We address each question below, and full videos are available at https://anonymous.4open.science/w/ecovideo-beta/.

W1 & W2 & Q1: EcoVideo++ outperforms large-only inference on many dimensions such as aesthetic quality, imaging quality, overall consistency, multiple objects, and scene, even though most post-switch computation comes from the small model, and no explanation is given. The main claims rest on VBench gaps of at most 0.0027 but are not supported by confidence intervals, human studies, or reference-based fidelity metrics against large-only outputs. How does EcoVideo++ surpass large-only quality on multiple dimensions, and can you provide confidence intervals and reference-based fidelity against large-only outputs?

A1: Thank you for raising these points. We address them in four parts: why EcoVideo++ scores higher on some dimensions, confidence intervals, reference-based fidelity, and human evaluation.

Why EcoVideo++ scores higher on some dimensions. We first clarify that EcoVideo++ does not surpass large-only inference overall. As shown in Table C1 of our paper, on Wan2.1 both its Quality score and Semantic score are slightly lower than those of large-only inference, and the gains on aesthetic quality, imaging quality, overall consistency, multiple objects and scene are offset by small drops on dimensions such as subject consistency, temporal flickering, human action and color. The paired prompt-level analysis in Table 1 makes this explicit. With the same prompts and seeds, Aesthetic Quality, Multiple Objects and Scene become slightly negative (−0.00852, −0.00819, −0.01221), Overall Consistency is +0.00484, and every per-dimension difference is below 0.02 in magnitude. We therefore regard these differences as fluctuations around the large-only quality rather than a systematic improvement. The consistent gain is on Imaging Quality (+0.01780 [+0.00719, +0.02848]). One reason is that collaboration changes the generation trajectory: the spatial cores are selected by the small model's conditional–unconditional response, which concentrates the large-model computation on prompt-related objects, while the remaining regions follow the small model's prediction. Full videos of large-only inference and EcoVideo++ are available at https://anonymous.4open.science/w/ecovideo-beta for direct comparison.

Confidence intervals. The official 16-dimension VBench prompt list contains 944 distinct prompts, and both methods use the same five seeds for every prompt. We pair EcoVideo++ with large-only inference by prompt, resampling the five seeds and all applicable dimensions of each prompt together, and compute 95% intervals of the paired difference with 10,000 prompt-stratified bootstrap resamples on normalized dimension scores.

Table 1: Paired differences (EcoVideo++ − large-only) and 95% intervals. Per-dimension rows: Wan2.1-14B + Wan2.1-1.3B.

Model pairMetric#PromptsΔ95% interval
Wan2.1Aesthetic Quality93−0.00852[−0.02343, +0.00598]
Wan2.1Imaging Quality930.0178[+0.00719, +0.02848]
Wan2.1Overall Consistency930.00484[−0.00442, +0.01382]
Wan2.1Multiple Objects82−0.00819[−0.01938, +0.00295]
Wan2.1Scene86−0.01221[−0.02508, +0.00018]
Wan2.1Dynamic Degree720.01944[−0.01389, +0.05556]
Wan2.1Object Class79−0.00755[−0.01550, −0.00014]
Wan2.1Spatial Relationship84−0.00570[−0.01787, +0.00649]
Wan2.1Temporal Style1000.00006[−0.01176, +0.01178]
Wan2.1VBench944−0.00013[−0.00375, +0.00359]
Wan2.2VBench944−0.00410[−0.00821, −0.00009]
CogVideoXVBench944−0.00373[−0.00783, +0.00042]

On all three model pairs, the paired VBench difference is at most 0.0041, and the whole interval lies within 0.0083 of large-only inference (below 1% of the score), which quantifies that EcoVideo++ is on par with large-only inference.

AuthorsOpenReview ↗

Response to Reviewer T3W1 (2/5)

A1 (continued). Reference-based fidelity against large-only outputs. We compute PSNR, LPIPS and SSIM between the videos generated by each method and the large-only videos under the same prompts and seeds on Wan2.1-14B + Wan2.1-1.3B.

Table 2: Similarity to Wan2.1-14B outputs and speedup.

MethodPSNR ↑LPIPS ↓SSIM ↑Speedup ↑
T-Stitch11.30460.65310.37471.66×
HybridSD8.48200.82490.18291.50×
EC-Diff8.62500.82880.18141.55×
SRDiffusion16.29050.44860.54691.96×
EcoVideo++14.39020.48150.45992.93×

EcoVideo++ ranks second on all three metrics while being the fastest. All collaborative methods show limited similarity, because large–small collaboration inherently changes the generation trajectory: once the small model participates in denoising, the difference at each step is carried into all subsequent steps, so no collaborative method can reproduce the large-only output pixel by pixel.

Human evaluation. We conduct a blind Good/Same/Bad (GSB) user study on Wan2.1. We randomly sample 100 VBench prompts and generate videos with large-only inference, SRDiffusion, EC-Diff and EcoVideo++ under the same seed. For each prompt, EcoVideo++ is paired with each of the three baselines, giving 300 pairs. 24 participants each rate 50 pairs, so every pair is rated by 4 participants and each row of Table 3 collects 400 valid votes. Videos are shown side by side in random order without method names, and participants judge prompt adherence, visual quality and temporal consistency.

Table 3: GSB preference on Wan2.1 (%, G / S / B = EcoVideo++ is better / same / worse; 400 votes per row).

EcoVideo++ vs.Prompt adherence (G/S/B)Visual quality (G/S/B)Temporal consistency (G/S/B)
Large-only28.25 / 37.00 / 34.7537.25 / 30.00 / 32.7521.75 / 32.75 / 45.50
SRDiffusion41.75 / 34.50 / 23.7549.25 / 29.00 / 21.7542.00 / 37.00 / 21.00
EC-Diff41.25 / 36.00 / 22.7569.50 / 21.00 / 9.5050.50 / 32.75 / 16.75

Against large-only inference, EcoVideo++ achieves a 2.93× speedup and is rated better or tied in 65.25% of votes on prompt adherence, 67.25% on visual quality (37.25% better, 30.00% tied), and 54.50% on temporal consistency. Against SRDiffusion, 49.25% of votes favor EcoVideo++ on visual quality and 42.00% on temporal consistency, rising to 78.25% and 79.00%, respectively, when ties are included. Against EC-Diff, 69.50% of votes favor EcoVideo++ on visual quality and 21.00% rate the two methods equally, giving 90.50% better or tied.

W3: The capability analysis uses only 40 prompts and discusses the proxy–gap correlations only qualitatively.

A2: Thank you for the comment. We clarify the scale and the quantitative evidence.

Scale. The 40 prompts are split into 20 simple and 20 complex prompts, each with three seeds, giving 120 trajectories per model pair. At every step, both models are evaluated on the same latent, timestep and text condition, so each trajectory provides paired large–small predictions at every step, frame and spatial position.

Quantitative analysis. The proxy–gap correlation is quantified by recall. In Figure 3(b) of our paper, ranking frames by the neighboring-frame latent difference is evaluated by Top-3 recall of the frames with the largest model gap, against uniform and random selection. In Figure 3(c–d), the small model's conditional–unconditional response is evaluated by Top-8% recall of the positions with the largest large–small gap, with 73% and 70% spatial overlap on the shown examples. Recall directly quantifies what routing needs: whether the selected positions are those where the large model matters most. At the generation level, the ablation in Table 2 of our paper tests each proxy on the full VBench under matched budgets: replacing the spatial signal with uniform token selection lowers VBench from 0.8348 to 0.8102, replacing the temporal signal with uniform frame selection gives 0.8235, and replacing the switching signal with fixed switching gives 0.8108.

AuthorsOpenReview ↗

Response to Reviewer T3W1 (3/5)

W4: Potential crop-induced boundary artifacts and global-coherence failures, especially for high-motion or multi-object videos, are not analyzed.

A3: Thank you for this point. EcoVideo++ is designed so that cropping does not break the global prediction.

Design. The small model always predicts the complete video, so every location receives a continuous full-frame update. The large model reads each crop together with surrounding context, uses the crop's global coordinates for 3D RoPE, and only its core prediction is smoothly blended back with a weight that tapers toward the boundary. Therefore, there is no hard seam between large- and small-model regions.

Quantitative evidence. The cumulative ablation in Table 4 of our paper isolates each component. Core cropping with hard replacement scores 0.8127; adding context expansion gives 0.8216; global-coordinate 3D RoPE gives 0.8294; and smooth fusion gives 0.8348. The full design recovers 0.0221 VBench over hard replacement with only 34 s additional latency (674.33 s vs. 640.18 s).

Visual evidence. At https://anonymous.4open.science/w/ecovideo-beta/, we provide videos with the crop boxes overlaid, including an object moving quickly across a crop boundary and two objects separating within the scene, which show no visible seams and no missed objects.

Q2: What switch step does the step-only variant in Table 3 use, and is it the same as in the full method?

A4: Thank you for the question. The step-only variant in Table 3 of our paper uses a stricter threshold (τ = 0.045) and hands the entire video over to the small model at around step 39 on average. The full method uses the same stability rule with τ = 0.15 and starts joint denoising at step 7.53 on average, after which the 14B model processes only the selected crops.

Q3: What are the average switch step, the actual fraction of tokens processed by the large model after context expansion, and the per-component latency breakdown?

A5: At the default Wan2.1 settings (P = 2, CFG = 5), the full method starts joint denoising at step 7.53 on average across 128 videos. After context expansion, the 14B model processes 6.60% of the full-frame video tokens per joint step on average, and the full 50-step trajectory, including the initial full-frame steps, uses 18.80% of the large-only 14B token count. A joint step uses 1.742 crops on average (at most two, median expanded size 4 × 48 × 62 in latent frames × height × width) and takes 8.9983 s, with 7.9374 s for the full-frame small-model pass, 0.8969 s for the 14B crop passes, 0.1553 s for region selection, 0.0064 s for fusion, and 0.0023 s for the scheduler.

Reviewer_1efnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Large and small video diffusion models from the same family differ in what they can render, and the difference is not uniform. It shrinks at different steps for different content, varies across temporal segments, and concentrates around objects. EcoVideo++ moves large model computation along all three axes. A stability test on the clean latent estimate decides when to leave full large model denoising, and a neighboring frame latent difference picks temporal segments. The small model's response to the text condition picks spatial cores, which the large model processes as small crops with global positional coordinates and blends into the small model's full video prediction. The method is evaluated on three Wan and CogVideoX model pairs for end to end speed and VBench quality.

Critical strengths
  1. The evaluation protocol is stronger than usual. It uses the full VBench prompt set with five shared seeds, identical checkpoints and sampling settings within each pair, and end to end latency including encoding and decoding. Ablations add the three axes cumulatively and replace each online signal under matched budgets.

  2. The TeaCache experiment answers the obvious question. The method dominates large only inference with caching and stacks with it, so it complements caching.

Critical weaknesses
  1. The operating point is tuned on the evaluation set. The switching threshold and both budgets are the best values of sweeps over the full benchmark, and those same scores are the headline. The gap to large only inference is a few units in the third decimal, with no intervals despite five seeds. The near lossless claim rests on an unquantified margin.

  2. The efficiency claim cannot be audited. There is no average switching step, crop size, large model token count, or per component timing. Patience and guidance scale are never given values, so Algorithm 1 cannot be run.

  3. One baseline row is broken. On CogVideoX, EC-Diff scores 0.2281 on Overall Consistency against 0.8837 for large only inference, and 0.1453 on Scene against 0.6918. It still posts the highest Motion Smoothness at 0.9542. The outputs likely degenerated, and the paper prints the row with only a note that the sample is darker.

Questions for authors
  1. Report the average switching step, the number and size of the crops, and a per component breakdown of one joint step. What fraction of large model computation remains at the default budgets.

  2. Provide paired prompt level intervals for the three quality margins, and separately for the dimensions where the method scores above large only inference.

  3. Give the values of the patience and the guidance scale, and retune the threshold and budgets on a held out prompt split.

  4. What happened to EC-Diff on CogVideoX. Was its switching schedule tuned per pair, and can it be repaired to restore a fair comparison point.

  5. What is the peak memory of holding both models on one card compared with large only inference.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 1efn (1/5)

W1 (The operating point is tuned on the evaluation set): The switching threshold and both budgets are the best values of sweeps over the full benchmark, and those same scores are the headline. The gap to large only inference is a few units in the third decimal, with no intervals despite five seeds. The near lossless claim rests on an unquantified margin.

A1: Thank you for pointing this out. We address this concern from three aspects: interval estimates of the quality margin, re-tuning on a held-out prompt split, and evaluation on an additional benchmark never used for tuning.

Interval estimates. The official 16-dimension VBench prompt list contains 946 entries with 944 distinct prompts, and both methods use the same five seeds for every prompt. We treat each prompt as one unit, resampling its five seeds and all its applicable dimensions together, and compute 95% intervals of the paired difference (EcoVideo++ − large-only) with 10,000 prompt-stratified bootstrap resamples on normalized dimension scores. The paired VBench difference is −0.00013 [−0.00375, +0.00359] on Wan2.1, −0.00410 [−0.00821, −0.00009] on Wan2.2 and −0.00373 [−0.00783, +0.00042] on CogVideoX (Table 1). On all three pairs, the whole interval lies within 0.0083 VBench of large-only inference, i.e., below 1% of the score, which quantifies the near-lossless margin. The per-dimension intervals are given in Table 3 (A4).

Re-tuning on a held-out prompt split. To separate tuning from evaluation, we randomly sample 10% of the prompts (94 prompts) as a tuning split and perform the sweeps of τ, ρf and ρs only on this split. The selected configuration is τ= 0.15, ρf = 10% and ρs = 8%. We then freeze this configuration and evaluate it on the remaining 850 prompts, which are never used in the sweep.

Table 1: Paired VBench differences (EcoVideo++ − large-only) with 95% intervals.

Model pairPrompt setPromptsΔ95% interval
Wan2.1Full VBench944−0.00013[−0.00375, +0.00359]
Wan2.2Full VBench944−0.00410[−0.00821, −0.00009]
CogVideoXFull VBench944−0.00373[−0.00783, +0.00042]
Wan2.1Tuning split94−0.00126[−0.01078, +0.00814]
Wan2.1Held-out split8500.00368[−0.00003, +0.00729]

On the 850 held-out prompts, EcoVideo++ scores +0.00368 relative to large-only inference, with an interval of [−0.00003, +0.00729]. The quality margin therefore holds on prompts never seen during tuning, and the headline result is not a product of tuning on the evaluation prompts.

Evaluation on T2V-CompBench. We further evaluate on T2V-CompBench, which is not involved in any tuning, using the default configuration without modification.

Table 2: T2V-CompBench results (Wan2.1-14B + Wan2.1-1.3B, default configuration).

MethodConsist. attr.Dynamic attr.SpatialMotionActionInteractionNumeracyMean
Large-only0.83690.0570.57090.26960.75040.72390.51130.5314
SRDiffusion0.82760.05690.5630.26950.74980.70960.49490.5245
EcoVideo++0.83390.05710.57450.26960.750.72790.520.5333

On this unseen benchmark, EcoVideo++ achieves a seven-category mean of 0.5333, on par with large-only inference (0.5314) and higher than SRDiffusion (0.5245), confirming that the default configuration generalizes beyond VBench.

AuthorsOpenReview ↗

Response to Reviewer 1efn (2/5)

W2 & Q1: The efficiency claim cannot be audited. There is no average switching step, crop size, large model token count, or per component timing. Patience and guidance scale are never given values, so Algorithm 1 cannot be run. Report the average switching step, the number and size of the crops, and a per component breakdown of one joint step. What fraction of large model computation remains at the default budgets?

A2: At the default Wan2.1 settings listed in A5, Table 5 (P = 2, CFG = 5), the full method starts joint denoising at step 7.53 on average across 128 videos. After context expansion, the 14B model processes 6.60% of the full-frame video tokens per joint step on average, and the full 50-step trajectory, including the initial full-frame steps, uses 18.80% of the large-only 14B token count. A joint step uses 1.742 crops on average (at most two, median expanded size 4 × 48 × 62 in latent frames × height × width) and takes 8.9983 s, with 7.9374 s for the full-frame small-model pass, 0.8969 s for the 14B crop passes, 0.1553 s for region selection, 0.0064 s for fusion, and 0.0023 s for the scheduler.

AuthorsOpenReview ↗

Response to Reviewer 1efn (3/5)

W3 & Q4: One baseline row is broken. On CogVideoX, EC-Diff scores 0.2281 on Overall Consistency against 0.8837 for large only inference, and 0.1453 on Scene against 0.6918. It still posts the highest Motion Smoothness at 0.9542. The outputs likely degenerated, and the paper prints the row with only a note that the sample is darker. What happened to EC-Diff on CogVideoX? Was its switching schedule tuned per pair, and can it be repaired to restore a fair comparison point?

A3: Thank you for the careful check. We explain the scores and their cause below.

Why these scores coexist. The three scores measure different properties and can hold at the same time. Motion Smoothness measures frame-to-frame motion consistency and does not depend on semantics, while Scene and Overall Consistency measure whether the video matches the prompt. For example, if the prompt asks for "a horse running in the snow" and the video shows a horse running on grass, the motion is smooth, so Motion Smoothness is high, but the scene is wrong, so Scene and Overall Consistency are low. This is exactly the pattern of EC-Diff on CogVideoX: smooth motion but lost scene semantics, which also matches the dark sequence in Figure 5 of our paper, where the requested motorcycle–bus relation is hard to recognize.

Cause at the method level. The failure stems from EC-Diff's mechanism rather than from its switching point. EC-Diff approximates the large-model predictions of multiple steps from historical predictions. During consecutive approximated steps on CogVideoX, the scene gradually drifts away from the prompt, and the subsequent 2B steps then continue the drifted scene coherently. The result is a temporally smooth video with lost scene semantics.

Switching schedule and fair comparison. We run EC-Diff with the same generation settings as all other methods within each pair. Since the semantic drift comes from the history-based approximation itself, it is not a matter of the switching point. A fair comparison is also provided by the strongest remaining baseline: on CogVideoX, EcoVideo++ achieves 0.7769 VBench in 375.18 s, versus 0.7708 in 520.16 s for SRDiffusion.

Reviewer_43UFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EcoVideo++, a training-free framework for accelerating video diffusion through large–small model collaboration across denoising steps, temporal segments, and spatial regions. Experiments on Wan2.1, Wan2.2, and CogVideoX demonstrate 2.29–2.93× speedups with minimal VBench degradation.

Critical strengths
  1. Fine-grained collaboration: The tri-dimensional analysis provides useful insights into large–small model capability differences and enables more efficient computation allocation.
  2. Promising experimental results: The method achieves substantial acceleration with competitive VBench scores across multiple model families, supported by comprehensive ablations.
Critical weaknesses
  1. Insufficient evaluation of generation fidelity: VBench alone may not adequately reflect prompt adherence or similarity to large-only outputs. Some qualitative results show noticeable deviations from the large model.
  2. Limited practical and generalization analysis: Memory overhead, model-loading strategies, and hyperparameter sensitivity across different model pairs are not sufficiently investigated.
Questions for authors
  1. How is EcoVideo++ adapted to Wan2.2-T2V-A14B, especially its expert-based architecture and compatibility with Wan2.1-1.3B?
  2. What is the peak GPU memory overhead? Are Wan2.2-T2V-A14B and Wan2.1-1.3B loaded simultaneously?
  3. Could the authors report similarity metrics (e.g., PSNR, SSIM, LPIPS) against large-only outputs?
  4. Given VBench's limitations, could the authors provide additional evaluations or full generated videos to demonstrate perceptual quality and temporal consistency?
  5. Some visual results deviate from large-only outputs, while SRDiffusion appears to preserve certain semantic details better. Could the authors provide more qualitative comparisons and explain these differences?
  6. Is the switching threshold τ = 0.15 also optimal for Wan2.2 and CogVideoX? Could the authors provide sensitivity analyses for different model pairs?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 43UF (1/3)

We sincerely thank you for the positive assessment and the constructive suggestions. We address each point below. Full videos are available on the anonymous page: https://anonymous.4open.science/w/ecovideo-beta/.

Q1: How is EcoVideo++ adapted to Wan2.2-T2V-A14B, especially its expert-based architecture and compatibility with Wan2.1-1.3B?

A1: Thank you for the question.

Expert routing. Wan2.2-T2V-A14B selects its high-noise or low-noise expert from the current timestep. EcoVideo++ keeps this rule unchanged: Wan2.2 decides which expert is active, and EcoVideo++ decides where it runs. Before the transition, the active expert denoises the full video, and the clean-latent stability test (Eq. 4) runs on its predictions. After the transition, Wan2.1-1.3B denoises the full latent, and the active Wan2.2 expert refines the selected regions.

Compatibility with Wan2.1-1.3B. Wan2.2-T2V-A14B uses the Wan2.1 VAE, the same text encoder and the same flow-matching scheduler as Wan2.1-1.3B. The two models therefore predict on the same latent grid and timestep, and their outputs are blended before a single scheduler update without any conversion.

Settings and results. We use guidance scales of 4 and 3 for the high- and low-noise stages, a patience of 2, and the default routing configuration. This pair reaches 0.8396 VBench at a 2.75× speedup, close to 0.8423 for large-only inference (Table 1 of our paper).

W2 & Q2: Limited practical and generalization analysis: memory overhead, model-loading strategies, and hyperparameter sensitivity across different model pairs are not sufficiently investigated. What is the peak GPU memory overhead? Are Wan2.2-T2V-A14B and Wan2.1-1.3B loaded simultaneously?

A2: Model loading. Yes. As described in Section 5.1, all experiments run on a single NVIDIA H200 GPU with batch size 1. The large and small models are loaded onto the same GPU at initialization and stay resident during generation; for Wan2.2, both experts and Wan2.1-1.3B are resident. Switching between models therefore requires no weight loading or CPU–GPU transfer, and the reported end-to-end latency (prompt encoding, denoising and VAE decoding) contains no model-swapping cost.

Peak memory. We measure the GPU memory of model weights and working tensors at 1280×720, 81 frames and batch size 1.

Table 1: GPU memory on a single H200 (model weights and working tensors, GB).

Model pairLarge-only (GB)EcoVideo++ (GB)Increase (GB)
Wan2.1-14B + 1.3B46.17549.0132.838
Wan2.2-A14B + Wan2.1-1.3B74.75277.592.838

EcoVideo++ increases memory by only 2.838 GB on both pairs. The increase is identical because both pairs add the same small denoiser, while the text encoder and VAE are shared and counted once. This small overhead brings 2.93× and 2.75× speedups.

Hyperparameter sensitivity across model pairs is addressed in A6 below.

AuthorsOpenReview ↗

Response to Reviewer 43UF (2/3)

W1 & Q3: Insufficient evaluation of generation fidelity. VBench alone may not adequately reflect prompt adherence or similarity to large-only outputs. Some qualitative results show noticeable deviations from the large model. Could the authors report similarity metrics (e.g., PSNR, SSIM, LPIPS) against large-only outputs?

A3: Thank you for this important suggestion. We provide reference-based similarity metrics and a blind human evaluation.

Similarity to large-only outputs. For every prompt and seed, we compare each accelerated method with the Wan2.1-14B large-only video generated from the same initial noise.

Table 2: Similarity to Wan2.1-14B outputs and speedup (Wan2.1-14B + Wan2.1-1.3B).

MethodPSNR ↑LPIPS ↓SSIM ↑Speedup ↑
T-Stitch11.30460.65310.37471.66×
HybridSD8.4820.82490.18291.50×
EC-Diff8.6250.82880.18141.55×
SRDiffusion16.29050.44860.54691.96×
EcoVideo++14.39020.48150.45992.93×

EcoVideo++ ranks second on all three metrics among the five collaborative methods and clearly outperforms T-Stitch, HybridSD and EC-Diff. Compared with SRDiffusion, EcoVideo++ is 1.50× faster (674.33 s vs. 1,010.13 s) and achieves higher VBench (0.8348 vs. 0.8238).

Why collaborative outputs differ from the large-only video. Large–small collaboration inherently changes the generation trajectory. Once the small model participates in denoising, its predictions differ from those of the large model, and the scheduler updates the latent with these predictions; Each difference is carried into all subsequent steps. Therefore, no collaborative method can reproduce the large-only output pixel by pixel, which is why all collaborative methods show limited PSNR/SSIM. These metrics measure the deviation from one particular large-only sample rather than generation quality itself, so we complement them with a human evaluation.

Blind human evaluation (GSB). We conduct a blind Good/Same/Bad (GSB) user study on Wan2.1. We randomly sample 100 VBench prompts and generate videos with large-only inference, SRDiffusion, EC-Diff and EcoVideo++ under the same seed. For each prompt, EcoVideo++ is paired with each of the three baselines, giving 300 pairs. 24 participants each rate 50 pairs, so every pair is rated by 4 participants and each row of Table 3 collects 400 valid votes. Videos are shown side by side in random order without method names, and participants judge prompt adherence, visual quality and temporal consistency.

Table 3: GSB preference on Wan2.1 (%, G / S / B = EcoVideo++ is better / same / worse).

EcoVideo++ vs.Prompt adherence (G/S/B)Visual quality (G/S/B)Temporal consistency (G/S/B)
Large-only28.25 / 37.00 / 34.7537.25 / 30.00 / 32.7521.75 / 32.75 / 45.50
SRDiffusion41.75 / 34.50 / 23.7549.25 / 29.00 / 21.7542.00 / 37.00 / 21.00
EC-Diff41.25 / 36.00 / 22.7569.50 / 21.00 / 9.5050.50 / 32.75 / 16.75

Against large-only inference, EcoVideo++ achieves a 2.93× speedup and is rated better or tied in 65.25% of votes on prompt adherence, 67.25% on visual quality (37.25% better, 30.00% tied), and 54.50% on temporal consistency. Against SRDiffusion, 49.25% of votes favor EcoVideo++ on visual quality and 42.00% on temporal consistency, rising to 78.25% and 79.00%, respectively, when ties are included. Against EC-Diff, 69.50% of votes favor EcoVideo++ on visual quality and 21.00% rate the two methods equally, giving 90.50% better or tied.

AuthorsOpenReview ↗

Response to Reviewer 43UF (3/3)

Q4: Given VBench's limitations, could the authors provide additional evaluations or full generated videos to demonstrate perceptual quality and temporal consistency?

A4: Thank you for the suggestion. The supplementary material of our submission already contains the generated videos. For easier viewing, we further provide full videos at https://anonymous.4open.science/w/ecovideo-beta/ , shown side-by-side for large-only, SRDiffusion, EC-Diff and EcoVideo++ and covering all prompts in Figure 5 and Figures E2–E4 of our paper, so that perceptual quality and temporal consistency can be examined directly over the entire video. In addition to VBench, Table 2 (A3) provides reference-based fidelity and Table 3 (A3) provides human judgments on the full videos.

Q5: Some visual results deviate from large-only outputs, while SRDiffusion appears to preserve certain semantic details better. Could the authors provide more qualitative comparisons and explain these differences?

A5: Thank you for the careful observation. We explain the differences below and provide more qualitative comparisons at https://anonymous.4open.science/w/ecovideo-beta/.

Explanation of the differences. As discussed in A3, large–small collaboration inherently changes the generation trajectory, and the degree of deviation depends on how much computation is left to the small model. SRDiffusion keeps the large model on the full video for more denoising steps, so some details stay closer to the large-only output, but at a much higher latency. EcoVideo++ instead assigns large-model computation to the selected temporal segments and spatial regions, so a few details outside these regions may differ from the large-only output. For example, in Figure E3 of our paper, for "Cafeteria." EcoVideo++ keeps the dining-room setting with a different foreground arrangement, and for "Classroom." it presents several desks and students instead of a single close-up. In both cases the prompt is fully satisfied, and the difference lies only in composition.

Advantages of EcoVideo++. Despite these local differences, EcoVideo++ achieves a better overall quality–efficiency trade-off than SRDiffusion. As shown in Table 1 of our paper, compared with SRDiffusion, EcoVideo++ reduces latency by 27.9%–34.4% across the three model pairs, while achieving higher overall VBench scores and higher subject and background consistency (e.g., Wan2.1: 0.8348 vs. 0.8238 VBench, 674.33 s vs. 1,010.13 s). Against SRDiffusion, 49.25% of votes favor EcoVideo++ on visual quality and 42.00% on temporal consistency, rising to 78.25% and 79.00%, respectively, when ties are included (Table 3 in A3). Full side-by-side videos are available at https://anonymous.4open.science/w/ecovideo-beta/.

Q6: Is the switching threshold τ = 0.15 also optimal for Wan2.2 and CogVideoX? Could the authors provide sensitivity analyses for different model pairs?

A6: Thank you for the suggestion. We extend the τ sweep in Table D1 of our paper to Wan2.2 and CogVideoX, keeping all other settings fixed.

Table 4: Switching threshold and speedup (VBench / speedup).

τWan2.1 VBench / speedupWan2.2 VBench / speedupCogVideoX VBench / speedup
Large-only0.8369 / 1.00×0.8423 / 1.00×0.7784 / 1.00×
0.10.8336 / 2.23×0.8383 / 2.01×0.7759 / 1.84×
0.15 (default)0.8348 / 2.93×0.8396 / 2.75×0.7769 / 2.29×
0.20.8232 / 3.01×0.8306 / 2.83×0.7679 / 2.34×
0.250.8122 / 3.26×0.8197 / 3.11×0.7589 / 2.49×

τ = 0.15 achieves the highest VBench on all three model pairs, although it was selected only on Wan2.1 and transferred without retuning. The trend is consistent across pairs: τ = 0.10 delays the transition and lowers speedup without improving quality, while τ ≥ 0.20 switches earlier and trades quality for speed. Since τ measures the relative change of the clean-latent estimate, it is scale-free and transfers robustly across backbones. The sensitivity to the budgets ρf and ρs is reported in Appendix D of our paper.

Thanks the authors for the detailed response. The additional experimental results have addressed some of my concerns, and I will maintain my positive rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviews and Author Responses

Dear Area Chair,

Thank you for handling our submission. As the discussion period draws to a close, we would like to summarize the reviews and the evidence we added in response.

Summary of the Reviews

All three reviewers gave EcoVideo++ a positive recommendation (Weak Acceptance). The reviews recognized the value of extending collaboration between large and small models from whole denoising steps to selected temporal segments and spatial regions. They also highlighted the practical speedups and the evaluation across model families. EcoVideo++ achieves 2.29–2.93× end-to-end speedups across Wan2.1, Wan2.2, and CogVideoX without additional training.

The main concerns fell into four groups.

  • Generation quality and fidelity (43UF, 1efn, T3W1). The reviewers asked for additional statistical analysis of VBench results, human preference evaluations, fidelity metrics against large-only outputs, and full-video comparisons.
  • Inference cost and reproducibility (43UF, 1efn, T3W1). The reviewers asked for the memory overhead of keeping both models resident, the observed switching steps, actual token counts after crop expansion, per-component latency, and key inference settings.
  • Hyperparameter selection and generalization (43UF, 1efn, T3W1). The reviewers asked how the switching threshold transfers across model pairs and how the method performs on held-out prompts and an additional benchmark.
  • Method design and scope (T3W1). The reviewer asked how to interpret the collaboration granularity ablation and how local cropping behaves under complex motion, with multiple subjects, and in longer videos.

Our Responses and Manuscript Revisions

We addressed the vast majority of the reviewers’ concerns through additional experiments, measurements, and clarifications.

  • Additional evidence on generation quality. We reported same-seed fidelity metrics, ran a blind human evaluation, and provided full-video comparisons. In the visual-quality comparison with SRDiffusion, 49.25% of votes favored EcoVideo++, while 21.75% favored SRDiffusion. Reviewer 43UF noted that the additional results addressed several of their concerns and reaffirmed their positive recommendation.
  • Measured inference costs and documented settings. We reported observed switching steps, actual token counts after context expansion, and per-component latency, along with the patience and classifier-free guidance settings. Keeping both models resident added approximately 2.84 GB of GPU memory for each of the two Wan model pairs relative to large-only inference. Reviewer 1efn explicitly confirmed that the efficiency breakdown and inference settings addressed their questions.
  • Evaluated transfer across models and benchmarks. We added threshold sensitivity results for Wan2.2 and CogVideoX and ran a held-out prompt evaluation on Wan2.1. On T2V-CompBench, EcoVideo++ scored 0.5333 with the default Wan2.1 configuration and no retuning, compared with 0.5314 for large-only inference. Reviewer 1efn also found this additional evaluation helpful.
  • Clarified the collaboration mechanism and its scope. We explained how EcoVideo++ continues to use the large model to refine selected regions at every joint denoising step and discussed where local cropping can fail. In the revision, we will report the switching settings for each Table 3 variant, reconcile the VBench aggregate and paired results, and specify the configuration used for the held-out evaluation.

We appreciate the reviewers’ constructive feedback. We believe the positive recommendations and the additional evidence strengthen the case for accepting EcoVideo++.

Best wishes,
Authors of Submission 8106

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.