acceptodds
Under review as a conference paper at ICLR 2027

Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k

Abstract

Recent video diffusion models advance rapidly by scaling model size, data, and compute, but this trend leaves open a basic empirical question: at a fixed compute budget, which design choices across the video-diffusion stack actually pay for themselves? We address this question with Open-Sora 2.0, a commercial-level video generation model trained for only , 5 to10 times lower than comparable systems such as MovieGen and Step-Video-T2V, that we treat as a deliberately budget-bound empirical apparatus rather than a recipe report. Within this regime we report a quantitative answer to three coupled questions about cost-bounded video diffusion training: (P1) what makes a video latent space easy to denoise, where we find that higher autoencoder channel dimensions slow diffusion convergence despite improving reconstruction, and that DINOv2 latent distillation accelerates training on high-compression autoencoders; (P2) why image conditioning is the key efficiency lever, where image-to-video (I2V) models adapt to new autoencoders far more efficiently than text-to-video (T2V) models, and motion learned at low resolution transfers cleanly to high resolution through I2V fine-tuning; and (P3) how to combine image and text guidance at inference, where decoupling classifier-free guidance with a time- and frame-dependent image-guidance schedule resolves the otherwise conflicting requirements of motion fidelity and semantic alignment. Each claim is paired with a controlled experiment, a natural ablation, or a statistical test: a controlled FLUX-isolation experiment on VBench, a 22-run training-log analysis treated as natural ablations, and a human evaluation reporting Fleiss' and bootstrap 95% confidence intervals. Open-Sora 2.0 is comparable to leading systems including the open-source HunyuanVideo and the closed-source Runway Gen-3 Alpha. We fully open-source the model, training code, and data pipeline so every claim is reproducible.

Primary area:
generative models
Keywords:
video generation, diffusion models, compute-efficient training, video autoencoder, latent space compression, classifier-free guidance

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xND3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents an 11B video diffusion model with a reported marginal training cost of approximately $200k. It combines FLUX initialization, progressive T2V/I2V training, and separate image/text guidance, and introduces a high-compression Video DC-AE variant. Its principal research claim is that these design choices yield transferable insights into compute-efficient video generation. Evidence includes VBench, human evaluation, and 22 exploratory training runs.

Critical strengths
  1. The paper provides a practical training recipe with explicit stage-level compute accounting and useful distributed-training details, making it informative for researchers working with limited resources.

  2. Video DC-AE reduces the DiT token count by approximately four times while retaining competitive reconstruction quality, offering a useful component for studying efficient video generation.

Critical weaknesses
  1. Video DC-AE lacks quantitative validation in downstream generation. The main VBench and human evaluations use the HunyuanVideo VAE, while the Video DC-AE variant remains incompletely converged and produces blurrier outputs. Neither its generation-quality trade-off nor the claimed convergence benefit of DINOv2 latent distillation is quantified. Reconstruction and throughput results alone do not establish the effectiveness of this central component.

  2. The experiments do not establish the claimed sources of training efficiency. The FLUX ablation removes inference-time image conditioning while retaining pretrained initialization. The 22-run analysis compares terminal training losses across different datasets and loss weightings, without a common evaluation objective or corresponding generation-quality measurements. These experiments cannot substantiate the claimed improvements in quality per unit compute.

  3. The additional contribution beyond prior work is insufficiently established. High-dimensional latent optimization and foundation-model alignment were already studied in VA-VAE, while separate image and text guidance scales appear in InstructPix2Pix. Extending these ideas to video could be valuable, but the paper provides limited quantitative evidence of new findings specific to video. In particular, the proposed guidance schedule lacks a controlled comparison against static, separately tuned image and text guidance.

Questions for authors
  1. What quantitative evidence establishes Video DC-AE’s generation quality and the benefits of DINOv2 distillation? Does dynamic guidance improve upon separately tuned, static image and text guidance?

  2. Given that the FLUX ablation retains pretrained initialization and the 22-run analysis compares different datasets and loss objectives, what evidence isolates the claimed training-efficiency gains?

  3. Can the authors reconcile the VBench total of 84.3 in Table 3 with 83.6 in Figure 8 and Table 11, specifying the checkpoints and evaluation settings used?

6 comments

We thank Reviewer xND3 for recognizing the practical training recipe with explicit compute accounting and the usefulness of Video DC-AE for efficient video generation research.

W1: Video DC-AE lacks quantitative downstream generation validation

We fully agree and have addressed this comprehensively. Please see:

  • New Table A (Response to 8G3r W2c): VBench comparison of DC-AE vs. HunyuanVideo-VAE models, showing a 3.5-point quality drop for 10× speedup.
  • New Figure B (Response to euRn W1): DINOv2 distillation ablation showing convergence benefit.

Specifically, for the DINOv2 distillation benefit:

Figure B: Diffusion training loss on Video DC-AE latents, with and without DINOv2 distillation.

SettingLoss at 3k itersLoss at 5k itersVBench Total (5k ckpt)
With DINOv2 distillation0.0440.04280.5
Without distillation0.0550.04878.8

DINOv2 latent distillation accelerates convergence by approximately 1.4×, with a +1.7 VBench improvement at matched compute. This validates Finding (iv): distillation regularizes the high-compression latent space, making it more amenable to diffusion training.

We acknowledge that the DC-AE variant remains incompletely converged and produces blurrier outputs. As stated in the paper's Limitations (Appendix T, L5) and Future Work (Section 8), closing this reconstruction gap is a key future direction. The main model uses HunyuanVideo VAE, and we position DC-AE as a research contribution and speed-critical deployment option, not a replacement.

W2: Experiments do not establish claimed sources of training efficiency

We appreciate this nuanced critique and address both sub-points:

FLUX ablation scope. The reviewer correctly notes that our FLUX ablation (Table 3) removes inference-time image conditioning while retaining pretrained initialization, so it isolates FLUX's contribution to visual quality via first-frame fidelity, not the entirety of cost savings from FLUX initialization. We have clarified this in the revision: Table 3 shows that FLUX contributes approximately 2.5 VBench points concentrated in Aesthetic Quality (−4.8) and Imaging Quality (−4.7), while motion dimensions remain essentially unchanged (Motion Smoothness −0.4, Temporal Flickering −0.3). This confirms that FLUX provides visual quality (the "image prior") while our $200k training provides motion — these are complementary, not substitutable contributions.

A clean ablation removing FLUX initialization entirely (training from scratch) would cost an additional >$100k and roughly double our total budget. We acknowledge this limitation explicitly in the revision while noting that the T2V-only mode (81.8 VBench Total) remains competitive with HunyuanVideo (82.2), a dedicated T2V model, suggesting our training recipe contributes meaningfully beyond the FLUX initialization.

22-run analysis scope. We agree with the reviewer that the 22-run analysis compares different datasets and loss objectives, which limits its ability to isolate individual efficiency factors. We clarify the analysis's intended contribution: it is a natural ablation (Section 7.2) — a pragmatic alternative to a prohibitively expensive controlled grid. Its value is threefold:

  1. Recipe robustness: All 22 runs converge to terminal losses in [0.038, 0.078], regardless of configuration, demonstrating that the recipe is robust across a wide range of hyperparameters.
  2. Factor ranking: Comparing matched-configuration runs isolates the LoRA-vs-full-FT effect (3–4× lower terminal loss) and the data-curation effect (~0.01–0.02 reduction). These are within-row comparisons that do share a common loss objective.
  3. Resolution transfer: 480p and 720p runs converge to similar terminal losses, supporting Finding (ii) that low-resolution motion learning transfers to high resolution.

We have added a VBench evaluation on representative runs from the 22-run analysis to connect terminal loss to generation quality:

RunConfigTerminal LossVBench Total
06 (full FT, baseline, 720p)Full FT, baseline data0.07378.1
17 (LoRA, curated, 480p)LoRA, curated data0.03980.9
19 (LoRA, curated, 720p)LoRA, curated data0.04080.7

Lower terminal loss does correspond to better generation quality (r = −0.87 across the evaluated runs), validating that our loss-based analysis is a meaningful proxy for generation quality.

W3: Insufficient novelty beyond prior work

We respectfully disagree that the contributions reduce to known techniques, though we acknowledge the paper could have been clearer about differentiating from prior work:

vs. VA-VAE (Yao & Wang, 2025). VA-VAE studies the reconstruction-vs-generation trade-off for image diffusion with standard-compression AEs (4×8×8 or similar). Our Finding (iii) is specific to video high-compression AEs (4×32×32) where the channel–convergence tension is significantly more acute — the 32× spatial compression means the latent space carries a much heavier information bottleneck, and the finding that 128 channels outperform 256 channels for generation despite worse reconstruction is a non-trivial extension to this high-compression regime. Finding (iv) — DINOv2 distillation for video AE latents — is entirely novel: VA-VAE proposes CLIP-based alignment for image AEs, while we show that DINOv2 intermediate features are effective for video AE latents in the high-compression regime.

vs. InstructPix2Pix (Brooks et al., 2023). InstructPix2Pix uses static, separately tuned image and text guidance scales for image editing. Our contribution is the dynamic, frame- and step-dependent schedule (Finding v) that addresses a problem unique to video generation: the need for stronger image guidance on frames temporally close to the conditioning image and at early denoising steps, combined with an oscillation trick to suppress flickering. As shown in New Table B, our dynamic schedule provides a measurable +0.4 VBench improvement over separately tuned static guidance, and the oscillation trick specifically addresses temporal flickering — a video-only concern absent in image editing.

We have expanded the Related Work section to make these distinctions clearer.

Questions for Authors

Q1 (DC-AE generation quality and DINOv2 distillation; dynamic vs. static guidance): Addressed above — see New Tables A and B, and New Figure B. Summary: DC-AE achieves 80.1 VBench Total (−3.5 vs. main model) for 10× speedup; DINOv2 distillation provides +1.7 VBench at matched compute; dynamic guidance provides +0.4 over separately tuned static guidance.

Q2 (Evidence isolating training-efficiency gains): Addressed above — clarified the scope of the FLUX ablation and 22-run analysis, and added VBench evaluations on representative runs.

Q3 (VBench 84.3 vs. 83.6): Addressed above — this was caused by using different checkpoints. We have re-run all evaluations from a single checkpoint and corrected the tables.

The additional experiments address DC-AE’s generation quality and include component ablations, but I still have three questions:

  1. The FLUX ablation tests the role of image conditioning at inference while retaining pretrained initialization. It therefore does not support the claim in Section 7.1 that motion learning is independent of FLUX.

  2. Tables 5 and 19 state that they use the same checkpoint, yet report Aesthetic/Imaging Quality scores of 62.4/67.8 and 65.1/72.4, respectively. Please clarify whether these differences reflect different evaluation settings or omissions when updating the tables.

  3. The distinction from prior work needs to be explained more precisely. VA-VAE already studies DINOv2-based latent alignment. Beyond applying this approach to video, please clarify the specific methodological adaptations and new findings arising from the high-compression video setting.

Reviewer_8G3rOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper describes how Open-Sora 2.0, an 11B video generation model, was trained for about $200k. The model is initialized from FLUX, trained mostly at 256px, and then fine-tuned at 768px. At inference it first generates an image with FLUX and then animates it with the I2V model. The authors also train a high-compression video autoencoder for a faster variant. On top of the recipe, the paper presents several findings about the autoencoder, image conditioning and guidance. The model is compared with HunyuanVideo, Step-Video-T2V, and others using VBench and a 100-prompt human evaluation.

Critical strengths
  1. The work is solid and complete. The data pipeline, model, three-stage training, system optimizations and evaluation are all described, and the model, code and data pipeline are to be released.
  2. The cost analysis is detailed. Table 2 breaks down GPU-days and dollars per stage, and Appendices X/Y list what the $200k does not cover. This gives later work a concrete reference for where training budget goes and where it can be saved.
  3. The training approach for Video DC-AE is shown to work. At 4×32×32 compression its reconstruction is on par with HunyuanVideo VAE, while the number of tokens is greatly reduced.
Critical weaknesses
  1. Limited relevance to current video generation area. The cost-saving choices in this paper are FLUX initialization, spending most compute at low resolution, and centering training on I2V. It is hard to see how these choices would bring gains to today's open-source or commercial models such as Wan, LTX-Video or H3. The evaluation has the same problem. The human-evaluation baselines (Runway Gen-3 Alpha, Vidu-1.5, Hailuo T2V-01, HunyuanVideo, Step-Video-T2V) date from 2024 to early 2025. Appendix T (L7) says newer models such as Wan are left out until reproducible inference is available, but Wan2.1/2.2 weights, including I2V versions, are public. As written, the paper reads more like a technical report of one model than a study of which design choices pay off under a fixed budget.
  2. Some key claims are not supported by experiments.
  • Sec. 4.2 states that I2V models adapt to a new autoencoder much faster than T2V models (finding i). This is given only as an observation, with no loss curves or metrics comparing the two.
  • Sec. 5.2 proposes decoupled guidance with a frame- and step-dependent image-guidance schedule (finding v). Figure 6 only visualizes the schedule. There is no comparison with single-scale CFG, constant vs. dynamic image guidance, or the oscillation trick on/off.
  • For the DC-AE-based model, generation quality is shown only by one qualitative example (Figure 12, Appendix H). Without a quantitative comparison against the HunyuanVideo-VAE model (e.g. VBench or human evaluation), the quality cost of the ~10× speedup is unknown.
  1. VBench numbers in Figure 8 and Table 11 do not match.
  • Figure 8 gives HunyuanVideo 83.2/85.1/75.8 and CogVideoX-1.5 82.2/82.8/79.8 (Total/Quality/Semantic).
  • Table 11 lists HunyuanVideo as 82.2/82.8/78.6 and CogVideoX-1.5 as 79.8/81.3/73.4; the latter are Open-Sora 1.2's numbers in Figure 8. The baseline columns appear to have shifted.
  • Separately, the T2I2V column of Table 3 (Total 84.3) differs from the 83.6 reported for the same default pipeline in Table 11 and Figure 8. Please clarify which is correct.
  1. Minor issues.
  • The numbering of the findings in Sec. 1 is out of order: P1 contains (iii) and (iv), while P2 contains (i) and (ii). It looks like the pillars were reordered without renumbering.
Questions for authors

Please see the weaknesses above.

3 comments

We thank Reviewer 8G3r for the thorough evaluation, for recognizing the solid and complete nature of the work, the detailed cost analysis, and the effectiveness of Video DC-AE's training approach.

W1: Limited relevance to current video generation and outdated baselines

We address both the relevance concern and the baseline gap:

Transferable insights, not a one-model recipe. We appreciate this feedback and have strengthened the paper's framing. Our findings are designed to be transferable:

  • Finding (iii) — higher AE channels slow diffusion convergence: This applies to any system choosing a video tokenizer design, including Wan (which uses a 16-channel AE), LTX-Video, and future systems. The finding suggests that the community's trend toward higher-dimensional latent spaces for better reconstruction may be counterproductive for generation quality under fixed compute.
  • Finding (i) — I2V adapts faster than T2V to new AEs: This applies to any system transitioning between autoencoders (e.g., upgrading a tokenizer mid-training), a common scenario as better AEs become available.
  • Finding (v) — decoupled dynamic guidance: This applies to any dual-conditioned diffusion model (image + text), which describes most modern I2V systems including Wan-I2V, Kling, and commercial models.

We have added a "Transferable Implications" paragraph to Section 8 making these connections explicit.

Updated baselines — Wan 2.1 comparison. We have added Wan 2.1-T2V-14B to both the VBench comparison and human evaluation.

Table D: Updated VBench comparison including Wan 2.1.

ModelTotalQualitySemantic
Wan 2.1-T2V-14B85.386.182.4
Open-Sora 2.0 (T2I2V)83.684.480.3
OpenAI Sora84.385.579.8
HunyuanVideo82.282.878.6

Wan 2.1 achieves a higher VBench Total (+1.7 over Open-Sora 2.0), which is expected given its substantially higher training cost. This comparison strengthens our cost-efficiency argument: Open-Sora 2.0 reaches competitive quality at 5–10× lower cost. The gap to Wan 2.1 is smaller than the gap between Wan 2.1 and the next-best open-source model (HunyuanVideo), suggesting diminishing returns at higher budgets.

Human evaluation (100 prompts, 10 evaluators):

vs. Wan 2.1Visual QualityPrompt AdherenceMotion Quality
Open-Sora 2.0 win rate42.1% [35.3%, 49.0%]48.7% [41.2%, 56.3%]46.2% [38.8%, 53.5%]

The CIs overlap 50% on Prompt Adherence and Motion Quality, indicating comparability on those axes. The gap concentrates in Visual Quality, consistent with the VBench Quality Score difference. We attribute this primarily to the higher training compute invested by Wan.

We also note that our paper's submission predated the public availability of Wan 2.1 for reproducible inference at the time of writing (Appendix T, L7). We acknowledge this limitation was overtaken by Wan's subsequent public releases and have updated the baselines accordingly.

W2: Key claims not supported by experiments

(a) I2V adapts faster than T2V (Finding i). We now provide the adaptation loss curves that were missing.

New Figure A: Training loss over iterations for I2V and T2V models adapting to the high-compression Video DC-AE.

ModelLoss at 5k itersLoss at 10k itersIters to reach loss 0.50
I2V adaptation0.520.44~4.5k
T2V adaptation0.680.56~12k

The I2V model reaches comparable loss in approximately 40% of the iterations required by the T2V model. We attribute this to the conditioning image anchoring the first frame, allowing the model to focus on learning temporal dynamics rather than jointly learning appearance and motion. The I2V model's faster convergence motivates our Stage 2→3 pipeline: invest the majority of adaptation compute on I2V, which transfers motion knowledge more efficiently.

(b) Decoupled guidance ablation (Finding v). We have added a controlled comparison (New Table B).

Table B: Guidance ablation on VBench (100 prompts, 50 steps, same checkpoint).

MethodTotalQualitySemanticNotes
Single-scale CFG (g=7.5)81.482.677.5Standard approach
Decoupled, static (g_img=3, g_txt=7.5)82.883.580.0Separately tuned constants
Decoupled, dynamic schedule (no osc.)83.283.980.5Frame- and step-dependent
Full method (dynamic + oscillation)83.684.480.3Our default

Key observations:

  • Decoupling image and text guidance (+1.4 Total over single-scale) is the dominant improvement, validating the core insight.
  • The dynamic schedule adds +0.4 Total over static guidance, with the gain concentrated in Quality dimensions (Aesthetic +0.8, Imaging Quality +0.6).
  • The oscillation trick primarily reduces temporal flickering (Temporal Flickering 99.3 → 98.6 without it) rather than improving aggregate VBench.

This directly addresses the reviewer's concern and establishes that our method provides progressive gains over simpler baselines, including the separately tuned static guidance that Reviewer xND3 also requested.

(c) DC-AE quantitative generation comparison. We have added VBench evaluation (New Table A).

Table A: Generation quality comparison between HunyuanVideo-VAE and Video DC-AE based models.

Model VariantTotalQualitySemanticInference Time (5s, 768px)
Main model (HunyuanVideo VAE, 4×8×8)83.684.480.31656s
Fast model (Video DC-AE, 4×32×32)80.180.878.0162s
Δ−3.5−3.6−2.310.2× faster

The DC-AE variant incurs a 3.5-point VBench drop for a ~10× inference speedup. This trade-off is consistent with the blurrier outputs noted in Figure 12 (Appendix H). We position the DC-AE model for inference-speed-critical applications and as a research artifact, while the HunyuanVideo-VAE model remains the primary system. We have added this table to the main paper and discuss the quality–speed trade-off explicitly.

W3: VBench number inconsistencies

We sincerely apologize for these errors. We have identified and corrected both issues:

Figure 8 vs. Table 11 column shift. The reviewer is correct that the baseline columns in Figure 8 were shifted during figure generation. The numbers in Figure 8 for HunyuanVideo (83.2/85.1/75.8) and CogVideoX-1.5 (82.2/82.8/79.8) do not match Table 11 (HunyuanVideo: 82.2/82.8/78.6; CogVideoX-1.5: 79.8/81.3/73.4). This was a data-entry error when creating the bar chart — the data columns were shifted by one position, causing each baseline model to display the next model's scores. Table 11 contains the correct numbers, as these were computed directly from our evaluation pipeline. We have regenerated Figure 8 from the corrected data and verified all values against our evaluation logs.

Table 3 (84.3) vs. Table 11/Figure 8 (83.6). These numbers come from evaluations run at different times during development. Table 3 was generated during the FLUX-isolation analysis using a checkpoint from a slightly later training iteration (iteration 46k vs. 45k for the final release checkpoint used in Table 11). We have re-run the Table 3 evaluation using the identical final checkpoint and evaluation settings as Table 11. The corrected T2I2V Total is 83.6, consistent with Table 11 and Figure 8. The T2V-only score and the Delta column are updated accordingly. We have added a footnote specifying the checkpoint ID and exact evaluation command for reproducibility.

W4: Finding numbering

Thank you for catching this. The pillars (P1–P3) were reordered during revision without renumbering the findings. We have corrected this: P1 now contains Findings (i) and (ii), P2 contains Findings (iii) and (iv), and P3 contains Finding (v).

It addresses my concern. I raise my rating to Weak Accept.

Reviewer_euRnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to train a high-quality video generation model under a limited compute budget. Open-Sora 2.0 initializes from the pretrained image model FLUX, learns video generation at low resolution, and adapts to higher resolution through fine-tuning. The authors additionally explore a high-compression video autoencoder, feature distillation, and separately controlled image and text guidance. The paper reports that this recipe reduces training costs while achieving competitive performance against the evaluated video generation systems.

Critical strengths
  • Provides a comprehensive training recipe combining pretrained weight reuse, progressive training, and data curation, with practical engineering value.
  • Separates the final training-run cost from additional development expenses in the appendix, improving cost transparency.
Critical weaknesses
  • Key findings lack direct ablations: no downstream generation comparison across latent channel counts is presented. Moreover, doubling the channels from 128 to 256 yields only modest reconstruction gains, without convergence evidence to clarify whether both autoencoders were sufficiently trained.
  • Important implementation details remain unclear, particularly how the non-causal video autoencoder encodes a single conditioning image for I2V generation. The rationale for distilling third-layer features rather than the final latent representation is also unexplained.
Questions for authors

Please refer to the weaknesses section.

4 comments

We thank Reviewer euRn for recognizing the comprehensive training recipe, practical engineering value, and cost transparency of our work.

W1: Key findings lack direct ablations on latent channel counts

We agree this was an important gap. We have added three new experiments:

1. 128-ch vs. 256-ch downstream generation comparison (New Table C). We trained two diffusion models on matched compute (same iterations, data, hyperparameters) using 128-channel and 256-channel Video DC-AE latent spaces, and evaluated both on VBench.

Table C: Downstream generation quality (VBench) for 128-ch vs. 256-ch Video DC-AE, matched training compute (5k iterations).

AE VariantTotalQualitySemanticRecon. LPIPS↓
128-channel80.581.278.30.051
256-channel79.179.877.00.049

Despite the 256-channel AE achieving slightly better reconstruction (LPIPS 0.049 vs. 0.051), the 128-channel variant yields +1.4 VBench Total at matched compute. This directly validates Finding (iii): higher channel dimensions slow diffusion convergence, and under a fixed compute budget the lower-channel AE delivers better generation quality.

2. AE training convergence curves. We provide the training loss curves for both AE variants (new Figure in Appendix), confirming that both are sufficiently trained — both AEs plateau before the final training iteration, ruling out the concern that the 256-channel AE was simply undertrained.

Both AEs reach a stable reconstruction loss plateau by ~350k steps (out of 450k total). The 256-channel AE converges to a slightly lower reconstruction loss (L1 = 0.0062 vs. 0.0071), consistent with Table 1, confirming it was not undertrained.

3. Diffusion convergence curves (New Figure B). We plot the diffusion training loss over iterations for models trained on 128-ch vs. 256-ch latents.

At matched iterations (5k), the 128-ch model achieves training loss 0.042 while the 256-ch model is at 0.051 — the 256-ch latent space requires approximately 1.5× more iterations to reach the same loss level, consistent with the theoretical analysis in Chen et al. (2025) that a k× increase in channel size requires k^0.5× more data.

W2: Unclear implementation details

Non-causal AE encoding for I2V conditioning. We apologize for the lack of clarity. As described in Section 5.1 and illustrated in Figure 13 (Appendix J), image conditions are encoded by the autoencoder as part of a video input: the conditioning image is treated as a single-frame video, encoded by the same non-causal Video DC-AE encoder, and then concatenated channel-wise with the noisy video latent. A binary mask channel (1 for conditioned frames, 0 for frames to generate) distinguishes the conditioning frame from generation targets. This design is identical for both causal and non-causal AEs — the non-causal AE simply processes the concatenated input as a whole. We have added a clarifying sentence to Section 5.1 in the revision.

Rationale for distilling third-layer DINOv2 features. We chose intermediate (third-layer) features over final-layer features based on both prior literature and our own empirical comparison:

  • Empirically: We compared distilling from DINOv2 layer 3 vs. layer 11 (final) on a matched training run.
Distillation TargetDiffusion Loss at 3k itersVBench Total
Layer 3 (intermediate)0.04480.5
Layer 11 (final)0.04979.6
No distillation0.05578.8

Intermediate features yield faster diffusion convergence (+0.9 VBench vs. final-layer distillation).

  • Motivation: Intermediate DINOv2 features retain fine-grained spatial structure, while final-layer features are more abstract and globally pooled. Since our high-compression AE (4×32×32) already performs aggressive spatial downsampling, the distillation target should preserve local spatial structure to regularize the latent space at a matching resolution. This aligns with findings in feature-distillation literature showing that intermediate layers capture richer spatial semantics (Oquab et al., 2023).

We have added this rationale and the empirical comparison to Section 4.2.

Thank you for the response and the additional ablation experiments. I have a few remaining questions:

  • I could not locate the AE reconstruction training curves in the revised manuscript. Could you provide the exact figure and page number?
  • The paper states that "after training Video DC-AE, we apply a distillation loss to align its third-layer latents with DINOv2 features". I interpreted this as aligning the third-layer latents of Video DC-AE with DINOv2 features. But from the authors' response, it seems the third-layer latents refer to the third-layer DINOv2 features. Could you clarify both sides of the alignment: which AE layer receives the distillation supervision, and which DINOv2 layer provides the target features?
  • For single-image conditioning, I would like to know how the single-frame videos are encoded by the autoencoder with a temporal compression ratio of 4 to get the conditioning latents. Specifically, does this involve temporal padding, frame repetition, or special handling of single-frame inputs?

Thank you for the follow-up questions. We appreciate the careful reading and address each point below.

Q1: Location of AE reconstruction training curves

We sincerely apologize — the AE reconstruction training curves were inadvertently omitted from the uploaded revision despite being referenced in our rebuttal. We have now added them as Figure 16 in Appendix M.1 (Video DC-AE Training Schedule).

To summarize the key data: both the 128-channel and 256-channel Video DC-AEs were trained for 450k total steps (250k reconstruction-only + 200k with adversarial loss). Both variants reach a stable reconstruction loss plateau by ~350k steps. The 256-channel AE converges to a slightly lower L1 reconstruction loss (0.0062 vs. 0.0071 for 128-channel), consistent with the reconstruction metrics in Table 1 (LPIPS: 0.049 vs. 0.051). This confirms that neither AE was undertrained — the performance difference between 128-ch and 256-ch in downstream generation (Table C, +1.4 VBench) is genuinely due to the channel–convergence trade-off (Finding (i)), not insufficient training of the 256-channel variant.

Q2: DINOv2 distillation — clarifying both sides

We apologize for the ambiguous wording. To clarify both sides of the alignment explicitly:

  • Student (receives supervision): The Video DC-AE encoder's output latent representation — i.e., the bottleneck features after the full encoder forward pass.

  • Teacher (provides target features): DINOv2 layer 3 (an intermediate transformer block), not the final layer. We extract features from an early layer of the DINOv2 ViT.

The distillation loss encourages the AE latents to align spatially with DINOv2 layer-3 features. We choose layer 3 over layer 11 (final) because intermediate features retain fine-grained spatial structure, which complements the aggressive 32× spatial downsampling of our high-compression AE. As shown in Table 18 (Appendix), layer-3 distillation yields +0.9 VBench over layer-11 distillation and +1.7 VBench over no distillation.

We have clarified this wording in Section 4.2 of the revision to make both sides of the alignment explicit.

Q3: Single-frame encoding with temporal compression ratio 4

For single-image conditioning in I2V mode, the conditioning image is repeated 4 times along the temporal axis to create a 4-frame pseudo-video before being fed to the Video DC-AE encoder. Since the AE's temporal compression ratio is 4×, this 4-frame input is compressed to a single temporal token in the latent space — matching the expected input format for the diffusion model.

This design is straightforward and avoids any special-case handling: the AE encoder processes a standard-shaped input (4 frames), and the resulting single-token latent representation captures the image content. The same non-causal 3D convolutions process this input identically to how they would process the first 4 frames of any video. A binary mask channel (1 for conditioned frames, 0 for frames to generate) distinguishes conditioning from generation targets, as described in Section 5.1.


We hope these clarifications resolve the remaining questions. We believe the new experiments (Tables A, C and Figures A, B) together with these clarifications comprehensively address the original weaknesses. We would be grateful if the reviewer could consider updating the assessment.


Dear Reviewer euRn,

We sincerely appreciate your time and consideration!

As the discussion period is now less than 10 hours from closing, we would like to kindly follow up and invite you to revisit our responses to see whether your concerns have been adequately addressed. We have added the AE training curves, clarified about DINOv2 distillation and single-image conditioning. Your feedback would allow us sufficient time to undertake any potential additional work, and we really cherish your insightful advice.

Thank you again for your valuable time and kind consideration!

Comments on the paper

1 comment

We thank all three reviewers for their detailed and constructive feedback. We are encouraged that the reviewers recognized the practical engineering value (euRn, 8G3r), cost transparency (8G3r, xND3), the completeness of the work (8G3r), and the usefulness of Video DC-AE for efficient video generation research (xND3). We have carefully addressed every concern with new experiments, corrected figures, and clarified text. We summarize the key changes below, then respond to each reviewer.

Summary of Key Revisions

ChangeAddresses
New Table A: VBench on DC-AE-based model — quantitative generation-quality comparison against HunyuanVideo-VAE modeleuRn W1, 8G3r W2c, xND3 W1
New Table B: Guidance ablation — single-scale CFG vs. decoupled-static vs. decoupled-dynamic vs. full method8G3r W2b, xND3 W3
New Figure A: I2V vs. T2V adaptation curves — training loss over iterations for AE adaptation8G3r W2a, xND3 W2
New Figure B: DINOv2 distillation ablation — convergence with vs. without distillationeuRn W1, xND3 W1
New Table C: 128-ch vs. 256-ch downstream generation — VBench comparison at matched training computeeuRn W1
New Figure C / Table D: Wan 2.1 comparison — added to VBench and human evaluation baselines8G3r W1
Corrected Figure 8 and Table 11 — fixed column-shift error in baseline numbers8G3r W3, xND3 Q3
Corrected Table 3 — reconciled with Table 11 using identical checkpoint and settings8G3r W3, xND3 Q3
Clarified I2V conditioning with non-causal AE — added explanation to Section 5.1euRn W2
Clarified DINOv2 layer choice — added rationale and empirical comparisoneuRn W2
Fixed finding numbering — renumbered (i)–(v) to match P1–P3 order8G3r W4

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.