acceptodds
Under review as a conference paper at ICLR 2027

Generation-Aware Latent Compression for Efficient Video Generation

Abstract

Highly compressed video autoencoders offer an effective way to accelerate video diffusion models, as the Diffusion Transformer (DiT) operates on far fewer tokens. However, such autoencoders are challenging to train, since a higher compression ratio degrades reconstruction quality and recovering it requires more channels, which is known to slow the convergence of the DiT. The compressed latent also differs from the one the DiT was trained on, so the pretrained DiT must be either retrained from scratch or adapted at considerable cost. Compressing the autoencoder the DiT was trained with appears to preserve compatibility, yet optimizing it for reconstruction alone still shifts the latent away from the distribution the DiT has learned. To address this, we propose **GRACE**, a two-stage framework that compresses a pretrained video autoencoder while keeping it compatible with the pretrained DiT. Specifically, we keep a frozen base latent from the pretrained encoder and learn a residual latent for the information lost under stronger compression, while aligning the compressed latent with the pretrained one in the feature space of the frozen DiT so that the autoencoder is optimized for generation. We then adapt the DiT with lightweight fine-tuning and asymmetric denoising, where the base is denoised ahead of the residual. GRACE reduces the token count of Wan2.1-I2V-14B by and its latency by at , while matching the generation quality of the pretrained pipeline on VBench.

Primary area:
generative models
Keywords:
video generation, latent compression, autoencoder

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_NtG1OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an efficient video generation model. The main goal is to improve the token compression efficiency of existing video generation models, which consist of an autoencoder and a diffusion transformer (DiT). Specifically, it aims to make the latent more compact than that of the pretrained model, while preserving as much knowledge of the pretrained autoencoder and DiT as possible. The key idea is to use the pretrained VAE encoder to produce a base latent from a low-resolution version of the input video, and a residual encoder to produce a residual latent from the full-resolution input, so that reconstruction quality is preserved with making the latent distribution easier for the DiT to learn. To make the two latents work as designed, a generation-aware alignment loss and asymmetric denoising are introduced. The effect of each key component is verified through experiments, and the full model is compared with the pretrained anchor (Wan2.1), a single-latent baseline at the same compression ratio, and other high-compression autoencoder-based methods, showing comparable VBench scores to Wan2.1 with 8× fewer tokens and higher scores than the other methods.

Critical strengths

Below are the two main reasons I recommend acceptance.

S1: The dual-latent design is simple yet effective. Instead of retraining a new high-compression autoencoder, the paper reuses the pretrained VAE encoder on a low-resolution input to produce the base latent, which stays close to the latent distribution the pretrained DiT already knows, and adds a residual encoder on the full-resolution input to recover the missing details. This asymmetric design directly addresses the key trade-off the paper identifies, where reducing tokens degrades reconstruction and increasing channels degrades generation, and its effectiveness is clearly demonstrated in the experiments (Tables 2 and 3). I therefore consider the paper to propose a novel architecture (and training strategies for it).

S2: Compared with Wan2.1 (its pretrained anchor), the full model maintains the generation quality of the anchor with 8× fewer tokens, while speeding up generation by 11.1×. It also achieves better generation quality and lower latency than both the single-latent baseline and DC-VideoGen, the closest prior work (Table 2). The human evaluation, presented with a detailed setup, is consistent with the quantitative results (Appendix D.2). I believe this result is a clear step forward in efficient video generation.

Critical weaknesses

None

Questions for authors

There are no questions whose answers could materially affect my assessment.

1 comment
AuthorsOpenReview ↗

Response to Reviewer NtG1

We sincerely thank the reviewer for the positive assessment and for recognizing the simplicity and effectiveness of our design and the efficiency gains. In response to the other reviewers, we have further strengthened the paper with a fine-tuned uncompressed baseline, two additional backbones (Wan2.2-A14B and Cosmos-Predict2.5-14B), a higher compression setting (f16t16), a comparison with an additional alignment target, and analyses of the training recipe, as summarized in the General Response.

Reviewer_8ZtkOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GRACE, which increases spatiotemporal compression for video generation by modifying a pretrained video autoencoder and fine-tuning the original DiT. The method concatenates, along the channel dimension, base latents produced by a frozen encoder from downsampled videos and residual latents extracted by a trainable encoder from the original videos. During autoencoder training, intermediate features from a frozen DiT align the compressed and original representations. The DiT is then adapted using LoRA and other parameter updates, with the base latents denoised ahead of the residual latents. The experiments support the effectiveness of this approach.

Critical strengths
  1. The paper is clear and easy to follow, and the evidence supports the effectiveness of the full generation-oriented pipeline.
  2. With the same Wan2.1-14B backbone, 50 sampling steps, and 100 DiT evaluations, GRACE reduces I2V latency from 863.2 s to 77.7 s and T2V latency from 851.5 s to 75.8 s at 480×832×81
Critical weaknesses

A major concern is the substantial overlap between GRACE and DA-VAE (CVPR 2026), which seems not appear to be discussed or cited in the current submission. The two methods adopt highly similar architectural designs. Although GRACE differs in its spatiotemporal video compression, intermediate DiT feature supervision, and asymmetric denoising, Table 4 compares only no alignment, V-JEPA alignment, and DiT feature alignment, without including DA-VAE’s direct alignment of detail latents to base latents. Furthermore, ProMAG (ICCV 2025) has already explored preserving pretrained low-compression video representations while learning complementary information needed for higher temporal compression. VA-VAE (CVPR 2025) and DC-Gen (ECCV 2026) have also demonstrated the importance of generation-oriented representation alignment and compatibility with pretrained diffusion models.

Questions for authors

GRACE is reported to preserve the generation quality of pretrained Wan2.1 while achieving substantial acceleration. However, the evaluation does not sufficiently support this conclusion. GRACE undergoes additional DiT adaptation using Panda-70M and rewritten captions, whereas the original Wan2.1 baseline seems not to receive comparable additional fine-tuning. Therefore, the substantial increase in the T2V semantic score from 78.70 to 84.98 seems unfair.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 8Ztk (1/5): W1

Due to the character limit, our response is split into five parts: W1 (Parts 1–3) and Q (Parts 4–5).

W1. Relation to prior works and additional alignment comparison

We thank the reviewer for careful comparison with prior work, extending to recent related work, which encouraged us to revisit how GRACE relates to existing designs and where it differs.

We would first like to note that, regarding the architectural similarity, most of GRACE's gain comes from generation-aware alignment and asymmetric denoising rather than from the dual latent itself. The two components combined add 2.26 points on VBench-T2V and 1.22 on VBench-I2V, compared with 1.43 and 0.20 from the dual latent (Tab. 3, rows II–V).

We have revised Sec. 2 and Sec. 4.2 to state more precisely how the dual latent relates to prior designs. Below, we address the architectural design and the alignment in turn, and clarify the relation to each work the reviewer mentions.

1. Architectural design (CV-VAE, ProMAG, DC-AE 1.5, DA-VAE)

Our dual latent follows a line of autoencoder designs that keep a pretrained representation and learn additional information on top of it. The original submission already cites CV-VAE, ProMAG, and DC-AE 1.5, and the revised Sec. 4.2 now refers to them together with DA-VAE when introducing the dual latent. CV-VAE (Zhao et al., 2024) extends a pretrained image VAE to video, keeping the video latent consistent with the pretrained image latents of temporally subsampled frames, which original submission cites for keeping a new latent compatible with the pretrained pipeline. ProMAG (Mahapatra et al., 2025) increases temporal compression progressively, building each new stage on the frozen previous one along the temporal axis, which original submission cites for progressive temporal compression. Separating the latent channels by role also appears in DC-AE 1.5 (Chen et al., 2025c), where the front channels capture structure and the later channels capture detail, while compressing only in space. The original submission notes that this separation helps generation and cites DC-AE 1.5 when introducing the dual latent. DA-VAE (Cai et al., 2026), which we have now added to the related work (Sec. 2, Tab. 4), follows the same lineage for images, compressing only in space.

Within this line of work, our dual latent compresses a pretrained video latent further in both time and space (from f8t4 to f16t8) at once, so its residual carries the motion between frames as well as the spatial detail that the base misses (App. F.1, Fig. F.5). This leaves more in the residual for the pretrained DiT to adapt to and generate, which our two components (generation-aware alignment and asymmetric denoising) are designed to address. Compressing both axes also lets GRACE use fewer tokens and run faster, although the temporal axis is known to be harder to compress than the spatial one (Tian et al., 2025). DC-Gen (He et al., 2026), built on the same Wan2.1, uses the same number of tokens per frame but does not compress time further, and runs 5.2× faster, against 11.1× for GRACE (Tab. 2).

Asymmetric denoising builds on this channel split by denoising the base ahead of the residual. It adds 1.13 to the VBench-T2V total and 0.35 to the VBench-I2V total (Tab. 3, rows IV and V), with T2V gains concentrated on spatial relationship, color, and scene (please refer to our response to Q below, Part 5).

AuthorsOpenReview ↗

Response to Reviewer 8Ztk (2/5): W1 (continued)

W1 (continued)

2. Alignment (VA-VAE, DC-Gen, DA-VAE)

We thank the reviewer for the careful reading of our alignment experiments, which gave us the opportunity to take a closer look at how each alignment target affects the latent.

As the reviewer notes, our alignment differs from prior work in that it supervises the intermediate features of the frozen DiT. The reviewer also points out that DC-Gen and VA-VAE (Yao et al., 2025) have shown the importance of generation-oriented alignment and compatibility with pretrained diffusion models. Our paper also credits this motivation to prior work (Sec. 2.2, lines 145–149, and Sec. 4.2, lines 277–279) and does not present it as new, while stating how our alignment differs (Sec. 2.2, lines 150–152). The rest of this section clarifies this difference and compares our alignment with DA-VAE's, which we have added to Tab. 4 as the reviewer suggested.

The approaches differ in what they use to align the latent:

  • DC-Gen trains its autoencoder for reconstruction alone and uses the pretrained DiT only afterward, to adapt the DiT (Sec. 2.2, lines 148–149).
  • VA-VAE aligns the latent to an external vision model, and DA-VAE aligns it to the latent of a downscaled image. Neither uses the pretrained generator or the original uncompressed latent.
  • Ours uses both the pretrained DiT and the original uncompressed latent to regularize the autoencoder. It matches the frozen pretrained DiT’s response to the compressed latent with its response to the original uncompressed latent, noised at the same timestep and conditioned on the same caption as in generation (Eqs. 8–9), which is why we call it generation-aware.

To compare these approaches directly, Tab. 4 already included alignment to an external vision model as in VA-VAE, using V-JEPA 2.1 features, and the submission compares with DC-Gen on the same backbone and training budget in Tabs. 1 and 2. In the revision, we also adapt our baseline with the full DC-Gen recipe, which scores nearly the same as the original baseline (86.42 vs. 86.44, App. E.3), still below GRACE.

The table below adds the base latent as an alignment target for the residual, following DA-VAE, and we have also added it to Tab. 4 of the revision.

We have added the implementation details of both alternative targets, including the loss and its weighting, to App. C.3.

Alignment targetPSNR ↑rFVD ↓Quality ↑Semantic ↑T2V-Total ↑
None33.1013.8984.7778.6583.55
V-JEPA 2.1 features32.6113.9384.9880.0483.99
Base latent (added)\color{#C65D00}{\text{Base latent (added)}}30.5418.4884.8779.8983.87
Pretrained DiT features (ours)32.6313.5385.2282.5584.68
Asymmetric denoising is disabled in all rows, as in Tab. 4.

We discuss each target below.

Aligning to V-JEPA 2.1 features keeps reconstruction but helps generation less than ours. This alignment takes the V-JEPA 2.1 features of the full-resolution video as its target. Reconstruction changes little, from 33.10 dB PSNR without alignment to 32.61. The semantic score rises from 78.65 to 80.04, while aligning inside the pretrained DiT raises it far higher, to 82.55. As noted in the submission (lines 474–476), vision foundation models provide useful semantic structure in general. In our setting, however, the pretrained DiT was trained on the original latent. The compressed latent has to match what this DiT already knows, not what an external model sees.

AuthorsOpenReview ↗

Response to Reviewer 8Ztk (3/5): W1 (continued)

W1 (continued)

Aligning to the base latent costs the most reconstruction and helps generation less than ours. This alignment uses the base latent of the spatially downsampled and temporally subsampled video as the target for the residual latent, so the residual comes to resemble the base (Fig. F.7). This likely makes the latent easier for the pretrained DiT to denoise, and the quality and semantic scores rise slightly, from 84.77 to 84.87 and from 78.65 to 79.89, both below ours (85.22 and 82.55). However, the DiT is then trained on a residual that mostly repeats the base, and may not learn to generate the spatial detail and inter-frame motion that the base misses. This also shows in reconstruction. PSNR falls from 33.10 dB to 30.54, and rFVD, which also reflects motion, rises from 13.89 to 18.48, against 32.63 and 13.53 with ours.

Although better reconstruction does not always lead to better generation (Tab. 1), reconstruction still sets an upper bound on generation, since the decoder cannot restore what the latent does not carry. Aligning to the base latent lowers this upper bound, because the residual is pulled toward the base and carries less detail and motion. This gap is likely to grow at higher compression ratios, since the base would be computed from a more heavily reduced video and the residual would have to carry more of what the base misses, while our target remains the original uncompressed latent at any ratio.

Our alignment is instead computed with the pretrained DiT itself. It compares how the DiT responds to the compressed and the original latents, which allows the DiT to make better use of the compressed latent. Neither alternative target, V-JEPA 2.1 features or the base latent, involves the pretrained DiT, and both help generation less than ours, which points to the benefit of generation-aware alignment.

The PCA visualization (App. F.1, Fig. F.7, Figs. I.18–I.20, and supplementary videos) shows how each target changes the residual latent, which helps interpret the results above (Tab. 4). Without alignment, noise dominates the residual, which shows little of the scene. Aligning to the base latent brings the residual close to the base, so that the two show nearly the same structure, which is consistent with the residual repeating the base as discussed above. Aligning to V-JEPA 2.1 features makes the residual coarser, following the large regions of the scene. Our alignment keeps the residual organized by objects, while its components still differ from those of the base.

We also provide the PCA as videos in the supplementary material, which make the changes across frames easier to follow. Each latent frame encodes 8 input frames, so the components change once every 8 frames, and their structure is easier to see when the video is viewed zoomed out.

3. Related Revisions

  • Related work (Sec. 2, Sec. 4.2). We added DA-VAE to Sec. 2 and cite it together with CV-VAE, ProMAG, and DC-AE 1.5 where the dual latent is introduced.
  • Alignment comparison (Tab. 4, Sec. 5.5, App. C.3, Fig. F.7, supplementary videos). We added DA-VAE's alignment to the base latent, with its implementation details and a PCA visualization of the residual under each target, with more examples in Figs. I.18–I.20 and as videos in the supplementary material.
Reviewer_8tpbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper accelerates video Diffusion Transformers (DiTs) by performing post-hoc latent compression on an existing pretrained pipeline (Wan2.1-VAE f8t4 + Wan2.1-14B). Standard video 3DVAEs use 8x spatial and 4x temporal downsampling (f8t4). Because DiT self-attention scales quadratically with sequence length, compressing latents to f16t8 (an 8x token reduction) is critical for efficiency. However, f16t8 degrades video reconstruction unless channel capacity expands, causing slow convergence for pretrained DiTs.

To compress latents without retraining from scratch, the method introduces three core techniques:

  • Dual-Latent Representation: Concatenates a frozen base latent (extracted by the pretrained VAE from a downsampled input) with a learned residual latent (produced by a trainable residual encoder from full-resolution video).
  • Generation-Aware Feature Alignment: Fine-tunes the autoencoder using an alignment loss that matches early-layer intermediate DiT features of noised compressed latents against noised pretrained latents via cosine similarity and adaptive loss weighting.
  • DiT Adaptation with Asymmetric Denoising: Adapts the DiT using LoRA on transformer blocks and fully fine-tuned channel projection layers, employing a single-pass sampling schedule where the base component is denoised ahead of the residual.

Evaluated on Wan2.1-I2V-14B, the approach achieves 11.1x (480p) and 15.6x (736p) latency reductions while maintaining overall VBench scores comparable to the uncompressed baseline.

Critical strengths
  • Significant efficiency gains: 11.1× speedup at 480p and 15.6× at 736p by reducing DiT input tokens from 32.8k to 4.3k.
  • Strong quantitative evaluation and empirical analysis: The paper includes thorough ablations on loss targets, denoising order, and convergence loss curves, analyzing the effects of the key design choices.
  • The key idea of "reusing the pretrained pipeline to guide its own latent compression" is practical and well-motivated. The dual latent representation and architectural design to support that is technically sound and validated with ablations.
Critical weaknesses
  • The T2V results outperform the uncompressed version by +1.9, driven by +6.3 semantic, suggesting the compressed latent somehow improves prompt following. It is not entirely clear whether that is the case or why it should be the case. One possibility is that the Stage 2 fine-tuning was performd on datasets with GPT-4o-rewritten captions and high-capacity adaptation. The key missing baseline is the uncompressed Wan2.1 fine-tuned under the identical Stage 2 recipe.
  • There's a somewhat disagreement between the VBench result and the user study on I2V: while the experiment claims the proposed method matches uncompressed Wan2.1 on VBench-I2V, the user study shows a preference for Wan2.1 in visual quality and temporal consistency, both are important perceptual quality aspects. This could indicate that the resulting latent compression incurs visual degradation on fine details and motion that automated VBench benchmarks fail to capture and will need further clarification and explanation.
  • The method is demonstrated on a single pipeline (Wan2.1) and a single compression setting. There is no study of other ratios or another backbone (e.g., HunyuanVideo, CogVideoX, Cosmos), so the generality of the proposed method and recipe is unclear.
Questions for authors
  • Can you report the uncompressed Wan2.1-14B fine-tuned under the same Stage 2 recipe?
  • Was the single-latent baseline adapted with the same LoRA rank, steps, shift sampling, and zero-initialization as GRACE? If not, how does the baseline score with GRACE's recipe?
  • Can you please help clarify and explain the gap in the user study where human raters prefer uncompressed Wan2.1 over GRACE in I2V visual quality, whereas VBench-I2V total scores seems to disagree?
  • How well can the proposed method be extended to higher compression ratio (e.g. f32 and/or t16)? How far can it go with the proposed recipe? And are there directions to extend beyond such limit?
8 comments
AuthorsOpenReview ↗

Response to Reviewer 8tpb (1/6): W1/Q1

Due to the character limit, our response is split into six parts: W1/Q1 (Parts 1–2), Q2 (Part 3), W2/Q3 (Part 4), and W3/Q4 (Parts 5–6).

W1 / Q1. Additional fine-tuning of the uncompressed baseline

We appreciate the reviewer's careful reading of our T2V results. As requested, we fine-tuned the uncompressed Wan2.1-14B under the identical Stage 2 recipe. This gives a stronger reference than the original model, and GRACE still scores higher on the VBench-T2V total (85.81 vs. 84.62) while using 8× fewer tokens. The gap in the semantic score becomes smaller but remains (84.98 vs. 81.84). We would like to note that we report this as an observation rather than as a claim of better prompt following, and we discuss possible reasons in Part 2.

1. Setup

The overall setup matches GRACE as follows.

  • Training data. The same Panda-70M videos and the GPT-4o-rewritten captions as GRACE.
  • Training setup. The same hyperparameters as GRACE, including the LoRA rank and effective batch size (App. B).
  • Compute. The same 38.5 H200 GPU days as the full pipeline of GRACE, since our question is whether the gain holds under the same training budget. GRACE spends 8.5 of these on its autoencoder and 30 on its DiT (Tab. G.12), whereas the uncompressed model needs no autoencoder training and spends the entire 38.5 on its DiT, more than GRACE. Each training step takes about 4× longer without compression, so it completes 1,320 steps within this budget.
  • Evaluation. The same expanded prompts and resolution (480×832×81) as GRACE.

2. Results

The table below reports the VBench-T2V scores of the three models at 480×832×81.

DimensionWan2.1-14B (original)Wan2.1-14B (fine-tuned, same budget)GRACE
Quality
Subject consistency96.4897.7096.76
Background consistency98.1498.8998.70
Temporal flickering99.2099.4399.42
Motion smoothness98.7199.3399.36
Dynamic degree56.9449.4462.50
Aesthetic quality70.4169.0168.66
Imaging quality67.5568.0267.65
Semantic
Object class92.0990.9894.15
Multiple objects79.5084.1485.67
Human action97.0099.0099.00
Color89.6888.6997.29
Spatial relationship78.9484.0291.16
Scene47.3149.7157.49
Appearance style22.5724.7324.79
Temporal style23.3425.4225.52
Overall consistency25.6326.5125.76
Quality score85.2485.3286.02
Semantic score78.7081.8484.98
Total score83.9384.6285.81
  • Quality. Fine-tuning leaves the quality score nearly unchanged (85.24 to 85.32), and GRACE scores 86.02.
  • Semantic. Fine-tuning raises the semantic score of the uncompressed model from 78.70 to 81.84, and GRACE still scores 3.14 points higher (84.98), mostly in spatial relationship, scene, and color (per-dimension scores above). We report this gap as an observation rather than as a claim of better prompt following, and discuss possible reasons in Section 3 below (Part 2). Fig. D.4, Figs. I.21–I.22, and the supplementary videos show samples selected from prompts where the VBench semantic score of GRACE is clearly higher, to illustrate what these scores capture, with both models using the same prompt and seed.
  • Total. GRACE scores higher than the fine-tuned model under the same compute (85.81 vs. 84.62), while using 8× fewer tokens and running 11.1× faster.

We have added this model to Tab. 2 and revised our claim accordingly: GRACE matches or exceeds the uncompressed model trained under the same compute, rather than exceeding the original pretrained model.

AuthorsOpenReview ↗

Response to Reviewer 8tpb (2/6): W1/Q1 (continued)

W1 / Q1 (continued)

3. Discussion on the T2V semantic gain

We hypothesize that the gain in the T2V semantic score over the fine-tuned Wan2.1 comes from the compact latent together with our two components, generation-aware alignment and asymmetric denoising, both of which allow the pretrained DiT to make better use of this latent.

Prior work has observed that a tokenizer can learn more semantic representations as its latent becomes more compact. Yu et al. (2024) report that "semantics emerges with compact latent space." In their tokenizer, linear-probing accuracy improves as the number of tokens decreases, even though reconstruction gets worse. Since the DiT generates the prompted content in this latent, concentrating semantic content in the latent may make the prompt easier to follow.

However, carrying more semantics in the latent does not by itself lead to better generation in our setting, since our DiT was pretrained on the uncompressed latent. Without our components, training on the same data as GRACE leaves the semantic score at 77.75 for the single-latent baseline and 78.65 for the dual latent (Tab. 3, rows II and III), below both the original Wan2.1 (78.70) and the fine-tuned one (81.84). Our two components allow the pretrained DiT to make better use of the compressed latent, and the semantic score rises to 84.98 (Tab. 3, row V). We discuss the contribution of each component below.

First, generation-aware alignment raises the semantic score from 78.65 to 82.55 (Tab. 3, rows III and IV). The gain spreads across most semantic dimensions and is relatively larger on multiple objects (whether all prompted objects appear) and overall consistency (how well the whole video matches the prompt). The principal components of the aligned latent also tend to follow object regions and keep their structure across frames (Sec. 4.2, lines 307–309, Fig. 3, and Figs. I.16–I.17).

Second, asymmetric denoising raises the semantic score further, from 82.55 to 84.98 (Tab. 3, rows IV and V). Its gain is concentrated on spatial relationship (whether objects are in the prompted relative positions), color (whether objects have the prompted colors), and scene (whether the prompted scene appears). The base appears to carry such coarse properties, as perturbing it breaks the scene layout (Fig. F.5). A possible explanation is that, with both parts at the same noise level, the DiT must handle the unfamiliar residual while deciding the layout. Keeping the base ahead avoids this, so that the semantic content of the compact latent can be used more fully.

Examples on these dimensions are shown in Fig. D.4, Figs. I.21–I.22, and the supplementary videos, selected from prompts where the VBench semantic score of GRACE is clearly higher than that of the fine-tuned Wan2.1, to illustrate what these scores capture, with both generated from the same prompt and seed.

We leave a more thorough analysis of this gain to future work. We thank the reviewer for raising this point, which led us to a clearer understanding of where the gain comes from.

Reference. Yu et al., An Image is Worth 32 Tokens for Reconstruction and Generation, NeurIPS 2024.

4. Related revisions

  • T2V claim (Sec. 5.3). We now compare GRACE with a stronger reference, the uncompressed model fine-tuned under the same recipe and compute budget (semantic score 78.70 to 81.84). GRACE remains higher than this reference (84.98), and we state that GRACE matches or exceeds it, rather than exceeding the original pretrained model.
  • Discussion on the semantic gain (Sec. 5.3 and 5.5, App. D.3). We added the per-dimension comparison and the discussion above.
  • Tab. 2. We added the fine-tuned Wan2.1 on both T2V and I2V.
  • Qualitative comparison (Fig. D.4, Figs. I.21–I.22, supplementary videos). We added examples where GRACE scores clearly higher than the fine-tuned Wan2.1 on the VBench semantic dimensions, generated with the same prompt and seed.
AuthorsOpenReview ↗

Response to Reviewer 8tpb (3/6): Q2

Q2. Clarification on the single-latent baseline's training recipe

We thank the reviewer for carefully checking the baseline's training setup.

1. Clarification on the baseline's training recipe

As described in Appendix B, the single-latent baseline is adapted with the same DiT recipe as GRACE, using the same data, captions, LoRA rank, number of steps, shift sampling (s∈[2,5]s \in [2, 5]), and initialization (B.1, B.2). Its autoencoder also follows GRACE's design and training setup (A.1, A.2).

2. Correction to Appendix B.2

Appendix B.2 described the baseline unclearly on two points. First, it stated that the baseline leaves "no offset or shift to sample". In fact, the baseline samples ss from the same range as GRACE, and only the offset does not apply to a single latent. Second, it described the initialization differently from B.1. We have corrected both points in B.2 and thank the reviewer for pointing them out. The single-latent baseline therefore follows GRACE's recipe except for asymmetric denoising, which requires the base and residual parts.

3. Additional ablation on initialization and DC-Gen adaptation

To test whether the gap between GRACE and the single-latent baseline holds under other recipes, we also trained the single-latent baseline in two alternative ways, keeping the autoencoder, data, captions, LoRA rank, batch size, and number of steps the same as for the reported baseline.

ModelVBench-I2V
Single-latent baseline (reported, Tabs. 1–3)86.44
Single-latent baseline, w/ alternative initialization86.25
Single-latent baseline, w/ DC-Gen adaptation86.42
GRACE w/o asymmetric denoising (Tab. 3, row IV)87.51
  • Alternative initialization. We adapt the single-latent baseline with a different initialization strategy for the weights added to the input and output projections. With random initialization, the baseline reaches nearly the same score over the full schedule (86.25 vs. 86.44 on VBench-I2V), with only slightly slower convergence early in training.
  • DC-Gen adaptation. We also adapt the single-latent baseline with the full DC-Gen recipe, as for the autoencoders in Tab. 1. This gives nearly the same score (86.42 vs. 86.44).

Under every recipe, the baseline stays between 86.25 and 86.44, below GRACE even without asymmetric denoising (87.51). The gap to GRACE therefore comes from our components, the dual latent, generation-aware alignment, and asymmetric denoising (Tab. 3), rather than from the baseline's recipe.

4. Related Revisions

  • Baseline description (App. B.2). We corrected the description of shift sampling and initialization for the single-latent baseline, consistent with B.1.
  • Recipe ablation (App. E.3). We added the single-latent baseline with an alternative initialization and with the DC-Gen adaptation (Tab. E.11), and refer to it in Sec. 5.1.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their thoughtful and constructive feedback. We are encouraged that the reviewers found the paper clear and easy to follow (8tpb, 8Ztk, NtG1), the idea of reusing the pretrained pipeline to guide its own latent compression practical and well motivated (8tpb), the design simple yet effective (NtG1), and the efficiency gains significant (8tpb, 8Ztk, NtG1). Below we summarize the main revisions, which address the shared concerns, and list the clarifications and corrections.

Changes in the revised paper are color-coded by reviewer: blue for Reviewer 8tpb, orange for Reviewer 8Ztk, and green for changes that address both. Other corrections are shown in magenta.

Since several analyses and qualitative results do not fit in this response, we would be grateful if the reviewers could refer to the revised paper and supplementary videos for the full details.

Summary of added experiments and revisions

Added experiments

  1. Fine-tuned uncompressed Wan2.1 (8tpb, 8Ztk). We added the uncompressed Wan2.1-14B fine-tuned under the identical Stage 2 recipe and the same compute budget to Tab. 2. This gives a stronger reference than the original model (VBench-T2V total 83.93 to 84.62), and GRACE still scores higher (85.81) while using 8× fewer tokens, and stays within 0.14 of it on the VBench-I2V total. We revised our T2V claim accordingly (Sec. 5.3) and added a discussion of where the remaining gain comes from (Sec. 5.3 and 5.5, App. D.3).

  2. Generality across backbones (8tpb). We added results of GRACE on two additional backbones, Wan2.2-A14B, which uses a two-expert denoiser, and Cosmos-Predict2.5-14B, which comes from a different model family, each compared with the uncompressed model fine-tuned under the same Stage 2 recipe (Sec. 5.6, App. D.4).

  3. Results at a higher compression ratio (8tpb). We added results for f16t16, which doubles the temporal compression of f16t8 and lowers the VBench-I2V total by only 0.64 with 1.9× fewer tokens, and a discussion of directions to extend beyond it (Sec. 6, App. D.5).

  4. Alignment comparison (8Ztk). We added DA-VAE's alignment to the base latent to the comparison of alignment targets in Tab. 4 (Sec. 5.5, App. C.3, Fig. F.7).

  5. Additional ablation on the single-latent baseline (8tpb). We added the results of the single-latent baseline trained with an alternative initialization and with the DC-Gen adaptation (App. E.3). The gap to GRACE remains in every case.

Other related revisions

  1. I2V results and user study (8tpb). We added the user study results to the main text (Sec. 5.3), where we now relate it to the corresponding VBench dimensions, and revised our I2V claim to refer to the VBench total score.

  2. Related work (8Ztk). We added DA-VAE to the related work (Sec. 2) and now relate the dual latent to the prior designs it builds on, including DA-VAE, in Sec. 4.2.

  3. Additional qualitative results (8tpb, 8Ztk). We added 31 videos to the supplementary material. After extracting the ZIP, open index.html and select a Rebuttal section from the menu.

    • (1) Fine-tuned Wan2.1 vs. GRACE on T2V (6 videos), selected from prompts where the VBench semantic score of GRACE is clearly higher than that of the fine-tuned Wan2.1, as in Fig. D.4 and Figs. I.21–I.22.
    • (2) Fine-tuned Wan2.2-A14B vs. GRACE on I2V (7 videos).
    • (3) Fine-tuned Cosmos-Predict2.5-14B vs. GRACE on I2V (7 videos).
    • (4) Wan2.1, GRACE at f16t8, and GRACE at f16t16 on I2V (7 videos).
    • (5) PCA of the residual latent under each alignment target, as in Fig. F.7 and Figs. I.18–I.20 (4 videos).

    To fit the size limit, all videos, including those from the original submission, are re-encoded with the same setting for every model (H.264, CRF 23).

  4. Limitations (8tpb). The submission already noted that GRACE renders small regions, such as distant faces, with less detail (App. J). We now also point to such cases in the samples from the comparison with the fine-tuned Wan2.1.

Clarifications and corrections

  • Eqs. 8–9. The alignment is computed with the caption embedding as the text condition of the frozen DiT, together with the first frame for I2V, which was omitted from the equations. We have corrected them.
  • Appendix B.2. B.2 stated that the single-latent baseline leaves "no offset or shift to sample". In fact, the baseline samples the shift from the same range as GRACE, and only the offset does not apply to a single latent. B.2 also described the baseline's initialization differently from B.1. We have corrected both points.
  • Tab. 2. We made the I2V quality score of the single-latent baseline consistent with Tab. 3.
AuthorsOpenReview ↗

Summary of the Author-Reviewer Discussion

We sincerely thank the reviewers for their continued engagement and the AC for coordinating the review process. We briefly summarize the outcome of the discussion.

  • Reviewer NtG1 recommended clear acceptance, noting no critical weaknesses.
  • Reviewer 8tpb raised concerns on the fine-tuned uncompressed baseline, the training recipe of the single-latent baseline, the agreement between the user study and VBench, and the generality across backbones and compression ratios. The reviewer confirmed that these concerns have been thoroughly addressed and keeps the recommendation for acceptance.
  • Reviewer 8Ztk raised concerns on the relation to prior work, an additional alignment comparison, and the fairness of the T2V comparison. The reviewer found that the added clarification and experiments show the effect of generation-aware alignment and asymmetric denoising, and raised the recommendation from Weak Reject to Weak Accept.

The revised paper reflects the changes discussed, as summarized in the General Response.

We are truly grateful to all reviewers for their constructive feedback, which has made the paper considerably stronger

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.