acceptodds
Under review as a conference paper at ICLR 2027

Enhancing Autoregressive Video Generation via Representation Adversarial Distillation

Abstract

Few-step autoregressive video generation enables efficient streaming synthesis, but errors introduced in early temporal blocks are reused as context and can propagate through subsequent rollouts, leading to detail degradation, structural drift, and unstable motion. Existing distribution matching distillation (DMD) primarily aligns student and teacher distributions in diffusion latent space, but provides no direct supervision over the perceptual quality of decoded videos. We introduce Radian, a representation-space adversarial distillation framework that complements on-policy DMD with real-data adversarial supervision in the feature space defined by a frozen visual foundation model (VFM). During training, Radian sparsely decodes frames from autoregressive student rollouts, extracts multi-level visual representations, and applies lightweight discriminator heads to distinguish generated outputs from real video frames. The DMD objective anchors the student to the pretrained teacher, while the representation-space adversarial objective supplies complementary perceptual and semantic gradients that promote high-quality modes. These additional components are discarded after training, leaving the generator architecture and inference-time denoising budget unchanged. Experiments on Wan2.1-1.3B cover four-step chunk-wise, one-step frame-wise, and minute-long autoregressive generation. Our method achieves a VBench Total of and a VideoAlign Total of under four-step generation, and improves VBench-Long from to over Rolling Forcing while using fewer denoising steps. Controlled comparisons across image, video, and diffusion representations further indicate that the choice of representation spaces induces distinct adversarial signals, and external VFM gradients complement DMD more effectively than adversarial supervision derived from diffusion-internal features.

Primary area:
generative models
Keywords:
Autoregressive Video Generation, Adversarial Distillation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_bTk2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Radian, a post-training framework that augments on-policy distribution matching distillation with representation-space adversarial supervision for autoregressive video generation. Generated rollouts are sparsely decoded into RGB frames and processed by a frozen visual encoder, with lightweight discriminator heads trained against real video frames. The framework uses ODE initialization, discriminator calibration, and joint DMD-adversarial refinement, without changing the generator's inference architecture. Experiments cover four-step chunk-wise generation, one-step frame-wise generation with first-frame enhancement, and minute-long rolling generation on Wan2.1-1.3B. The paper also compares image, video, and diffusion-internal discriminator representations, reporting improvements on VBench, VideoAlign, and VBench-Long.

Critical strengths
  1. The method provides a practical way to incorporate real-data perceptual supervision into an autoregressive distilled generator without increasing inference cost. Combining a teacher-based objective with adversarial feedback on decoded outputs is well motivated, and the reported results suggest that this combination can improve perceptual quality under small denoising budgets.

  2. The empirical evaluation covers several generation regimes and a useful range of discriminator representations. Comparisons across image encoders, video encoders, diffusion features, encoder sizes, and feature depths provide evidence about practical design choices. The use of matched evaluation prompts and 15 inference seeds, together with qualitative comparisons, strengthens the empirical investigation.

Critical weaknesses
  1. The methodological novelty beyond existing adversarial distillation is somehow limited. ADD already combines teacher distillation with adversarial supervision through frozen visual features and lightweight multi-level discriminator heads. Radian primarily adapts this recipe to on-policy autoregressive video distillation, with sparse decoding and staged optimization. While this adaptation has practical value, the paper does not sufficiently establish which video-specific design choices are essential or what additional insight distinguishes the method.

  2. The method exhibits substantial sensitivity to the external encoder and feature configuration, limiting confidence in the robustness of the proposed recipe. In Table 2, replacing DINOv2 with V-JEPA 2.1 reduces Dynamic Degree from 0.8056 to 0.5991 and VideoAlign Total from 0.8033 to 0.5892. Tables 3 and 4 also show considerable variation across encoder sizes and feature-layer selections.

Questions for authors
  1. Table 2 shows that replacing DINOv2 with V-JEPA 2.1 substantially reduces Dynamic Degree from 0.8056 to 0.5991, while VideoAlign Motion Quality also decreases from 0.2094 to 0.1275. Why does a video encoder with explicit temporal modeling produce this behavior compared with the image-based DINOv2 encoder? Does this reflect a bias toward low-motion outputs, or differences in input sampling, discriminator design, and effective adversarial gradient strength? Does the gap persist after separately calibrating the adversarial weight and optimization settings?

  2. DMD can exhibit mode-seeking behavior which would cause motion degradation, while the default frame-wise adversarial objective does not explicitly constrain temporal dynamics. This raises the concern that prolonged joint training could improve individual-frame quality at the expense of motion diversity or action progression. Does Radian exhibit motion collapse or other quality regressions when trained beyond the reported schedule?

I'm willing to raise my score if the authors satisfactorily address these concerns, particularly the sensitivity to encoder choice and the stability of motion quality and diversity under prolonged training.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

Thank you for your comments. Here are our responses to your comments:

Q1: Novelty and Video-Specific Design

We thank the reviewer for this comment and agree that frozen visual features and lightweight multi-level discriminator heads are not entirely new. Our implementation draws on StyleGAN-T and shares similarities with ADD. More precisely, our contribution lies in adapting this approach to autoregressive video distillation through on-policy conditioning, sparse decoding, and staged optimization. In addition, our pair-rank mechanism distributes the generator and VAE decoder/VFM/discriminator heads across two GPUs, forming one logical training unit through cross-device gradient propagation to alleviate the memory pressure of long-video training. Our method also enables video VFMs to extract features from an entire clip while preserving the computational graph, allowing GAN gradients to train the generator through clip-level supervision. We will revise our novelty claims and clarify the existing limitations.

Q2: V-JEPA 2.1 Encoder and Hyperparameter Sensitivity

Regarding whether the performance gap between DINOv2 and V-JEPA 2.1 arises from the encoders themselves or mismatched hyperparameters, we conducted the following analyses. All hyperparameter and architecture comparisons were evaluated using checkpoints trained for 800800 steps.

(1) Different Training Stages: Under the configuration retained in our paper, V-JEPA's Dynamic Degree decreases from 0.62690.6269 at step 200200 to 0.61080.6108 at step 500500 and 0.59910.5991 at step 800800, remaining within a relatively low range.

(2) Hyperparameter and Architecture Calibration: We tested different GAN weights (1.01.0, 0.50.5, 0.10.1, 0.020.02), discriminator learning rates (5e−65e-6, 1e−51e-5, 3e−53e-5), generator learning rates (5e−75e-7, 2e−62e-6, 5e−65e-6, 1e−51e-5), and three feature aggregation designs: multi-level discriminator heads, spatial average pooling followed by temporal convolutions and multi-level heads, and a Q-Former global head that summarizes the entire video through learnable queries. Other settings, including the G:D update frequency (1:51:5), batch size (88), and training duration (800800 steps), were kept consistent. We ultimately selected a GAN weight of 0.020.02, D LR of 1e−51e-5, G LR of 2e−62e-6, and multi-level discriminator heads. Detailed results are shown in the table below.

(3) Gradient Strength: V-JEPA training logs show that the weighted GAN/DMD gradient-norm ratio increases from 0.24320.2432 at step 200200 to 1.58561.5856 at step 800800, indicating that the effective adversarial supervision strength changes throughout training. The corresponding ratio for DINO-GAN is approximately 1.79841.7984, suggesting that V-JEPA-GAN's gradient strength remains within a reasonable range.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

Config.GAN W.D LRG LRAggregationVBench TotalDynamicAestheticImagingVideoAlign TotalVQMQTAGNR.
Ours0.021e-52e-6Multi-head discriminator0.83560.59910.68230.70310.59370.05060.12730.41191.5856
GAN W.0.11e-52e-6Multi-head discriminator0.83370.56580.68130.69020.57480.04520.11870.41092.7429
GAN W.0.51e-52e-6Multi-head discriminator0.82680.47060.67830.65750.53660.03530.10570.39567.8147
D LR0.025e-62e-6Multi-head discriminator0.83490.57570.68040.71030.56750.04350.11940.40461.0361
D LR0.023e-52e-6Multi-head discriminator0.83120.54030.67580.69020.55680.04040.11060.40581.6608
G LR0.021e-55e-7Multi-head discriminator0.82780.58060.67930.69950.56110.04360.12030.39721.1392
G LR0.021e-55e-6Multi-head discriminator0.82840.49080.68320.70040.51200.03570.08560.39071.6594
G LR0.021e-51e-5Multi-head discriminator0.80290.35090.66530.68060.40640.01030.04570.35041.8748
Aggregation0.021e-52e-6Spatial + Temporal0.81410.38770.67020.68530.42060.02970.11060.28031.3529
Aggregation0.021e-52e-6Q-Former0.80520.32070.66230.67750.46000.01530.10090.34381.4438

Aggregation designs: Multi-head discriminator = Multi-Discriminator Heads; Spatial + Temporal = Spatial Average + Temporal Conv + Multi-Discriminator Heads; Q-Former = Q-Forme Heads.

Metrics: GAN W. = Weight of the GAN Branch; GNR. = Weighted GAN/DMD Gradient-Norm Ratio at Step 800800.

(4) Sensitivity to Frame Sampling Rate: We further investigate the influence of temporal sampling by varying the number of discriminator input frames and sampling stride. The results show that both frame count and temporal stride affect generation quality and dynamics. Our default configuration (3232 frames, stride 22) achieves the best overall VBench and VideoAlign scores, while other configurations exhibit different trade-offs between motion magnitude and perceptual quality. These observations highlight the importance of temporal sampling design for effective adversarial supervision.

ConfigurationVBench TotalDynamicAestheticVideoAlign TotalMQTA
16 frames, stride 20.83190.55580.68040.55760.10730.4051
32 frames, stride 10.83320.57290.68110.57610.11780.4149
32 frames, stride 2 (Ours)0.83560.59910.68230.59370.12730.4119
32 frames, stride 40.83020.53250.67840.56010.11050.3975
64 frames, stride 20.83280.60550.67070.58420.12030.4137

Overall, DINOv2 achieves better overall performance under the configurations selected in our paper. V-JEPA's Motion Quality is affected by hyperparameters and architecture choices, and the performance gap may also depend on supervision strength, input sampling, discriminator design and frame sampling strategy. Nevertheless, V-JEPA generally performs worse across multiple configurations, suggesting that its representations may exhibit an inductive bias toward low-motion outputs. Its discriminator heads may more readily reward consistent, conservative generations without equally constraining motion diversity and naturalness, thereby encouraging the generator to reduce motion. We therefore retain the current configuration in our paper. On the other hand, experiments on an internal closed-source TAI2V model show that V-JEPA-GAN can improve speech-prompt following more effectively than DINO-GAN in certain digital-human scenarios. This is also partly reflected in its VideoAlign Text Alignment score of 0.41190.4119, suggesting that V-JEPA-GAN remains useful in specific applications.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

Q3: Prolonged Training and Motion Degradation

We explicitly acknowledge that the mode-seeking behavior associated with DMD can also occur in our method, particularly during prolonged training, where noticeable motion degradation may emerge.

However, our extended training results show that motion degradation does not occur immediately. In fact, both dynamic degree and motion quality improve during the earlier stages of training, with noticeable degradation emerging only after 1,0001,000 steps, around step 1,2001,200. During Stage III training of DINO-GAN, we randomly tracked checkpoints from 00 to 1,5001,500 steps across 55 seeds and averaged the results. Dynamic Degree increases from 0.70420.7042 at step 800800 to 0.73670.7367 at step 900900 and 0.78750.7875 at step 1,0001,000, before dropping to 0.68030.6803 at step 1,2001,200 and continuing to decline. Meanwhile, VideoAlign Motion Quality increases from 0.19120.1912 at step 800800 to 0.21520.2152 at step 1,0001,000, then decreases to 0.17280.1728 at step 1,2001,200. Thus, we observe no substantial motion collapse during the earlier training stages.

This issue also exists in video-VFM GANs and may emerge earlier and more severely. In our V-JEPA joint training experiments, these discriminators tend to favor consistent, conservative outputs, making the generator more prone to reduced motion. Dynamic Degree decreases from 0.62690.6269 at step 200200 to 0.59310.5931 at step 800800. Meanwhile, VBench Total remains nearly unchanged, even increasing slightly from 0.83510.8351 to 0.83560.8356. Therefore, introducing a video feature extractor that explicitly models temporal relationships is not sufficient to prevent motion degradation, and aggregate scores may conceal a substantial decline in motion quality.

In summary, our method cannot fundamentally eliminate the inherent limitations of DMD, and we will explicitly acknowledge this limitation in the revised paper. We do not claim to completely resolve DMD's mode-seeking problem. Rather, while mode seeking remains present, our goal is to cover more diverse, higher-quality generation modes and improve perceptually sensitive dimensions such as human preference, color, subject consistency, and motion.

Reviewer_4QnYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Radio, a representation-space adversarial distillation framework for few-step autogressive video generation. Building on on-policy DMD, Radian decodes frames from student rollouts, extracts multi-level features from a frozen VFM, and trains lightweight, dense discriminator heads against real video frames using hinge loss. Training has three stages: ODE initialization, DMD with discriminator warmup, and joint DMD with adversarial refinement. Experiments cover 4-step chunk-wise, 1-step frame-wise, and one-minute generation. Reported VBench outperforms prior works. The paper also compares image VFMs, video VFMs and diffusion-internal features as discriminator spaces.

Critical strengths
  • The motivation is reasonable. On-policy DMD supervises the student only through teacher score differences in different latent space, adding real-data supervision on decoded RGB frames is a reasonable way to penalize blur, rendering artifacts, and VAE decoding errors.
  • Comparison of discriminator feature spaces is informative. The table-2, comparison across image VFMs, video VFMs, and diffusion-internal features under a shared distillation pipeline, are insightful.
  • Evaluation and qualitative results show visible gains over One Forcing in the 1-step setting.
Critical weaknesses
  • The frozen DINOv2 discriminator with multi-level feature spectra-norm Conv1d heads, and hinge loss on decoded RGB closely follows ADD. Combining DMD reverse-KL score matching with a real-data GAN objective is already established in DMD2. Applying joint DMD plus real-data GAN post-training to causal video self-rollouts is already used in One Forcing and AAD-1.
  • LADD explicitly dropped ADD's frozen DINOv2 pixel-space discriminator in favor of generative teacher latent features because RGB decoding during training is memory-heavy and DINOv2 restricts discrimination to fixed square inputs. The paper does not discuss why LADD moved away from DINOv2 or why causal video distillation reverses that trade-off.
  • Section 4.3 claims that DiT-GAN gradients are strongly coupled with DMD whereas image VFMs and video VFMs gradients are approximately orthogonal to DMD. But didn't discuss the default DINOv2-S model or DiT-GAN.

Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. "Adversarial Diffusion Distillation." European Conference on Computer Vision (ECCV), 2024.

Sauer, A., Boesel, F., Dockhorn, T., Blattmann, A., Esser, P., and Rombach, R. "Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation." SIGGRAPH Asia, 2024.

Li, H., Zeng, Y., Lu, Y., Zhu, J., Ouyang, H., Wang, Q., Cheng, K. L., Shen, Y., and Zhang, Z. "AAD-1: Asymmetric Adversarial Distillation for One-Step Autoregressive Video Generation." arXiv:2606.03972, 2026.

Questions for authors
  • Beyond applying ADD's multi-level frozen DINOv2-S discriminator to sparsely decoded frames from on-policy DMD video rollouts, which of Radian is different compared to ADD, DMD2, and One Forcing?
  • LADD found generative teacher latent features preferable to ADD's frozen DINOv2 pixel-space discriminator. What makes causal autoregressive video generation or on-policy DMD having different conclusion here?
  • What are the GAN–DMD gradient cosine similarity and norm ratio for the default DINOv2-S configuration and for DiT-GAN over Stage III training?
5 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Thank you for your comments. Here are our responses to your comments:

Q1: Contributions Beyond Existing Methods

We thank the reviewer for the opportunity to clarify our contributions. We acknowledge that frozen visual features and lightweight multi-level discriminator heads are not entirely new. Our implementation draws on StyleGAN-T and shares similarities with ADD. Our main contribution is adapting representation-space adversarial supervision to on-policy autoregressive video distillation, rather than proposing a new discriminator architecture.

Specifically, unlike ADD and DMD2, Radian introduces training-framework and system-level optimizations tailored to on-policy autoregressive video distillation, combining causal self-generated rollouts, sparse differentiable RGB decoding, and staged DMD–GAN joint optimization to address error accumulation and training instability. Given the substantial memory overhead of autoregressive rollouts, VAE decoding, and VFM feature extraction and backpropagation, we design a pair-rank strategy that distributes the generator and VAE decoder/VFM/discriminator across two H200 GPUs as one logical training unit through cross-device gradient propagation. Together with sparse decoding and gradient checkpointing, this alleviates the memory pressure of jointly training full on-policy rollouts and RGB-space adversarial supervision. It also enables video VFMs to extract features jointly from multi-second clips while preserving the full gradient path for clip-level adversarial supervision, rather than frame-level discrimination alone.

In contrast, DMD2 and One Forcing use adversarial supervision based on diffusion-related latent features, which are closely associated with the DMD objective and may provide less complementary perceptual information. Specifically, both extract features from DMD's fake-score branch, whose parameters are continuously updated, making it difficult to provide sufficiently independent optimization signals. We therefore consider the real-score branch a potentially better feature extractor. We validate this choice in our four-step chunk-wise experiments using the configuration denoted as DiT-GAN. Moreover, in the one-step setting, our method outperforms One Forcing in VideoAlign Total (0.25630.2563 vs. 0.04480.0448) and Temporal Style (0.70340.7034 vs. 0.69070.6907). Overall, compared with DMD2 and One Forcing, our method better mitigates DMD's mode-seeking behavior by covering more diverse, higher-quality generation modes and improving perceptually sensitive dimensions such as color, subject consistency, motion, and semantics.

We will revise our novelty claims and clarify the limitations of existing approaches.

Q2: Relationship with LADD

We thank the reviewer for this question. We believe our findings do not contradict LADD. Although LADD demonstrates the efficiency advantages of discriminators based on generative latent features, prior findings on the relative generation quality of these two approaches are mixed and metric-dependent. For example, SenseFlow (ICLR 2026, Table 2) shows that replacing DMD2's diffusion-based discriminator with a VFM-based discriminator improves HPSv2 (0.2964→0.29950.2964 \rightarrow 0.2995), ImageReward (0.9324→0.97440.9324 \rightarrow 0.9744), and aesthetic score (5.530→5.6255.530 \rightarrow 5.625), despite a worse FID-T (15.04→18.5515.04 \rightarrow 18.55). This suggests that VFM supervision can improve perceptual quality even when latent-space discriminators perform better on distribution-matching metrics. Similarly, Mage-Flow (arXiv 2026) further demonstrates the benefits of VFM-based adversarial supervision for improving perceptual quality in few-step generation and editing. In on-policy autoregressive video generation, rendering artifacts and structural degradation can accumulate throughout generation, making direct supervision on decoded RGB outputs particularly valuable. We mitigate additional memory overhead through sparse frame decoding.

Our experiments extend this observation to autoregressive video generation rather than contradicting previous image-generation findings. On VBench and VideoAlign, which evaluate visual quality, motion, and semantics, our DINOv2 discriminator outperforms latent-space DiT-GAN in Imaging Quality (0.71480.7148 vs. 0.69840.6984), Dynamic Degree (0.80560.8056 vs. 0.59540.5954), and VideoAlign Total (0.80330.8033 vs. 0.43270.4327). Thus, our conclusion is not specific to autoregressive video generation but further supports the metric-dependent advantages of VFM-based adversarial supervision.


Ge, X., Zhang, X., Xu, T., Zhang, Y., Zhang, X., Wang, Y., & Zhang, J. (2025). “Senseflow: Scaling distribution matching for flow-based text-to-image distillation.” The Fourteenth International Conference on Learning Representations (ICLR), 2026.

Zhang, X., Zhang, P., Zheng, S., Guo, J., Jia, Z., Shen, Y., ... & Lu, Y. (2026). “Mage-Flow: An Efficient Native-Resolution Foundation Model for Image Generation and Editing.” arXiv preprint arXiv:2607.19064.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Q3: Stage III GAN–DMD Gradient Statistics

We thank the reviewer for the suggestion. To examine how different adversarial feature spaces relate to DMD optimization directions, we measured GAN–DMD gradient cosine similarities and gradient-norm ratios for DINOv2-GAN, VideoMAE-GAN, and DiT-GAN throughout Stage III training.

The results reveal substantial differences in gradient directions. The mean absolute gradient cosine similarities of DINOv2-GAN and VideoMAE-GAN with DMD are only 0.0146740.014674 and 0.02340.0234, respectively, indicating nearly orthogonal optimization directions. In contrast, DiT-GAN reaches 0.4401520.440152, exhibiting considerably stronger gradient coupling. Their mean GAN/DMD gradient-norm ratios are 1.79841.7984, 0.86580.8658, and 1.67461.6746, respectively, suggesting that all three provide gradient signals of comparable magnitude to DMD. Their main difference therefore lies in directional complementarity rather than gradient strength alone.

Notably, although all three methods compute gradients in the same high-dimensional parameter space, DiT-GAN still exhibits substantially stronger coupling. Thus, the low cosine similarities of DINOv2 and VideoMAE cannot simply be attributed to natural high-dimensional orthogonality. This partly supports our interpretation that independently pretrained visual representations provide perceptual and temporal supervision approximately orthogonal to DMD. It is also consistent with the advantages of external VFMs, particularly DINOv2-S, on perceptual quality metrics in Table 2. We will include the complete Stage III gradient statistics and analysis in the revised manuscript.

Dear Reviewer 4QnY,

Thank you again for your valuable and constructive feedback. We have provided detailed responses and additional experimental analyses to address your concerns.

As the discussion period is nearing its end, we would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns.

Thank you for your time and consideration.

Best regards,

Authors of Paper #4333

Reviewer_PSfjOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors built their work upon Self-Forcing by incorporating the adversarial loss performed on different feature space. Instead of directly having the discriminator to work in the video latent space as standard video generation literature do, they reconstructed the videos in RGB space and use vision foundation model to reencode the generated video / images, and ask the discriminator to work with the new representations. They argue that gradients in a different feature space provides complementary signal to the DMD loss, thereby providing better training gradients. Experiment results show that the method does improve over other distillation methods.

Critical strengths

I appreciate how the authors redesigned the discriminator's feature space in section 3.4. Projecting the latent space onto feature spaces extracted by VFS is a smart regularization to me that provides better gradient information.

I also find it interesting that empirically using image VFS (like DINO-v2) actually increases the motion of the generated video. Though not a major conclusion, such opservation might motivate future works.

Critical weaknesses

The major concern is the cost of this method, as using VFS and propagating loss and gradient through them can limit the applicability of the method.

The design of the discriminator architecture (sec 3.3) and training pipeline (sec 3.1 and 3.2) also seems pretty standard and common in prior literature, despite the novelty of the use of different feature space in sec 3.4.

Questions for authors
  1. Using adversarial loss in decoded RGB space does provide additional information, yet the cost of training seperate discriminators and forwarding and backwarding through VFS and VAE can be inhibiting. The authors mentioned that temporarily the frames are sparsely sampled, which indeed reduces the cost, but can the authors provide more details or breakdown on the VRAM requirement and training time measurement and compare with the baselines so that readers can have a straightforward and realistic assessment of the applicability of the method?

  2. Also there was little explanation on how to interpret figure 1. It seems to me that the first rows are before training and the second rows are after training, but could the authors explain more about why the first row turns end up in the case of the second row in each cases?

  3. It's understandable that video features (like DiT-GAN / VideoVFM) might suppress dynamic movement over a more consistent one, but why can DINO improve MotionQuality in VideoAlign benchmarks over video features?

  4. I appretiate the authors providing the cosine similarity of VideoMAE vs DMD, but can the authors also provide this number for DiT-GAN vs DMD to support this is a better gradient, as cosine similarity of high-dimensional gradients can be small naturally?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

Thank you for your comments. Here are our responses to your questions:

Q1: Training Cost and Memory Overhead

We thank the reviewer for highlighting this practical deployment concern. We measured training overhead on the equivalent of 88 H200 GPUs, using four-step generation, 8181 frames at 832×480832 \times 480 resolution, an effective video batch size of 88, and one generator update every 55 training iterations. We have added the details of this configuration to the appendix.

MethodPeak Allocated Memory (GiB/GPU)Average Iteration Time (s)Time per 5-Iteration Cycle (s)Relative Time vs. DMD
DMD57.2257.2218.9318.9394.6494.641.00×1.00\times
DMD + DINOv2-S GAN60.1960.1933.8733.87169.37169.371.79×1.79\times
DMD + DiT-GAN60.5060.5044.8344.83224.16224.162.37×2.37\times
DMD + VideoMAE-GAN117.42117.4250.2750.27251.33251.332.66×2.66\times

Each method was measured twice, with 1010 warm-up iterations and 5050 timed iterations per measurement. The timing includes scheduled updates of the generator, discriminator, and fake-score critic, but excludes initialization, checkpoint saving, and evaluation. The total training is usually completed in around 800800 steps.

For the default DINOv2-S configuration, RGB-space adversarial supervision increases peak allocated memory by 2.972.97 GiB per GPU (5.25.2\\%) and average iteration time by 79.079.0\\%. Although the VAE and visual encoder parameters are frozen, generator updates still require backpropagation through them. Sparse differentiable decoding limits activation memory but cannot eliminate the additional computational cost. The added adversarial modules are used only during training and do not increase the generator's inference cost. For longer videos or larger batches that cannot fit in the memory of a single GPU, we adopt a pair-rank strategy that places the generator and the VAE decoder, VFM, and discriminator on two separate GPUs, forming one logical training unit through cross-device gradient propagation while preserving the full gradient path from the adversarial loss to the generator. This strategy enables training to scale to longer videos and larger batches, but requires additional GPU resources and cross-device communication.

Q2: Explanation of Figure 1

We thank the reviewer for the suggestion. Figure 1 is intended to conceptually illustrate the differences in optimization signals provided by different supervision methods, rather than changes in distributions before and after training. The two rows present complementary perspectives: the first illustrates the distribution-matching objectives of different methods, while the second illustrates their corresponding gradient optimization directions. Specifically, DMD aligns the student distribution with that of the diffusion teacher through teacher–student score differences in latent space. Although DiT-GAN introduces adversarial supervision from real data, its gradients may substantially overlap with DMD because it also relies on diffusion-related representations. In contrast, Radian uses independently pretrained visual representations to provide additional perceptual optimization directions for decoded RGB outputs, thereby suppressing low-quality generation modes that may remain under DMD alone. Overall, we do not claim to completely eliminate DMD's mode-seeking problem. Rather, while this problem still exists, we aim to cover more diverse, higher-quality generation modes and improve perceptually sensitive dimensions such as human preference, color, subject consistency, and motion.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

Q3: Why DINOv2 Improves Motion Quality

Although DINOv2 processes only individual frames, improving spatial visual quality can also indirectly improve motion quality in autoregressive video generation. During autoregressive generation, each generated frame or temporal block serves as conditioning context for subsequent predictions. Structural deformation, inconsistent object appearance, and rendering artifacts can therefore accumulate over time and eventually manifest as unstable motion. DINOv2 supervision helps maintain more reliable object structures and perceptual details, thereby improving the continuity and plausibility of motion in subsequent generation. In contrast, video VFMs explicitly model temporal dependencies and may favor predictable, stable temporal representations. Their discriminator heads may be more likely to reward consistent, conservative outputs without equally constraining motion richness and naturalness, thereby encouraging the generator to reduce motion. Table 2 supports this observation: DINOv2 achieves higher Dynamic Degree (0.80560.8056) and VideoAlign Motion Quality (0.20940.2094) than VideoMAE (0.52130.5213, 0.14520.1452) and V-JEPA 2.1 (0.59910.5991, 0.12750.1275). This suggests that even without explicit temporal constraints, improving single-frame visual structure can indirectly enhance video motion quality through the propagation of autoregressive conditioning.

Q4: GAN–DMD Gradient Comparison

We thank the reviewer for pointing out that gradient cosine similarities may naturally be small in high-dimensional spaces. For a controlled comparison, we further measured the GAN–DMD gradient relationships of DINOv2-GAN, VideoMAE-GAN, and DiT-GAN throughout Stage III training under the same student architecture. Their mean absolute gradient cosine similarities are 0.0146740.014674, 0.02340.0234, and 0.4401520.440152, respectively, while their mean GAN/DMD gradient-norm ratios are 1.79841.7984, 0.86580.8658, and 1.67461.6746. Notably, although gradients from all three methods are computed in the same high-dimensional parameter space, DiT-GAN still exhibits substantially stronger gradient coupling. This indicates that the low cosine similarities of DINOv2 and VideoMAE cannot simply be attributed to natural orthogonality in high-dimensional spaces. These results further support our hypothesis that adversarial supervision based on diffusion-internal features overlaps more with DMD optimization directions, whereas external image and video VFMs provide more complementary gradient signals.

I'd like to thank the authors' rebuttal, and they have resolved all my concerns.

  • Regarding Q1 despite the higher training cost, I think the paper does provide a novel way for people with abundance budget to push the limit of video quality.
  • Regarding Q2, it make perfect sense to me now, though I suggest the authors to put the explanation in the rebuttal, specificly "DMD aligns the student distribution with that of the diffusion teacher through teacher–student score differences in latent space. Although DiT-GAN introduces adversarial supervision from real data, its gradients may substantially overlap with DMD because it also relies on diffusion-related representations. In contrast, Radian uses independently pretrained visual representations to provide additional perceptual optimization directions for decoded RGB outputs, thereby suppressing low-quality generation modes that may remain under DMD alone.", in line 93-96 in the revised manuscript or the figure caption, as I think the explanation of Fig.1 in the revised manuscript is still a bit too concise and unclear.
  • Regarding Q3 and Q4, I appreciate the author's explanation.

I believe the paper is ready for acceptance and therefore am willing to raise my score to 4.

Dear Reviewer PSfj,

Thank you for your encouraging feedback and recognition of our work. We appreciate your helpful comments and will further clarify Fig. 1 in the revised manuscript as you suggested.

Best regards,

Authors of Paper #4333

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear ACs and Reviewers,

Thank you very much for your dedication, support, and insightful feedback.

Few-step autoregressive video generation remains challenging, particularly in maintaining visual fidelity and motion quality over long rollouts and extremely-few steps. We are grateful for the reviewers' constructive assessments of our work's strengths:

  • Effective Representation-Space Supervision: Reviewers highlighted that using VFM feature spaces for adversarial supervision is a "smart regularization" strategy (PSfj), and found the motivation of supervising decoded RGB outputs reasonable (4QnY).

  • Comprehensive Experimental Evaluation: Reviewers appreciated the informative comparisons across image, video, and diffusion-internal representations (4QnY, bTk2), as well as the extensive evaluation across different generation settings (bTk2).

  • Strong Empirical Results: Reviewers acknowledged the improvements over existing distillation methods (PSfj), visible gains in one-step generation (4QnY), and the practical value of improving perceptual quality without additional inference cost (bTk2).

We have reviewed all the comments, addressed the reviewers' questions, and conducted additional experiments. All revisions are highlighted in blue in the updated manuscript, and we summarize the major updates below:

  • Contribution Clarification (4QnY, bTk2; Sec. 1–3): Clarified our contributions and video-specific adaptations beyond prior adversarial distillation methods.
  • Relationship with LADD (4QnY; Sec. 2): Clarified the trade-off between latent-space efficiency and RGB-space perceptual quality.
  • V-JEPA Calibration (bTk2; Appendix C.1): Added hyperparameter, aggregation, and temporal sampling ablations to investigate encoder sensitivity.
  • Training Stability (bTk2; Appendix C.1): Analyzed motion quality across extended training and discussed motion degradation.
  • Gradient Complementarity (PSfj, 4QnY; Sec. 4.3, Appendix C.2): Added GAN–DMD gradient direction and magnitude comparisons across feature spaces.
  • Training Efficiency (PSfj, 4QnY; Appendix C.3): Reported GPU memory, runtime, and pair-rank training details.
  • Motion Quality Analysis (PSfj; Sec. 4.3): Explained how image-based supervision improves motion through autoregressive conditioning.
  • Figure Clarity (PSfj; Fig. 1): Clarified the distribution-level and gradient-level interpretations of the illustration.

Request for Feedback

We respectfully invite the reviewers to carefully evaluate our revisions and individual responses. We are more than willing to address any remaining questions or concerns. If our additional experiments and clarifications sufficiently address your concerns, we would appreciate your consideration of increasing your scores. We sincerely appreciate your thoughtful engagement and constructive suggestions.

Best regards,

Authors of Submission #4333

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.