- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies compounding error in autoregressive video diffusion by examining the effective rank of intermediate representations. It proposes Video Representation Regularization (VRR), which adds SIGReg or Uniformity losses to Diffusion Forcing training. Experiments on Minecraft, with additional validation on RealEstate10K, show improved video quality. The main contribution is an empirical association between representation collapse and rollout degradation, together with a simple regularization-based intervention.
Critical strengths
-
Substantial performance gains from a simple training-time intervention. The method improves long-rollout video quality by regularizing existing hidden states, without introducing an additional network or modifying the inference procedure.
-
A useful representation-level perspective on long-horizon rollout failure. The observed correspondence between declining hidden-state effective rank and visual degradation provides a potentially useful diagnostic for autoregressive rollout instability.
Critical weaknesses
-
Unclear connection between representation regularization and reduced error compounding. The current evidence does not establish whether VRR specifically mitigates error amplification during autoregressive rollout, beyond the general generation-quality gains already demonstrated by representation regularization methods such as DispLoss.
-
Missing comparisons with directly relevant approaches to compounding error. Despite targeting compounding error, the paper does not compare against representative Self-Forcing-based approaches that address the same problem through training on self-generated contexts.
-
Limited evidence of generality beyond the evaluated Diffusion Forcing setups. Although two backbones are evaluated, validation remains within Diffusion Forcing-based setups, leaving the transferability of VRR to larger pretrained video models and alternative autoregressive training frameworks unestablished.
-
Insufficient detail and analysis of the regularization ablations. Section 5.4 lacks sufficient implementation details and analysis of the evaluated regularization methods. In particular, the poor performance of direct effective-rank regularization remains unexplained, raising questions about the proposed connection between effective-rank collapse and long-horizon generation failure.
Questions for authors
-
Generalization to different models. Does the proposed regularization provide similar improvements when applied to a broader range of video generation models, particularly larger and more recent architectures?
-
Effectiveness in fine-tuning. Does the proposed regularization remain effective when fine-tuning pretrained video generation models, rather than training models from scratch?
Please also address the concerns raised in the Weaknesses section. I am open to revising my rating upward if the authors provide convincing explanations and supporting evidence that adequately address these concerns.
4 comments
W3: Generalization is limited to Diffusion Forcing setups.
Q1: Does VRR provide similar improvements on broader architectures and larger, more recent video-generation models?
We extend the architecture comparison to a full 3D-attention DiT on Minecraft, trained for the same 16k steps (Appendix E.4, Table 11):
| Method | AQ ↑ | IQ ↑ |
|---|---|---|
| Full 3D-attention DF | 35.49 | 51.73 |
| + VRR-Unif | 51.67 | 57.69 |
This yields gains of 16.18 AQ and 5.96 IQ points. Figure 12 shows the corresponding representation dynamics: DF loses rank sharply during rollout, while VRR-Unif retains higher rank at layers 7, 11, and 15. The result therefore extends beyond factorized spatial/temporal attention.
For a larger pretrained backbone and a different autoregressive training framework, the Self Forcing experiment in Section 5.6 improves total VBench scores from 74.47 to 76.97 with SIGReg and 76.92 with Unif, alongside semantic and quality improvements. We also inspect the official Self Forcing DMD checkpoint during a one-minute rollout. Appendix E.4, Figure 13, shows generally declining erank at layers 19, 24, and 29, excluding chunk 0 because of causal-VAE handling. These additional settings broaden both the representation observations and the intervention results beyond the original DF experiments.
W4: Regularization ablations lack implementation details, and the poor direct-erank results leave the representation-collapse connection unclear.
Appendix D specifies the implementations. SIGReg, Unif, and direct-erank regularization use zero-indexed Minecraft layers 0, 7, and 15. The coefficients are for SIGReg and 1 for Unif and direct erank, with the weighted selected-layer losses averaged. SIGReg uses 256 normalized random projections and characteristic-function matching to a standard Gaussian on a 17-point grid over [-5,5]. Unif normalizes flattened token vectors and minimizes .
Direct erank centers each frame's spatial features and minimizes , so its sign encourages higher rank. Figure 10 confirms that it does maintain high rollout rank. However, Figure 11 shows that its diffusion loss stays higher than SIGReg and Unif, and its 16k AQ/IQ is 37.05/37.49, compared with DF's 38.65/44.37. Thus, optimizing dimensionality alone can result in a poor fit to the denoising task. This explains why the direct-erank ablation need not produce the same gains as the successful regularizers. RankMe [2] provides related work on rank-based representation assessment; high rank does not fully account for task performance.
Appendix D also gives the Barlow Twins and VICReg implementations and their corrected results. We will provide the implementation code in the supplementary material. Frame copying remains a hypothesis, as acknowledged in our limitations; the experimental findings concern the observed representation dynamics and the effectiveness of regularization.
Q2: Is VRR effective when fine-tuning pretrained models rather than training from scratch?
Yes. In the added Self Forcing experiment, all runs begin from the same pretrained ODE-initialized checkpoint. Under the same 6,400-step budget, SIGReg improves semantic/quality/total scores from 56.56/78.94/74.47 to 59.02/81.46/76.97; Unif achieves 60.29/81.08/76.92 (Section 5.6, Table 4 and Figure 8).
To apply VRR to the pretrained representations, we warm up its coefficient and set its contribution to approximately 10–20% of the DMD loss, regularizing all transformer layers. These results directly demonstrate that VRR remains useful during continued training of a pretrained video model, in addition to the from-scratch experiments.
Reference:
- Wang and He. Diffuse and Disperse: Image Generation with Representation Regularization. 2025.
- Garrido et al. RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank. ICML 2023.
Thank you for the additional experiments and clarifications. However, i still have a question:
- The results of adding VRR to Self-Forcing are useful. However, my original request was a direct comparison between VRR and Self-Forcing. Since both aim to mitigate rollout errors, I would also appreciate some discussions about how do their effects differ, and what failure modes does VRR address that Self-Forcing still leaves unresolved.
Thank you for clarifying your request. VRR is designed to improve autoregressive diffusion models during pretraining, while the Self-Forcing configuration uses few-step distillation during post-training. A direct comparison under their original settings raises two issues.
- Training setup. VRR is applied during Diffusion Forcing pretraining, whereas the published Self-Forcing variants are trained as few-step autoregressive generators during post-training. The DMD variant requires a pretrained bidirectional teacher, and the SiD variant also requires a pretrained real-score model. For a direct comparison in our setting, either variant would require a suitable pretrained score model for our data and conditioning setup. The GAN variant uses real video clips and a learned discriminator. This introduces an adversarial training stage, for which the original implementation uses a global batch size of 768 to improve stability. All three published variants use a four-step generator, so adopting any of them would change both the training objective and the denoising schedule relative to our 20-step DDIM setting.
- Inference setup. Self-Forcing produces a few-step diffusion model, while our Diffusion Forcing (DF) models, including those trained with VRR, use multi-step DDIM inference. Comparing their default configurations would give VRR more denoising steps. Reducing the DDIM step count for VRR would evaluate it in a few-step regime for which it was never trained.
This gap is difficult to bridge in practice. Self-Forcing uses few-step sampling to keep training-time rollouts tractable. Running 20 or 50 DDIM steps for each autoregressive prediction during training would be costly for a large DiT. The DMD objective is part of this few-step distillation setup. These considerations motivated the Self-Forcing + VRR experiment in our previous response, which evaluated the effect of adding VRR within the same post-training setup.
To address your question more directly, we ran an additional experiment that isolates the core idea of training on self-generated history. We let the DiT roll out autoregressively with a window size of 20 and 20 DDIM steps per prediction, then apply the same v-prediction diffusion loss as in our original setup. This setup requires no DMD loss or teacher model, and KV caching keeps the rollout cost manageable. We tested two settings: training from scratch and fine-tuning from the 16,000-step DF checkpoint. Both used a global batch size of 128 and ran for 4,000 training steps over approximately 16 hours. Each training rollout involves 20 × 20 = 400 DDIM denoising steps, so the training-step counts are not directly comparable to those of DF. The two settings below are our self-rollout variants, distinct from the original Self-Forcing distillation recipe.
| Method | Aesthetic Quality ↑ | Imaging Quality ↑ |
|---|---|---|
| DF (16,000 steps) | 38.65 | 44.37 |
| DF (40,000 steps) | 38.10 | 52.20 |
| Self-rollout training (from scratch) | 48.35 | 49.80 |
| Self-rollout training (fine-tuned from DF at 16,000 steps) | 53.42 | 62.91 |
| VRR-SIGReg (16,000 steps) | 55.56 | 69.51 |
Training on self-generated history improves both metrics relative to the 16,000-step DF baseline, and fine-tuning with self-rollouts improves both metrics relative to training with self-rollouts from scratch. These results are consistent with the idea that exposing the model to its own predictions during training helps mitigate error accumulation. VRR-SIGReg gives the highest scores on both metrics in this comparison.
The two approaches act on different parts of the problem. Training on self-generated history reduces the mismatch between the conditioning histories seen during training and inference. VRR regularizes the model’s internal representations during pretraining to improve long-horizon stability. The two approaches can be combined by using VRR during pretraining and then applying Self-Forcing during post-training to further reduce the training–inference gap.
We thank the reviewer for the questions about error accumulation, relevant comparisons, and generalization. We provide additional representation measurements, Self Forcing and BAgger experiments, and a full 3D-attention model. Figure and table numbers below refer to the current manuscript; AQ/IQ denote Aesthetic/Imaging Quality.
Our focus is maintaining quality as autoregressive generation proceeds. Each Minecraft evaluation begins with 100 ground-truth prompt frames, so all methods receive the same high-quality real context. The challenge is to preserve generation quality as generated frames progressively become the conditioning history. Section 5.3 evaluates precisely this process: Figure 6 measures quality in successive 5-second segments of each long rollout. DF deteriorates as the rollout progresses, whereas VRR retains substantially more stable quality. Thus, the evaluation concerns deterioration over time in addition to average generation quality.
We further measure the representations themselves in Appendix D.1, Figure 9. Across 20 rollouts, VRR-SIGReg and VRR-Unif maintain higher late-rollout erank than DF at layers 7, 11, and 15. These curves provide evidence that regularization improves the representations encountered during long autoregressive inference. The additional recorded-continuation metrics in Table 2 and action-following probes in Appendix B.1, Table 5, also support improvements beyond frame appearance alone.
The distinction from DispLoss [1] is the research focus: our motivation is representation degradation during long autoregressive rollout, and we use established regularization principles to study and alleviate that failure. We do not present Unif as a new loss. We also distinguish the representation/rollout association from a complete causal explanation: direct rank maximization maintains high rank but has higher diffusion loss and poorer quality (Figures 10 and 11). An effective intervention therefore needs to preserve both representation quality and denoising fit.
We add experiments in the official Self Forcing implementation, retaining its self-generated contexts and DMD loss. All three models start from the same ODE-initialized checkpoint and train for 6,400 steps with eight GPUs and global batch size 8. VRR regularizes all transformer layers with coefficient warmup; the regularization term is scaled to approximately 10–20% of the DMD loss. Evaluation uses more than 900 prompts expanded following the official implementation.
Section 5.6 presents the numerical results in Table 4 and the dimension-wise radar plot in Figure 8:
Both variants improve Self Forcing, showing that VRR can complement training on self-generated contexts. We use this official setting because adapting Self Forcing directly to our Minecraft experiment would require an additional bidirectional teacher and costly training rollouts with a 20-frame window and 20 DDIM steps. This would change the training resources and make that adaptation impractical within our budget.
We also reproduce BAgger's corrective data-aggregation approach on Minecraft because implementation code was unavailable for our experiment. Each round collects 6,000 trajectories by generating 10 frames from 10 real history frames, then reverses the latent trajectory and actions to create corrective examples. The data are accumulated across rounds, while training restarts from the fixed base checkpoint. The initial training stage and each of the three rounds use 6k steps, giving 24k training steps plus rollout collection, at approximately the cost of 40k-step DF. Appendix E.3 provides the detailed protocol and Table 10 reports the reproduction results:
BAgger improves over the 16k DF baseline in this reproduction, but VRR-SIGReg performs better at both reported training budgets. The decline in round 3 suggests that increasing the amount of corrective data is not uniformly beneficial under this setup. We discuss possible effects of reversed-trajectory supervision and generated-data weighting in Appendix E.3.