acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Compounding Error via Video Representation Regularization

Abstract

Video diffusion-based world models enable long autoregressive video generation for robotics, autonomous driving and simulation tasks, yet sliding-window autoregressive inference suffers from severe error accumulation that degrades frame quality over time. Although this phenomenon has been widely observed, the underlying mechanism of compounding error and how to achieve stable long-horizon generation remain largely unresolved. In this paper, we investigate the internal representation dynamics of video world models and discover that compounding error is tightly coupled with dimensional collapse of hidden representations. Specifically, the effective rank of model representations sharply decreases at the onset of generation drift, revealing a strong connection between representational degradation and long-term rollout instability. Furthermore, we find that pure training data scaling fails to boost model resistance to error drift, contradicting mainstream scaling paradigms. To address this problem, we propose video representation regularization, a lightweight training constraint that stabilizes latent representations and suppresses iterative error accumulation. Compared with Diffusion Forcing, our method achieves improvements from 38.65 to 53.78 and from 44.37 to 72.08 on the Aesthetic Quality and Imaging Quality metrics of VBench. Our work establishes the first connection between autoregressive video drifting and model internal representations, adopts erank as a quantitative metric for error accumulation, reveals counterintuitive scaling limitations for video world models, and presents a simple yet effective regularization strategy to improve long video generation robustness.

Primary area:
generative models
Keywords:
world model, video generation, compounding error, representation regularization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bVxeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies compounding error in autoregressive video diffusion by examining the effective rank of intermediate representations. It proposes Video Representation Regularization (VRR), which adds SIGReg or Uniformity losses to Diffusion Forcing training. Experiments on Minecraft, with additional validation on RealEstate10K, show improved video quality. The main contribution is an empirical association between representation collapse and rollout degradation, together with a simple regularization-based intervention.

Critical strengths
  1. Substantial performance gains from a simple training-time intervention. The method improves long-rollout video quality by regularizing existing hidden states, without introducing an additional network or modifying the inference procedure.

  2. A useful representation-level perspective on long-horizon rollout failure. The observed correspondence between declining hidden-state effective rank and visual degradation provides a potentially useful diagnostic for autoregressive rollout instability.

Critical weaknesses
  1. Unclear connection between representation regularization and reduced error compounding. The current evidence does not establish whether VRR specifically mitigates error amplification during autoregressive rollout, beyond the general generation-quality gains already demonstrated by representation regularization methods such as DispLoss.

  2. Missing comparisons with directly relevant approaches to compounding error. Despite targeting compounding error, the paper does not compare against representative Self-Forcing-based approaches that address the same problem through training on self-generated contexts.

  3. Limited evidence of generality beyond the evaluated Diffusion Forcing setups. Although two backbones are evaluated, validation remains within Diffusion Forcing-based setups, leaving the transferability of VRR to larger pretrained video models and alternative autoregressive training frameworks unestablished.

  4. Insufficient detail and analysis of the regularization ablations. Section 5.4 lacks sufficient implementation details and analysis of the evaluated regularization methods. In particular, the poor performance of direct effective-rank regularization remains unexplained, raising questions about the proposed connection between effective-rank collapse and long-horizon generation failure.

Questions for authors
  1. Generalization to different models. Does the proposed regularization provide similar improvements when applied to a broader range of video generation models, particularly larger and more recent architectures?

  2. Effectiveness in fine-tuning. Does the proposed regularization remain effective when fine-tuning pretrained video generation models, rather than training models from scratch?

Please also address the concerns raised in the Weaknesses section. I am open to revising my rating upward if the authors provide convincing explanations and supporting evidence that adequately address these concerns.

4 comments

We thank the reviewer for the questions about error accumulation, relevant comparisons, and generalization. We provide additional representation measurements, Self Forcing and BAgger experiments, and a full 3D-attention model. Figure and table numbers below refer to the current manuscript; AQ/IQ denote Aesthetic/Imaging Quality.

W1: It is unclear whether VRR specifically mitigates autoregressive error amplification beyond general generation-quality gains from representation regularization methods such as DispLoss.

Our focus is maintaining quality as autoregressive generation proceeds. Each Minecraft evaluation begins with 100 ground-truth prompt frames, so all methods receive the same high-quality real context. The challenge is to preserve generation quality as generated frames progressively become the conditioning history. Section 5.3 evaluates precisely this process: Figure 6 measures quality in successive 5-second segments of each long rollout. DF deteriorates as the rollout progresses, whereas VRR retains substantially more stable quality. Thus, the evaluation concerns deterioration over time in addition to average generation quality.

We further measure the representations themselves in Appendix D.1, Figure 9. Across 20 rollouts, VRR-SIGReg and VRR-Unif maintain higher late-rollout erank than DF at layers 7, 11, and 15. These curves provide evidence that regularization improves the representations encountered during long autoregressive inference. The additional recorded-continuation metrics in Table 2 and action-following probes in Appendix B.1, Table 5, also support improvements beyond frame appearance alone.

The distinction from DispLoss [1] is the research focus: our motivation is representation degradation during long autoregressive rollout, and we use established regularization principles to study and alleviate that failure. We do not present Unif as a new loss. We also distinguish the representation/rollout association from a complete causal explanation: direct rank maximization maintains high rank but has higher diffusion loss and poorer quality (Figures 10 and 11). An effective intervention therefore needs to preserve both representation quality and denoising fit.

W2: The paper lacks comparisons with Self-Forcing-based approaches that train on self-generated contexts.

We add experiments in the official Self Forcing implementation, retaining its self-generated contexts and DMD loss. All three models start from the same ODE-initialized checkpoint and train for 6,400 steps with eight GPUs and global batch size 8. VRR regularizes all transformer layers with coefficient warmup; the regularization term is scaled to approximately 10–20% of the DMD loss. Evaluation uses more than 900 prompts expanded following the official implementation.

Section 5.6 presents the numerical results in Table 4 and the dimension-wise radar plot in Figure 8:

MethodSemantic ↑Quality ↑Total ↑
Self Forcing DMD56.5678.9474.47
+ VRR-SIGReg59.0281.4676.97
+ VRR-Unif60.2981.0876.92

Both variants improve Self Forcing, showing that VRR can complement training on self-generated contexts. We use this official setting because adapting Self Forcing directly to our Minecraft experiment would require an additional bidirectional teacher and costly training rollouts with a 20-frame window and 20 DDIM steps. This would change the training resources and make that adaptation impractical within our budget.

We also reproduce BAgger's corrective data-aggregation approach on Minecraft because implementation code was unavailable for our experiment. Each round collects 6,000 trajectories by generating 10 frames from 10 real history frames, then reverses the latent trajectory and actions to create corrective examples. The data are accumulated across rounds, while training restarts from the fixed base checkpoint. The initial training stage and each of the three rounds use 6k steps, giving 24k training steps plus rollout collection, at approximately the cost of 40k-step DF. Appendix E.3 provides the detailed protocol and Table 10 reports the reproduction results:

MethodAQ ↑IQ ↑
BAgger round 2, 18k training steps46.7452.80
BAgger round 3, 24k training steps40.1849.07
DF, 16k steps38.6544.37
VRR-SIGReg, 16k steps55.5669.51
DF, 40k steps38.1052.20
VRR-SIGReg, 40k steps53.4466.57

BAgger improves over the 16k DF baseline in this reproduction, but VRR-SIGReg performs better at both reported training budgets. The decline in round 3 suggests that increasing the amount of corrective data is not uniformly beneficial under this setup. We discuss possible effects of reversed-trajectory supervision and generated-data weighting in Appendix E.3.

W3: Generalization is limited to Diffusion Forcing setups.

Q1: Does VRR provide similar improvements on broader architectures and larger, more recent video-generation models?

We extend the architecture comparison to a full 3D-attention DiT on Minecraft, trained for the same 16k steps (Appendix E.4, Table 11):

MethodAQ ↑IQ ↑
Full 3D-attention DF35.4951.73
+ VRR-Unif51.6757.69

This yields gains of 16.18 AQ and 5.96 IQ points. Figure 12 shows the corresponding representation dynamics: DF loses rank sharply during rollout, while VRR-Unif retains higher rank at layers 7, 11, and 15. The result therefore extends beyond factorized spatial/temporal attention.

For a larger pretrained backbone and a different autoregressive training framework, the Self Forcing experiment in Section 5.6 improves total VBench scores from 74.47 to 76.97 with SIGReg and 76.92 with Unif, alongside semantic and quality improvements. We also inspect the official Self Forcing DMD checkpoint during a one-minute rollout. Appendix E.4, Figure 13, shows generally declining erank at layers 19, 24, and 29, excluding chunk 0 because of causal-VAE handling. These additional settings broaden both the representation observations and the intervention results beyond the original DF experiments.

W4: Regularization ablations lack implementation details, and the poor direct-erank results leave the representation-collapse connection unclear.

Appendix D specifies the implementations. SIGReg, Unif, and direct-erank regularization use zero-indexed Minecraft layers 0, 7, and 15. The coefficients are 5×10−45 \times 10^{-4} for SIGReg and 1 for Unif and direct erank, with the weighted selected-layer losses averaged. SIGReg uses 256 normalized random projections and characteristic-function matching to a standard Gaussian on a 17-point grid over [-5,5]. Unif normalizes flattened token vectors and minimizes ∥U⊤U∥F2/Nu2\|U^\top U\|_F^2/N_u^2.

Direct erank centers each frame's spatial features and minimizes 1−Eb,t[erank]/min⁡(HW−1,D)1-\mathbb E_{b,t}[\mathrm{erank}]/\min(HW-1,D), so its sign encourages higher rank. Figure 10 confirms that it does maintain high rollout rank. However, Figure 11 shows that its diffusion loss stays higher than SIGReg and Unif, and its 16k AQ/IQ is 37.05/37.49, compared with DF's 38.65/44.37. Thus, optimizing dimensionality alone can result in a poor fit to the denoising task. This explains why the direct-erank ablation need not produce the same gains as the successful regularizers. RankMe [2] provides related work on rank-based representation assessment; high rank does not fully account for task performance.

Appendix D also gives the Barlow Twins and VICReg implementations and their corrected results. We will provide the implementation code in the supplementary material. Frame copying remains a hypothesis, as acknowledged in our limitations; the experimental findings concern the observed representation dynamics and the effectiveness of regularization.

Q2: Is VRR effective when fine-tuning pretrained models rather than training from scratch?

Yes. In the added Self Forcing experiment, all runs begin from the same pretrained ODE-initialized checkpoint. Under the same 6,400-step budget, SIGReg improves semantic/quality/total scores from 56.56/78.94/74.47 to 59.02/81.46/76.97; Unif achieves 60.29/81.08/76.92 (Section 5.6, Table 4 and Figure 8).

To apply VRR to the pretrained representations, we warm up its coefficient and set its contribution to approximately 10–20% of the DMD loss, regularizing all transformer layers. These results directly demonstrate that VRR remains useful during continued training of a pretrained video model, in addition to the from-scratch experiments.

Reference:

  1. Wang and He. Diffuse and Disperse: Image Generation with Representation Regularization. 2025.
  2. Garrido et al. RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank. ICML 2023.

Thank you for the additional experiments and clarifications. However, i still have a question:

  • The results of adding VRR to Self-Forcing are useful. However, my original request was a direct comparison between VRR and Self-Forcing. Since both aim to mitigate rollout errors, I would also appreciate some discussions about how do their effects differ, and what failure modes does VRR address that Self-Forcing still leaves unresolved.

Thank you for clarifying your request. VRR is designed to improve autoregressive diffusion models during pretraining, while the Self-Forcing configuration uses few-step distillation during post-training. A direct comparison under their original settings raises two issues.

  1. Training setup. VRR is applied during Diffusion Forcing pretraining, whereas the published Self-Forcing variants are trained as few-step autoregressive generators during post-training. The DMD variant requires a pretrained bidirectional teacher, and the SiD variant also requires a pretrained real-score model. For a direct comparison in our setting, either variant would require a suitable pretrained score model for our data and conditioning setup. The GAN variant uses real video clips and a learned discriminator. This introduces an adversarial training stage, for which the original implementation uses a global batch size of 768 to improve stability. All three published variants use a four-step generator, so adopting any of them would change both the training objective and the denoising schedule relative to our 20-step DDIM setting.
  2. Inference setup. Self-Forcing produces a few-step diffusion model, while our Diffusion Forcing (DF) models, including those trained with VRR, use multi-step DDIM inference. Comparing their default configurations would give VRR more denoising steps. Reducing the DDIM step count for VRR would evaluate it in a few-step regime for which it was never trained.

This gap is difficult to bridge in practice. Self-Forcing uses few-step sampling to keep training-time rollouts tractable. Running 20 or 50 DDIM steps for each autoregressive prediction during training would be costly for a large DiT. The DMD objective is part of this few-step distillation setup. These considerations motivated the Self-Forcing + VRR experiment in our previous response, which evaluated the effect of adding VRR within the same post-training setup.

To address your question more directly, we ran an additional experiment that isolates the core idea of training on self-generated history. We let the DiT roll out autoregressively with a window size of 20 and 20 DDIM steps per prediction, then apply the same v-prediction diffusion loss as in our original setup. This setup requires no DMD loss or teacher model, and KV caching keeps the rollout cost manageable. We tested two settings: training from scratch and fine-tuning from the 16,000-step DF checkpoint. Both used a global batch size of 128 and ran for 4,000 training steps over approximately 16 hours. Each training rollout involves 20 × 20 = 400 DDIM denoising steps, so the training-step counts are not directly comparable to those of DF. The two settings below are our self-rollout variants, distinct from the original Self-Forcing distillation recipe.

MethodAesthetic Quality ↑Imaging Quality ↑
DF (16,000 steps)38.6544.37
DF (40,000 steps)38.1052.20
Self-rollout training (from scratch)48.3549.80
Self-rollout training (fine-tuned from DF at 16,000 steps)53.4262.91
VRR-SIGReg (16,000 steps)55.5669.51

Training on self-generated history improves both metrics relative to the 16,000-step DF baseline, and fine-tuning with self-rollouts improves both metrics relative to training with self-rollouts from scratch. These results are consistent with the idea that exposing the model to its own predictions during training helps mitigate error accumulation. VRR-SIGReg gives the highest scores on both metrics in this comparison.

The two approaches act on different parts of the problem. Training on self-generated history reduces the mismatch between the conditioning histories seen during training and inference. VRR regularizes the model’s internal representations during pretraining to improve long-horizon stability. The two approaches can be combined by using VRR during pretraining and then applying Self-Forcing during post-training to further reduce the training–inference gap.

Reviewer_Fg7iOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies collapse in long autoregressive rollouts of an action-conditioned Diffusion Forcing video model on Minecraft. The setup is 18k MineRL sequences, a 16-layer DiT, a 100-frame prompt and 1,100 generated frames. The authors report that the effective rank of intermediate DiT hidden states drops sharply at the frame where a rollout collapses, and that longer training does not fix the collapse. They propose VRR: during training, add SIGReg or a uniformity loss on the hidden states of layers 0, 7 and 15. On 20 evaluation sequences at 16k steps, VRR raises VBench Aesthetic Quality from 38.65 to 55.56 (SIGReg) / 53.78 (Unif), and Imaging Quality from 44.37 to 69.51 / 72.08. The baselines are DF, Frame Anchor and SVI. A smaller UNet experiment on RealEstate10K shows the same trend for the uniformity variant.

Critical strengths

VRR removes the visual collapse of the DF baseline at almost no cost. It is a training-time loss on three layers, with no change to the architecture or to inference. The supplementary videos agree with the tables. DF rollouts turn into near-static blue or white frames after a few hundred frames. Both VRR variants keep producing Minecraft terrain for the full minute. The effect holds for two different regularizers and for every checkpoint from 8k to 16k. This is the main reason I do not score the paper lower: the method clearly changes model behavior.

Critical weaknesses

Main: the evaluation shows that VRR frames do not look broken. It does not show that VRR is a better world model, which is what "mitigating compounding error" claims.

  • All method comparisons (Tables 1, 2, 4) use only VBench Aesthetic and Imaging Quality. §5.1 says these "mainly evaluate whether the frames contain noise or suffer from visual collapse." Both are per-frame scores that ignore the action input and the history. A model that stops following actions, or drifts into generic Minecraft scenery, would score just as well. Pushing hidden states toward an isotropic Gaussian or a uniform distribution could plausibly weaken action conditioning, and nothing in the paper rules this out.
  • The data to test this is available. The 20 evaluation sequences are held-out recordings with actions, and 100 prompt frames plus 1,100 generated frames cover the full 1,200-frame recording. The real continuation under the same actions therefore exists. The paper already computes SSIM/PSNR/LPIPS against it in Fig. 2, and VRAG, whose protocol is followed, reports these metrics. None of them appear in the comparison tables. There is also no FVD, no action-following measure and no Dynamic Degree.
  • The reported metrics can reward a frozen video. DF scores higher than VRR on Temporal Flickering (Table 3). In the supplementary videos DF is nearly static after collapse. Mean absolute consecutive-frame difference over frames 600–1200 (0–255 scale) is 0.71 and 3.52 for the two DF videos. On the same sequence as the second DF video, it is 12.18 for SIGReg and 8.54 for Unif.
  • The evaluation uses 20 sequences, one run and no variance. Layers 0/7/15 were selected on the same 20 sequences: the last row of Table 4 is identical to the SIGReg row of Table 1.

Minor:

  1. Two of the three analysis claims conflict with the paper's own evidence.
    • §3.3 says more training makes long rollouts worse. Fig. 3 (right) shows the 4k checkpoint degrading first, while the 32k–40k checkpoints hold up until about 15–25 s. The experiment also varies training steps on a fixed dataset, so it says nothing about dataset size.
    • §3.2 says SSIM/PSNR/LPIPS do not pinpoint the collapse frame. In erank_collapse.mp4, all three jump at the same frame as erank (~400). Fig. 2 plots them only for the 4k checkpoint, which never collapses.
  2. The causal role of erank is unsupported. Directly maximizing erank does not help: Table 2 gives 37.05/37.49 at 16k, below DF. The erank of VRR models is never reported, and the frame-copying shortcut hypothesis is not tested.
  3. VRR is trained only to 16k steps, while the degradation argument relies on 40k-step DF checkpoints.
  4. Neither regularizer is given a formula. The uniformity loss has no citation (presumably Wang & Isola, 2020). VRR-Unif looks nearly identical to Dispersive Loss (Wang & He, 2025), which is cited in related work but not compared against. In Table 2, Barlow Twins and VICReg have identical 16k values (41.44/36.74), likely a copy error.
Questions for authors
  1. For DF, Frame Anchor, SVI and both VRR variants on the 20 held-out sequences, please report two things. First, LPIPS/SSIM against the recorded continuation over the first few seconds, where the continuation is still largely determined by the prompt and the actions. Second, FVD against held-out real clips over the full minute.
  2. Does VRR still follow actions? Please report action accuracy from an inverse dynamics model run on the generated videos (e.g., VPT's IDM), or agreement between commanded camera rotation and rotation estimated from optical flow. Please also report Dynamic Degree.
  3. Please report mean and std over several seeds. Please also report results with layers and coefficients selected on sequences other than the 20 test sequences.
4 comments

We thank the reviewer for pointing out the need to evaluate action following and long-horizon behavior beyond frame appearance. We add recorded-continuation metrics, FVD, action-following probes, Dynamic Degree, multiple inference seeds, and 100 new evaluation prompts. References to figures and tables below use the current manuscript numbering; AQ/IQ denote Aesthetic/Imaging Quality.

W1: Per-frame quality cannot establish a better world model: VRR could weaken action conditioning or drift away from the input history.

We agree that frame quality alone does not answer these questions. The added action-following evaluation in Appendix B.1, Table 5, shows that VRR improves agreement with ground-truth motion under both optical-flow and camera-pose probes: DF scores 39.86/39.14, SIGReg 42.88/53.53, and Unif 43.91/52.61 (optical flow/VGGT, %). This provides evidence that the quality gains are accompanied by better directional motion agreement, rather than being achieved by ignoring the action-conditioned continuation.

Long-term memory and resistance to visual degradation address complementary aspects of world modeling. To test their compatibility, we combine VRR with VRAG's retrieval-based memory. Following VRAG, 10 retrieved frames condition 10 target frames; VRR regularizes the target-frame representations with its original settings. Appendix E.5, Table 12, reports long-rollout quality and separate memory-task results:

MethodLong-rollout AQ ↑Long-rollout IQ ↑Memory PSNR ↑Memory SSIM ↑Memory LPIPS ↓
DF38.6544.3714.920.330.52
VRAG47.9554.2818.470.520.43
VRAG + VRR-SIGReg53.3266.1518.510.530.44
VRAG + VRR-Unif54.3068.8017.550.500.46

The memory evaluation uses VRAG's 60 turning-in-place and round-trip videos of 300 frames, separately from the 20 long-rollout videos. Adding SIGReg substantially improves long-rollout quality while largely preserving VRAG's memory performance. Unif also improves long-rollout quality, with some reduction in the memory scores relative to VRAG alone. Both combinations retain better memory scores than DF. Thus, VRR can complement a method designed for memory; our contribution focuses on long-rollout stability rather than replacing memory mechanisms.

W2: Recorded continuations are available, but comparative PSNR/SSIM/LPIPS, FVD, action following, and Dynamic Degree are missing.

Q1: For DF, Frame Anchor, SVI, and both VRR variants, report early-continuation LPIPS/SSIM and FVD against held-out real clips over the full rollout.

We provide the requested early-window comparison below. It uses frames [100,300), approximately the first ten generated seconds after the prompt. These results are included in this response rather than added to the manuscript.

MethodPSNR (dB) ↑SSIM ↑LPIPS ↓FVD ↓
Diffusion Forcing11.48280.38910.64381632.98
Frame Anchor11.39490.39300.63801486.21
DF-SVI12.04870.38880.62961475.26
VRR-SIGReg12.13460.40940.62661398.95
VRR-Unif12.58420.41280.61271319.63

Both VRR variants improve all four early-window metrics over the baselines. Table 2 additionally reports the comparison over the full generated continuation, excluding the 100 conditioning frames:

MethodPSNR (dB) ↑SSIM ↑LPIPS ↓FVD ↓
Diffusion Forcing7.16680.36280.81504937.55
Frame Anchor7.27070.36570.80884091.36
DF-SVI7.37420.37640.78504828.16
VRR-SIGReg11.22920.38600.65071568.02
VRR-Unif10.48180.38490.63381685.97

VRR also improves all four metrics over the full continuation. In particular, SIGReg reduces FVD from 4937.55 to 1568.02 and LPIPS from 0.8150 to 0.6507. The early window checks short-term prediction, but most methods have not yet collapsed during that interval. The full-continuation comparison and segment-wise quality curves in Figure 6 therefore remain important for evaluating the long-horizon problem studied here. Reference-based distances also penalize plausible divergence from the recorded trajectory, so we interpret them together with visual quality and the additional motion evaluations.

W3: Temporal Flickering can reward a frozen video, as seen in the nearly static collapsed DF outputs.

Q2: Does VRR still follow actions? Please report inverse-dynamics or optical-flow agreement and Dynamic Degree.

We agree that low flicker alone is insufficient to establish useful dynamics. We add two frozen visual probes, detailed in Appendix B.1. RAFT-Small estimates optical flow; horizontal/vertical motion and radial expansion are then used as directional motion proxies. VGGT estimates camera poses, and the relative transformations between them provide yaw and vertical/forward displacement. Motion signals from generated videos and their paired ground-truth videos are discretized into negative, stationary, and positive directions. Stationary offsets and thresholds are calibrated for each sequence using the other 19 ground-truth videos. Directional balanced agreement is averaged over videos and motion channels.

The results in Table 5 are:

MethodOptical-flow agreement (%) ↑VGGT agreement (%) ↑
Diffusion Forcing39.8639.14
VRR-SIGReg42.8853.53
VRR-Unif43.9152.61

Both variants achieve higher scores under both probes. Under VGGT, SIGReg and Unif gain 14.39 and 13.47 percentage points, respectively. The paired bootstrap confidence intervals over videos exclude zero for the improvement of each VRR variant over DF under both probes. We report these as motion-based action-following proxies, rather than VPT inverse-dynamics accuracy.

We also report Dynamic Degree across eight inference seeds in Appendix F.1 (Figure 14 and Table 13): DF scores 49.38 ± 8.21, versus 100.00 ± 0.00 for both VRR variants on the 0–100 scale. These results address the static-output concern together with directional motion agreement. Further visual comparisons are provided in Appendix F.2, Figures 15–18, and we will update the supplementary videos.

W4: Evaluation uses only 20 sequences and one run, and the layer ablation appears to use the same sequences.

Q3: Please report mean and standard deviation over several seeds and results on sequences excluded from layer/coefficient selection.

We add eight independently seeded inference runs and report their mean and sample standard deviation (Appendix F.1, Table 13 and Figure 14):

MethodAQ ↑IQ ↑Dynamic Degree ↑
Diffusion Forcing38.55 ± 0.7844.17 ± 0.7349.38 ± 8.21
VRR-SIGReg55.16 ± 0.2869.15 ± 0.50100.00 ± 0.00
VRR-Unif53.87 ± 0.5471.84 ± 0.65100.00 ± 0.00

The quality gains are large relative to inference-seed variation. These runs evaluate inference randomness for the trained models.

The original evaluation sequences are used for layer selection. After freezing the selected layers and coefficients, we evaluate 100 new sequences that were not used for selection (Appendix F.1). Table 14 reports:

MethodAQ ↑IQ ↑
Diffusion Forcing35.3942.48
VRR-SIGReg53.8967.08
VRR-Unif51.3769.08

The advantage persists on the new prompts: SIGReg improves AQ/IQ by 18.50/24.60 points over DF, and Unif by 15.98/26.60 points. This shows that the improvement extends beyond the sequences used for layer selection.

M1.1: More training does not necessarily worsen rollouts, and checkpoints on a fixed dataset do not isolate dataset size.

We have revised the Introduction and Section 3.3 to use the narrower conclusion that more training alone does not reliably resolve compounding error in this setting. Higher erank in an earlier checkpoint does not necessarily imply higher visual quality, since denoising fit also matters. The original experiment increased training steps and exposure to training windows together under the epoch definition inherited from Diffusion Forcing; overlapping clips can reuse frames.

To test dataset size separately, we train on 3k, 6k, and 18k sequences at matched steps (Appendix E.1, Table 8):

Training steps3k sequences: AQ / IQ6k sequences: AQ / IQ18k sequences: AQ / IQ
16,00034.22 / 48.4429.25 / 45.3438.65 / 44.37
40,00037.93 / 53.1942.22 / 54.8138.10 / 52.20

A sixfold increase does not produce a consistent gain. We have revised the Introduction and the Conclusion and Discussion to limit this finding to the tested Minecraft setting; computational constraints prevent extending the dataset-size study to other domains at present.

M1.2: SSIM/PSNR/LPIPS also change at collapse in the video, whereas Figure 2 plots them only for the stable checkpoint.

Our intended distinction is between collapse and divergence from the particular recorded continuation. Reference-based metrics respond to both, so a decrease does not uniquely identify collapse. The stable rollout in Figure 2 illustrates this ambiguity: a plausible video can receive poorer reference scores as its trajectory diverges from ground truth. We have revised Section 3.2 to clarify this point and present erank as an internal representation diagnostic, without claiming that reference-based metrics cannot respond to collapse. Their comparative results are now provided in Table 2.

M2: Direct rank maximization does not help, VRR-model rank is missing, and the frame-copying shortcut hypothesis is untested.

In Appendix D.1, Figure 9 compares DF, VRR-SIGReg, and VRR-Unif during inference, with erank averaged over 20 videos at layers 7, 11, and 15. Both VRR variants maintain higher late-rollout erank than DF and become comparatively stable after an initial decline.

Direct erank regularization also maintains high rank (Appendix D.2, Figure 10), but its AQ/IQ at 16k is only 37.05/37.49 versus DF's 38.65/44.37. As shown in Figure 11, its diffusion loss remains higher than that of SIGReg and Unif. Thus, even when the rank objective is successfully optimized, the model can fit the denoising task less effectively. High rank alone does not guarantee good generation; dimensional collapse is not the only relevant factor. RankMe [1] provides related work on rank-based representation assessment.

Frame copying is a hypothesis, as acknowledged in our limitations. The added representation curves and regularization experiments support the connection between representation dynamics and rollout stability, while the proposed shortcut mechanism remains to be investigated.

M3: VRR is trained only to 16k while the degradation argument relies on 40k-step DF.

We extend both VRR variants to 40k steps, alongside DF and Frame Anchor (Appendix E.2, Table 9). At the common 40k endpoint, AQ/IQ is 38.10/52.20 for DF, 39.43/53.37 for Frame Anchor, 53.44/66.57 for SIGReg, and 54.51/69.43 for Unif. Both variants retain substantial gains. Table 9 also reports the 24k and 32k checkpoints.

The initial 16k endpoint was chosen because the VRR quality curves in Figure 6 already showed little visible decline during long rollouts. The longer training run confirms the result beyond that initial budget without assuming that quality increases monotonically with training.

M4: Neither regularizer has a formula; Unif lacks a citation and resembles DispLoss; the identical Barlow Twins/VICReg scores may be erroneous.

Please see Appendix D for the formulas used to implement SIGReg [2] and Unif. We cite Wang and Isola [3] for the uniformity motivation and clarify that our squared inner-product objective differs from their exponential-distance formulation. We also discuss its relationship to DispLoss [4]. Our contribution is to characterize and mitigate representation degradation during long autoregressive rollouts using existing regularization principles.

The identical Barlow Twins/VICReg values were caused by invalid rollout evaluations: training instability led to NaNs during generation. We reran inference at higher precision, resolving the NaN issue, and the corrected 16k results in Table 3 are:

RegularizerAQ ↑IQ ↑
Barlow Twins28.1538.65
VICReg29.4331.16

The correction leaves the main conclusion unchanged. Appendix D specifies the ablation implementations and settings, and we will include the code in the supplementary material.

Reference:

  1. Garrido et al. RankMe: Assessing the Downstream Performance of Pretrained Self-Supervised Representations by Their Rank. ICML 2023.
  2. Balestriero and LeCun. LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics. 2025.
  3. Wang and Isola. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. ICML 2020.
  4. Wang and He. Diffuse and Disperse: Image Generation with Representation Regularization. 2025.

Dear Reviewer Fg7i,

Thank you again for your thoughtful review. With one day left in the discussion period, we would appreciate it if you could take a look at our response and let us know whether it addresses your concerns. We’d be happy to clarify any remaining points.

Best regards, The Authors

Reviewer_s37sOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies error accumulation (drift) in autoregressive video diffusion world models. It reports that the effective rank (erank) of DiT hidden states drops sharply when a long rollout collapses, and that training a Diffusion Forcing model for longer on a fixed Minecraft dataset lowers erank, which the authors interpret as data scaling failing to prevent drift. It then proposes video representation regularization (VRR), which adds a representation regularizer (SigReg or a uniformity loss) on DiT hidden states to the Diffusion Forcing loss. On 1-minute Minecraft rollouts with a ~500M-parameter DiT and 20 evaluation sequences, VRR improves VBench aesthetic quality from 38.65 to 53.78 and imaging quality from 44.37 to 72.08 over Diffusion Forcing at 16k training steps; a smaller experiment trains a UNet on RealEstate10K.

Critical strengths

1. Important problem and an interesting direction. Error accumulation is a central obstacle for long-horizon video world models. Connecting it to the degradation of hidden representations, and regularizing those representations during training, is an interesting direction, and the method is simple: one regularization term added to the Diffusion Forcing loss.

2. Promising long-rollout qualitative results. In the supplementary videos, Diffusion Forcing, VRR-SigReg, and VRR-Unif are similar for the first 10 s, and all start to show repetitive, blurred textures and object boundaries at 10–20 s. From 20 s onward, Diffusion Forcing collapses into mostly one or two colors without semantic meaning, while both VRR variants mostly maintain the quality reached at 20 s, with SigReg better than Unif. This result is promising, but it needs the training details requested below before it can be relied on.

Critical weaknesses

1. The main claims are not supported by the experiments, and several are contradicted by the paper's own results. The paper feels like the claims were pieced together after running the experiments, rather than the experiments being designed to test hypotheses.

  • "Data scaling fails" (Section 3.3; abstract: "contradicting mainstream scaling paradigms"). The evidence is checkpoints within a single "epoch" on the fixed 18k-sequence dataset, so the amount of data seen is the same axis as the number of training steps, and the experiment cannot separate more data from longer training. The paper's "epoch" (about 40k steps at batch size 512) only matches counting every overlapping 20-frame window as a separate sample. Assuming 20-frame training clips, each frame has been seen about 1.9 times by step 4k, 7.6 times by 16k, and 19 times by 40k, so "a single epoch ... to rule out overfitting" does not hold. Under the common per-video definition, one epoch is about 35 steps, and 40k steps is over 1,000 epochs. A valid test would train on datasets of different sizes for the same number of steps.
  • "More training accelerates degradation" (Fig. 3). Fig. 3 (right) shows the opposite for quality: later Diffusion Forcing checkpoints keep aesthetic quality at about 0.52 until 10–15 s, while the 4k checkpoint drops to about 0.35 by 10 s. Fig. 6 shows the same pattern. Erank (Fig. 3, left) and quality therefore disagree across checkpoints.
  • Erank as a "uniquely informative indicator" (Section 3.2). The comparison with SSIM, PSNR, and LPIPS (Fig. 2, bottom row) uses the stable 4k rollout, not the collapsing rollout in the top row, so the claim that "none of these alternatives consistently pinpoints the collapse frame" is not shown. The claim is also unnecessary: collapse should be detectable by quality metrics, and that would not undermine using erank to motivate the regularizer.
  • "Erank tracks drift, so regularizing hidden states solves drift." Directly regularizing erank performs no better than the baseline (Table 2: 37.05 / 37.49 vs. 38.65 / 44.37 at 16k), Barlow Twins and VICReg are unstable, and only SigReg and Unif work. The erank regularizer itself (sign, weight, layers) is not specified, and the erank of the VRR-trained models is never reported, so the link from regularizer to higher erank to less drift is not demonstrated.
  • Text that does not match the tables. Table 1's Diffusion Forcing row (aesthetic 39.57 / 37.52 / 36.82 / 38.65, imaging 46.20 / 33.78 / 39.46 / 44.37 at 4k / 8k / 12k / 16k) is flat and non-monotonic, not "a matching declining trend across all VBench scores". "Consistent performance gains as training proceeds" is shown only up to 16k steps, about 40% of the paper's own epoch. Appendix B.2 says gains hold "regardless of the regularization coefficient", but the table varies layers (likely a typo), and regularizing only layer 7 is worse than the baseline at 4k (imaging 24.10 vs. 46.20).

2. The experimental setup cannot establish the method's benefit.

  • Comparisons across training steps instead of final models. The main results compare methods at checkpoints between 4k and 16k steps, rather than at final models trained to a fixed budget and selected on a validation split. The whole-minute average also blends opposite trends: in Fig. 6, later Diffusion Forcing checkpoints collapse later but to a lower imaging score (about 0.17–0.38, vs. about 0.43 at 4k).
  • Missing training details. The optimizer, EMA, learning-rate schedule, and the VRR-Unif configuration are not given, so it is unclear whether the baselines were trained soundly; EMA in particular is standard in diffusion training. Together with the unusual definition of an epoch, this makes it hard to tell whether the baselines' failure to improve with training reflects unregularized hidden states or a lack of generic training stability. The strong VRR result needs these details to be trusted.
  • Narrow evaluation. Only VBench aesthetic and imaging quality are used, on 20 sequences. The dataset has ground-truth continuations and actions, but there is no ground-truth-referenced metric (PSNR, LPIPS, FVD) and no measure of action following. These two metrics also miss the repetitive, blurred textures visible in the supplementary videos at 10–20 s: VRR-SigReg's curves in Fig. 6 stay flat over that interval.
  • Missing baselines. Self Forcing (NeurIPS 2025) and BAgger (CVPR 2026) are cited but not compared. They would fit a comparison of final models, even if they are hard to fit into the across-steps tables.
  • Generality. The DiT uses separate spatial and temporal attention, unlike the mainstream full 3D attention, so it is unclear whether the erank observation and the regularization transfer to full-attention models. The RealEstate10K experiment switches to a UNet, uses 50 samples, and reports only curves.
Questions for authors
  1. (Clarification) Please give the full training setup for all methods: optimizer, EMA, learning-rate schedule, and, for VRR-Unif and the erank regularizer, the layers and weights, plus the sign of the erank objective.
  2. (Clarification) Please report the erank of the VRR-trained models and of the erank-regularized model. Does directly regularizing erank actually increase it?
  3. (Clarification) How do you reconcile Fig. 3 (left), where earlier checkpoints keep a higher erank, with Fig. 3 (right) and Fig. 6, where later checkpoints keep higher quality for longer?
  4. (Experiment) Can you test the data-scaling claim by training on datasets of different sizes for the same number of steps, and define how an epoch is counted?
  5. (Experiment) Can you compare all methods, including Self Forcing and BAgger, at final checkpoints trained to a fixed, longer budget and selected on a validation split, reporting quality over the 1-minute rollout, ground-truth-referenced metrics, and action following?
  6. (Experiment) Do the erank observation and VRR hold for a full 3D attention video model?
5 comments

We thank the reviewer for the careful analysis. We address the claims, experimental controls, and additional evaluations below. Figure, table, and appendix numbers refer to the current manuscript. AQ and IQ denote VBench Aesthetic Quality and Imaging Quality, respectively.

W1.1: "Data scaling fails" is inferred from checkpoints on the fixed 18k-sequence dataset; overlapping training windows mean that a single epoch does not rule out overfitting.

Q4: Can you train on datasets of different sizes for the same number of steps, and define how an epoch is counted?

Our epoch definition follows the official Diffusion Forcing implementation: the training unit is a 20-frame clip rather than an entire 1,200-frame recording. Different clips can overlap, so the reviewer is correct that their frames can be revisited. We have revised Section 3.3 and Appendix E.1 to clarify this distinction and avoid interpreting a window-based epoch as evidence that overfitting has been ruled out. We therefore add a separate dataset-size experiment with matched optimization steps.

The results are reported in Appendix E.1, Table 8:

Training steps3k sequences: AQ / IQ6k sequences: AQ / IQ18k sequences: AQ / IQ
16,00034.22 / 48.4429.25 / 45.3438.65 / 44.37
40,00037.93 / 53.1942.22 / 54.8138.10 / 52.20

Increasing the dataset sixfold does not consistently improve either metric. At 40k steps, for example, the 3k-sequence model already achieves IQ 53.19, compared with 52.20 using 18k sequences. These results support our observation that simply providing more trajectories does not reliably resolve the problem in this task. As stated in the limitations, computational constraints restrict this dataset-size study to Minecraft.

W1.2: "More training accelerates degradation" conflicts with the later checkpoints maintaining quality longer in Figures 3 and 6.

Q3: How do you reconcile higher erank in earlier checkpoints with higher quality lasting longer in later checkpoints?

We agree that "more training accelerates degradation" is too strong. We have revised the Introduction and Section 3.3 to state that more training alone does not reliably resolve compounding error in this setting. Figure 3 does show that the 40k checkpoint can collapse earlier and more severely than the 24k–32k checkpoints, but the comparison does not imply that degradation accelerates monotonically with training.

Erank and visual quality are not in one-to-one correspondence across checkpoints. Erank describes representation dimensionality, whereas good generation also requires the diffusion model to fit the denoising task accurately. An earlier checkpoint can retain higher erank while having poorer visual quality. We therefore use erank primarily to inspect representation changes within a rollout, rather than to rank different checkpoints by visual quality. The direct-erank experiment makes this distinction explicit: it maintains high rank but has higher diffusion loss and worse generation quality, as shown in Appendix D.2, Figures 10 and 11.

W1.3: The SSIM/PSNR/LPIPS comparison uses only the stable 4k rollout and does not establish erank as a "uniquely informative indicator."

Our intended point is that reference-based metrics cannot, by themselves, distinguish visual collapse from divergence away from the recorded continuation. A plausible generated trajectory can differ from the ground truth and receive lower PSNR/SSIM or higher LPIPS without collapsing. Figure 2's bottom row illustrates this ambiguity: a change in these metrics does not identify whether the video has collapsed or has followed a different plausible trajectory.

These metrics can also respond to collapse, so we have revised Section 3.2 to clarify that they do not uniquely distinguish collapse from plausible trajectory divergence. Erank provides a complementary view of the model's internal representations. We additionally report PSNR, SSIM, LPIPS, and FVD as comparative evaluation metrics in Table 2.

W1.4: Direct erank regularization does not improve quality, only SIGReg and Unif work, and the erank of VRR-trained models is missing.

Q2: Please report the erank of VRR and direct-erank models. Does directly regularizing erank actually increase it?

Yes. The direct-erank model sustains high rank during inference, as shown in Appendix D.2, Figure 10. However, at 16k steps its AQ/IQ is 37.05/37.49, compared with DF's 38.65/44.37. Figure 11 shows that its diffusion loss remains higher than that of SIGReg and Unif. Thus, successfully optimizing rank is insufficient if the model does not fit the denoising task well. This is consistent with treating dimensional collapse as an important factor rather than the sole determinant of generation quality. RankMe [1] also motivates examining rank as a representation diagnostic; our explanation of the denoising trade-off comes from the added training curves.

For VRR-SIGReg and VRR-Unif, Appendix D.1, Figure 9, compares their inference-time erank with that of DF over 20 videos at layers 7, 11, and 15. Both variants retain higher late-rollout erank and become comparatively stable after an initial decline, consistent with their improved long-rollout quality. The direct-erank counterexample also explains why arbitrary regularizers that increase rank need not improve generation. The precise loss definitions, signs, layers, and weights are given in Appendix D and detailed below. We will include the implementation in the supplementary material.

W1.5: The text does not match the non-monotonic DF results, VRR is evaluated only to 16k, and the layer ablation is incorrectly described as varying coefficients; layer 7 alone is worse at 4k.

W2.1: Comparisons should use final models at a fixed budget, and whole-minute averages hide different collapse timing and severity. In Fig. 6, later Diffusion Forcing checkpoints collapse later but to a lower imaging score.

Q5: Compare final models at a longer fixed budget, including Self Forcing and BAgger, with ground-truth metrics and action following.

We have revised Section 5.2 to describe DF's results as showing no consistent improvement with additional training, rather than a monotonic decline. We have also revised Appendix C.2 to clarify that the experiment varies the regularized layers. At 4k steps, layer-7-only regularization gives IQ 24.10 versus DF's 46.20. Introducing an additional objective can initially slow optimization of the diffusion objective, so improvements need not occur at every early checkpoint or with every layer selection.

We now provide longer runs in Appendix E.2, Table 9:

MethodAQ 16kAQ 24kAQ 32kAQ 40kIQ 16kIQ 24kIQ 32kIQ 40k
DF38.6538.5540.5138.1044.3752.4048.2752.20
Frame Anchor39.1540.1238.5639.4345.3850.9547.4353.37
VRR-SIGReg55.5649.3452.1253.4469.5163.0066.0666.57
VRR-Unif53.7852.8954.5354.5172.0868.2169.3369.43

Both VRR variants remain substantially ahead of DF and Frame Anchor at the fixed 40k endpoint. We initially evaluated up to 16k because the VRR curves in Figure 6 already showed little visible long-horizon degradation; the longer experiment confirms that the advantage persists beyond that budget.

The original comparison evaluates methods at matched training steps on the same evaluation sequences. We include intermediate checkpoints to show the training trajectory. Checkpoints are evaluated at scheduled steps; the original evaluation sequences are used for layer selection. After freezing the selected layers and coefficients, we additionally evaluate 100 new sequences that were not used for selection, obtaining AQ/IQ 35.39/42.48 for DF, 53.89/67.08 for SIGReg, and 51.37/69.08 for Unif (Appendix F.1, Table 14).

Collapse timing and severity are different properties: a model that collapses later can still end up with a lower quality score. Figure 6 therefore reports quality in successive 5-second segments in addition to the overall averages. The added ground-truth-referenced evaluation and action-following results are reported in Tables 2 and 5. Self Forcing and BAgger are discussed below with their respective protocols. Our BAgger reproduction uses 24k training steps with three collection rounds, targeting approximately the cost of 40k-step DF. A longer DF-SVI run is unavailable within the rebuttal time budget.

W2.2: Optimizer, EMA, learning-rate schedule, and VRR-Unif configuration are missing.

Q1: Please give the full training setup and the layers, weights, and sign of the direct-erank objective.

Appendices A and D provide the training and regularization settings:

SettingMinecraft configuration
OptimizerAdamW, betas (0.9, 0.99), weight decay 0.002
Learning rate8×10−58 \times 10^{-5}; 1,000-step linear warmup followed by cosine decay
EMANot used
Batch size64 per GPU on eight GPUs; global batch size 512
Backbone16 DiT blocks, hidden dimension 1,024, approximately 500M parameters
Regularized layersZero-indexed layers 0, 7, 15
SIGRegCoefficient 5×10−45 \times 10^{-4}; 256 random projections
UnifCoefficient 1 at each selected layer
Direct erankCoefficient 1 at each selected layer
Layer aggregationAverage of the weighted selected-layer losses

The baseline and VRR share the training settings; the representation regularizer is the added component. For direct erank, we center each frame's spatial feature matrix, normalize its singular values to obtain probabilities pjp_j, and minimize

Lerank=1−Eb,t[exp⁡(−∑jpjlog⁡pj)]min⁡(HW−1,D).\mathcal L_{\mathrm{erank}}=1-\frac{\mathbb E_{b,t}\left[\exp(-\sum_jp_j\log p_j)\right]}{\min(HW-1,D)}.

The sign therefore encourages larger, not smaller, rank. Unif normalizes the flattened token features and minimizes ∥U⊤U∥F2/Nu2\|U^\top U\|_F^2/N_u^2, the mean squared pairwise inner product including diagonal terms. SIGReg matches random-projection characteristic functions to a standard Gaussian. Appendix D specifies the token aggregation and numerical details. For RealEstate10K, the global batch size is 128, with the same learning rate and warmup; SIGReg and Unif coefficients are 0.02 and 10. We will provide implementation code in the supplementary material and subsequently release the code publicly.

W2.3: Evaluation is limited to two VBench metrics on 20 sequences, lacks reference metrics and action following, and flat curves can hide blurred or repetitive examples.

We add the following comparison against the recorded continuations, excluding the 100 prompt frames (main-text Table 2):

MethodPSNR (dB) ↑SSIM ↑LPIPS ↓FVD ↓
DF7.16680.36280.81504937.55
Frame Anchor7.27070.36570.80884091.36
DF-SVI7.37420.37640.78504828.16
VRR-SIGReg11.22920.38600.65071568.02
VRR-Unif10.48180.38490.63381685.97

Both VRR variants improve all four metrics over the baselines. FVD complements the frame-level reference metrics by comparing generated and ground-truth video-feature distributions.

We also evaluate motion-based action following with frozen RAFT-Small optical-flow and VGGT camera-pose probes (Appendix B.1, Table 5). Their directional balanced agreement with paired ground-truth motion is:

MethodOptical flow (%) ↑VGGT (%) ↑
DF39.8639.14
VRR-SIGReg42.8853.53
VRR-Unif43.9152.61

VRR achieves higher scores under both probes; the VGGT improvements are 14.39 and 13.47 percentage points. These are motion-based proxies, evaluated over the generated continuation, rather than inverse-dynamics action-label accuracy. Dynamic Degree and eight-seed results are additionally provided in Appendix F.1, Figure 14 and Table 13; Table 14 reports the 100 new evaluation sequences.

We agree that an average curve can hide artifacts in individual videos. For the supplied 000001.mp4, IQ is 68.33 over 10–15s and 66.87 over 15–20s, below the corresponding means of 70.32 and 69.64. Thus, a flat mean does not imply that every example remains equally good. Appendix F.2, Figures 15–18, provides additional qualitative comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.