acceptodds
Under review as a conference paper at ICLR 2027

Stream4D: 4D-Consistency for Streaming Autoregressive Diffusion Video Models

Abstract

Streaming autoregressive diffusion models enable real-time, long-horizon video generation, but their training objectives optimize local frame prediction rather than the geometry and dynamics of a coherent world: long rollouts accumulate geometric drift and degrade into static or unnatural motion. Recent bidirectional approaches address this problem using rewards signals built upon 3D Gaussian-Splatting reconstruction. However, a single rigid 3d reconstruction cannot model a dynamic scene, so this critic *penalizes* genuine object motion as reconstruction error and is maximized by freezing the video. This shortcut is especially detrimental in the AR setting, where each chunk can propagate an already-static configuration. In this work, we propose Stream4D, which replaces the static critic with a feed-forward 4D reconstruction reward that explicitly models scene dynamics, allowing coherent motion to receive high consistency rewards. To further guide motion magnitude and quality, we add a motion prior that rewards natural scene-flow magnitude while penalizing jitter and non-rigid artifacts. Our final recipe combines these two terms with a lightweight perceptual anchor. On the 5-second Self-Forcing and Causal-Forcing backbones and the 10-second LongLive backbone, Stream4D improves 4D reconstruction PSNR by 3.46, 5.53, and 6.76 dB, respectively, while better preserving motion dynamics and earning higher human preference. Videos can be found in the supplementary material.

Primary area:
generative models
Keywords:
Streaming Autoregressive Video Generation, 4D Consistency, Long-Horizon Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M9TAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Stream4D explores RL-based post-training for streaming autoregressive video diffusion models (evaluated on Self-Forcing, Causal-Forcing, and LongLive). Because standard static-scene 3D reconstruction rewards can inadvertently reward frozen or nearly static videos, the authors combine a dynamic 4D reconstruction reward (via MoVieS) with a motion prior (enforcing smoothness and local rigidity) and a perceptual reward. Quantitative and automated evaluations cover both a motion-prominent set and a random held-out subset. The human study is conducted on 50 prompts from the motion-prominent set using LongLive.

Critical strengths
  1. Pointing out that static 3D reconstruction rewards encourage motion suppression in video RL is a very practical observation, and the reward ablation studies clearly show why optimizing geometric consistency in isolation leads to degenerate, frozen outputs.

  2. The method is tested across three distinct autoregressive backbones rather than a single model, and the authors go beyond in-domain reward metrics by including 4DGT cross-checks, an external video reward model, position-debiased VLM evaluations, and human studies. Including a randomly sampled held-out prompt set alongside the main motion-heavy split also helps check against overfitting to high-motion prompts, and the appendix documents training hyperparameters and compute costs well.

Critical weaknesses
  1. While the paper attributes its gains to replacing a static reconstruction critic with a dynamic 4D critic, the experiments do not cleanly isolate this swap from the newly added motion reward. Because Stream4D modifies the reconstruction critic and introduces an explicit motion prior at the same time, we are missing the most direct control: keeping the exact same motion and perceptual rewards, normalization, and RL hyperparameters, and only swapping the static vs. dynamic reconstruction critic. Without this ablation, it is hard to tell how much of the improvement comes from the 4D critic itself versus the motion regularization.

  2. The primary reconstruction evaluation uses the same MoVieS model as the training reward and is therefore aligned with the optimized objective. The 4DGT cross-check provides useful additional evidence, although both pipelines share the camera estimator. These results support improved reconstructability, but do not by themselves establish independent geometric correctness.

  3. Several details regarding how World-R1 and VideoGPA were adapted from bidirectional diffusion to the shared distilled autoregressive trainer need clarification. While running them in a unified autoregressive setup makes sense, the paper should explicitly list which components, conditioning signals, and reward weights were kept, modified, or dropped, and how hyperparameters were tuned (noting that the World-R1 dynamic-only phase is included). Also, on the random prompt subset, VideoGPA appears to use a different weight configuration than the best-performing variant discussed in the main motion-prominent split. The submitted supplementary ZIP contains video examples and webpage assets, but no training or adapted-baseline source code or raw judgment records. The public project page also lists code as forthcoming. Releasing the exact implementations, prompt lists, adapters, and archived judgments would enable independent verification.

Questions for authors
  1. If you keep the motion prior, perceptual reward, and all training settings fixed, how much difference does switching solely between a static reconstruction critic and the dynamic MoVieS critic make? Also, how are coordinate frames, scale ambiguity, camera motion, and occlusions handled when computing the motion prior?

  2. When adapting World-R1 and VideoGPA to the autoregressive trainer, which exact components were modified, how were reward weights tuned on a separate validation split, and why does the random subset use a different VideoGPA variant?

  3. Table 2 compares human motion naturalness with an automated motion-preservation judgment, so the two columns need not agree. Could you discuss the observed difference in this light and provide the archived per-pair records and uncertainty for the human comparisons?

5 comments
AuthorsOpenReview ↗

Response to Reviewer M9TA Part I

We thank the reviewer for the detailed and constructive review. We answer your questions as follows:

Weakness 1 and Quesion 1: The 4D critic swap is not isolated from the new motion reward

Thanks for your valuable advice! It is very insightful. To isolate the effect of the motion term and validate the 4d design, we replaced the 4D MoVieS critic with the static 3D critic built by DA3-GIANT in World-R1 S_recon term while keeping all the other components identical. We evaluate on the 500 Table 1 prompts and report the LLM Judge motion score, consistenct score, overall VideoReward score and the Track failure rate described in Weakness 1 and Question 1.

BackboneCriticLLM Motion ↑LLM Consistency ↑VideoReward Overall vs base (%) ↑Track failure rate ↓
SFstatic 3D0.680.5347.20.89
SF4D (ours)0.830.8266.20.67
CFstatic 3D0.490.5354.20.77
CF4D (ours)0.770.7476.00.47
LLstatic 3D0.480.6077.00.94
LL4D (ours)0.710.7484.40.48

As shown in the table, replacing the 4d consistency critic with a static one solely harms the motion, consistency score and general videoreward judge. This is due to the fact that the 3d static critic is trained on the static scenes, so every moving object inside would be viewed as failure and punished.

Weakness 2: Primary evaluation uses the training reward's MoVieS; 4DGT shares the camera estimator

Thanks for raising this up! We added three evaluations that do not share the reward pipeline.

Point tracking. Following the photometric-consistency metric of WorldScore [1], which measures the forward–backward round-trip error of SEA-RAFT optical flow between consecutive frames, we chain the flow into long-range tracks. We evaluate it on the 500 motion-prominent test prompts of Table 1. From the first 81 frames of each video we take every second frame, start tracks from a uniform grid in the first frame (one point every 8 px), and chain the frame-to-frame flow forward over all 40 steps and then back. A point counts as a failure if it does not return to within 3 px of where it started; points that leave the image are excluded. The lower the failure rate, the more the motion agrees with the prior that the tracking model learned from natural videos. This cycle-failure rate drops on every backbone: 0.77 → 0.67 (SF), 0.84 → 0.47 (CF) and 0.69 → 0.48 (LL), validating the motion quality of our method.

Different cameras. To decouple the camera from our reward pipeline, we re-ran the MoVieS evaluation with three camera estimators that are different of StreamVGGT (VGGT-1B, MonST3R and DA3-GIANT), and evaluated it on the same 500 prompts used in Table 1:

CamerasMethod (MoVieS 4D-PSNR ↑, dB)SFCFLL
StreamVGGT (paper)Base16.8815.4417.44
World-R118.5219.1822.64
VideoGPA17.7518.0420.50
Ours20.3420.9724.20
VGGT-1BBase17.2315.5217.70
World-R118.8620.3024.06
VideoGPA18.2219.0922.33
Ours20.6622.0325.17
MonST3RBase17.5915.4017.86
World-R118.9519.6122.94
VideoGPA18.4618.8821.75
Ours20.9521.7823.82
DA3-GIANTBase18.0616.0318.50
World-R119.7020.0823.85
VideoGPA18.8518.9522.21
Ours21.3021.8825.32

Across all three different cameras, ours leads base, World-R1 and VideoGPA in all 27 comparisons.

Another pipeline. We report the results based on different pair of reconstructor and caremas, 4DGT and VGGT-1B cameras. Results show that ours leads base, World-R1 and VideoGPA on every backbone.

Ours − row, 4DGT PSNR, VGGT-1B camerasSFCFLL
vs base+0.91+2.24+4.68
vs World-R1+0.63+1.11+2.51
vs VideoGPA+1.11+2.34+2.86

Reward-blind MLLM judge. We use Gemini-3.5-Flash to judge whether our video is better than the base on two conditions at once: (A) it keeps roughly as much genuine object motion as the base; and (B) on that moving content, objects hold their identity, shape and structure better, with less morphing, warping, popping or flicker. Our win rate against base is 82.2% / 73.9% / 74.2% (SF / CF / LL), the highest of all methods on every backbone.

[1]Duan, Haoyi, et al. "Worldscore: A unified evaluation benchmark for world generation." Proceedings of the IEEE/CVF international conference on computer vision. 2025.

AuthorsOpenReview ↗

Response to Reviewer M9TA Part II

Weakness 3 and Question 2: Baseline adaptation details, VideoGPA variant on the random subset, and code/record release Adaptation. Both baselines were designed for bidirectional T2V models. We therefore reimplemented both on our three AR bases with the same Astrolabe online trainer, optimizer settings like LoRA configs and RL configs, training steps and training prompts. Only the rewards and each method's key design components are taken from the original work.

World-R1 We replace FLOW-GRPO-Fast with Astrolabe while retaining the original reward structure. Consistent with our method, we apply GDPO to normalize each reward component before summation, whereas the original implementation does not perform this normalization. We remove the S_traj reward because we donot have the camera descirption in the prompt. To accommodate our total budget of 150 training steps, we shorten each training cycle from 100 main-phase steps and 50 dynamic-phase steps to 20 and 10 steps, respectively. Finally, instead of computing the HPSv3 reward on a single randomly selected frame in their original recipe, we score all frames using HSPv2 and average the top 30% of frame scores to improve stability. VideoGPA (Table S2, standard). We use exactly the same reward, −3DCS, as the original method and adapt its DPO training alogrithm on preference pairs to our online group-relative RL algorithm. VideoGPA (Table 1, enhanced). We observe that the standard recipe in Table S2 quickly leads to nearly static videos. To address this, we incorporate the same aesthetic anchor and dynamic-phase schedule used in our World-R1 implementation. We apologize for not clearly describing these modifications in the paper.

We report the the results of standard and enhanced recipe for Table S2 tables as follows:

Random-500 table with both VideoGPA variants

BackboneMethodMoVieS PSNR ↑SSIM ↑LPIPS ↓4DGT PSNR ↑SSIM ↑LPIPS ↓LLM Motion ↑LLM Consist. ↑VR VQ ↑VR MQ ↑VR TA ↑VR Ovr ↑
SFBase18.190.8310.23916.230.5460.439——————
World-R119.350.8850.18216.600.5600.4120.77368.261.449.454.256.8
VideoGPA enhanced18.860.8960.17815.880.5220.4330.6853.450.643.458.250.2
VideoGPA standard24.880.9080.14319.870.6210.3960.47738.363.063.465.067.4
Ours21.230.8950.17717.700.6570.4110.81675.159.856.654.459.8
CFBase16.300.8020.25214.940.5120.455——————
World-R119.670.8940.16216.340.5750.4170.63761.172.660.657.065.8
VideoGPA enhanced18.380.8820.18614.900.5060.4960.6749.769.258.261.865.4
VideoGPA standard22.940.8430.14719.330.6050.3770.48436.567.654.853.462.2
Ours20.990.8900.15617.490.6500.4040.71664.971.465.852.868.6
LLBase18.110.8630.21715.450.5090.447——————
World-R123.010.9400.13717.530.6300.4380.45746.874.868.663.475.8
VideoGPA enhanced21.180.9390.14517.210.6070.3910.4535.863.252.056.258.2
VideoGPA standard22.990.9430.13019.240.6680.3390.1369.866.063.654.667.0
Ours24.600.9100.14719.590.6460.4170.70168.574.876.665.478.2
  • The standard recipe is the stronger reconstructor but the weaker video model. On the random subset it reaches the good reconstruction scores by freezing the scene. Stream4D achieves higher motion-preservation and joint motion–consistency scores than both VideoGPA configurations on every backbone in both sets.
AuthorsOpenReview ↗

Response to Reviewer M9TA Part III

Question 1: motion-prior handling of coordinate frames, scale, camera motion and occlusion

Motion prior. The motion prior is computed from MoVieS's per-pixel 3D trajectories.

  • Coordinate frame and camera motion. StreamVGGT first estimates per-frame cameras, which MoVieS takes as input. MoVieS then predicts each pixel's 3D trajectory in the shared world frame, and m is the frame-to-frame displacement along these trajectories. Under a moving camera, a static background stays fixed in the world frame and contributes near-zero motion, so m measures object motion.
  • Scale. StreamVGGT normalizes each video's reconstruction to a canonical scene size, so m measures displacement relative to the scene's own extent. This makes m comparable across close-ups and wide shots without metric scale.
  • Occlusion and unreliable regions. The motion and smoothness terms are weighted by MoVieS's per-pixel confidence, so occluded, disoccluded and low-confidence pixels contribute little.

Question 3: Human motion naturalness vs automated motion preservation We agree that the two columns measure different things.

  • The MLLM judge measures preservation. It asks whether the amount of motion is kept, reduced or lost.
  • Human raters judge naturalness. They pick which clip's motion looks more natural, which also weighs smoothness, physical plausibility and visual quality.

Read this way, the two columns are consistent: Against base, the motion is preserved. Against the geometry-RL baselines, ours outperforms in both motion preservation and quality.

Question 3: per-pair records and uncertainty The 95% CIs come from a 10,000-sample bootstrap over trials. The results are as follows:

Ours vsMotion W / T / LMotion win%Overall W / T / LOverall win%
base15 / 13 / 2243.0 [31, 55]21 / 18 / 1160.0 [49, 71]
World-R128 / 16 / 672.0 [62, 81]28 / 20 / 276.0 [68, 84]
VideoGPA39 / 9 / 287.0 [79, 94]33 / 14 / 380.0 [71, 88]

As shown form the table, our method beats World-R1 and VideoGPA by a large margin.

Reviewer_RdQvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a reinforcement learning framework called Stream4D for improving 4D consistency in AR video generation. The authors identify that existing static 3D reconstruction-based rewards can penalize genuine object motion, encouraging models to generate overly static videos. To address this, Stream4D introduces a 4D reconstruction reward using a pretrained 4DGS model to evaluate the geometric and temporal consistency of generated videos. Additionally, the method incorporates a motion prior that encourages natural motion magnitude and a perceptual reward to preserve visual quality. These rewards are combined to fine-tune AR video models using reinforcement learning. Experiments on multiple backbones show improvements in 4D reconstruction quality and motion preservation.

Critical strengths

The paper identifies an important problem in AR video generation: existing geometry-based rewards can be misaligned with the goal of generating realistic and dynamic videos. In particular, the paper shows that optimizing static 3D reconstruction rewards can encourage models to suppress genuine object motion rather than improve video quality. Since AR video generation relies on previous frames, such errors can propagate and accumulate over generation.

Critical weaknesses

W1. The paper primarily evaluaties how accurately a 4D reconstruction model can reconstruct and re-render the generated video from the same camera viewpoints. However, I do not agree that high reconstruction fidelity leads to better 3D geometric or temporal consistency. And the independent 4DGT evaluation shares the same camera estimation model (StreamVGGT). So, I think the claimed improvement in 4D consistencty is not sufficiently validated.

W2. To my understanding, the proposed method combines 4D reconstruction with an explicit motion prior, but the current experiments do not isolate whether the improvements actually originate from the 4D representation itself or the additional motion regularization. For example, the paper does not compare agianst a static 3D reconstruction reward while using the same motion prior. Since the ablation results show that removing the motion reward reduces motion preservation, such a comparison would be important to establish the specific contribution of the proposed 4D reconstruction reward.

Questions for authors

Q1. Regarding W1, could the authors provide more direct evidence that the improvements in reconstruction fidelity translate into better 3D geometric and temporal consistency? For example, would it be possible to evaluate cross-frame point correspondence or long-term geometric consistency using metrics that do not rely on the same reconstruction pipeline?

Q2. Regarding W2, could the authors compare the proposed method against a static 3D reconstruction reward under identical training settings? It would help clarify whether the improvements primarily come from the 4D reconstruction reward or the additional motion regularization.

Q3. Since the paper emphasizes error accumulation in AR video generation, could the authors provide evaluations on longer generation horizons beyond 10 seconds? It would be interesting to see whether the improvements in geometric consistency and motion preservation persist as the number of generated chunks increases.

Q4. The proposed motion prior uses a fixed target motion magnitude derived from the base model. It relies on a strong assumption that a single target is suitable across different scenes and prompts. Considering that natural motion magnitudes can vary substantially across different types of motion (e.g., slow object movements versus fast dynamic actions), how well does this fixed target generalize across diverse scenarios? This is a point of curiosity regarding the method's simplicity.

5 comments
AuthorsOpenReview ↗

Response to Reviewer RdQv Part I

We thank the reviewer for recognizing the importance of reward misalignment in AR generation. We answer your questions as follows:

Weakness 1 and Question 1: the evaluation is coupled with the training reward Thanks for raising this up!

First, we added three evaluations that do not share the reward pipeline to validate our design.

Point tracking. Following the photometric-consistency metric of WorldScore [1], which measures the forward–backward round-trip error of SEA-RAFT optical flow between consecutive frames, we chain the flow into long-range tracks. We evaluate it on the 500 motion-prominent test prompts of Table 1. From the first 81 frames of each video we take every second frame, start tracks from a uniform grid in the first frame (one point every 8 px), and chain the frame-to-frame flow forward over all 40 steps and then back. A point counts as a failure if it does not return to within 3 px of where it started; points that leave the image are excluded. The lower the failure rate, the more the motion agrees with the prior that the tracking model learned from natural videos. This cycle-failure rate drops on every backbone: 0.77 → 0.67 (SF), 0.84 → 0.47 (CF) and 0.69 → 0.48 (LL), validating the motion quality of our method.

Different cameras. To decouple the camera from our reward pipeline, we re-ran the MoVieS evaluation with three camera estimators that are different form StreamVGGT (VGGT-1B, MonST3R and DA3-GIANT), and evaluated it on the same 500 prompts used in Table 1:

CamerasMethod (MoVieS 4D-PSNR ↑, dB)SFCFLL
StreamVGGT (paper)Base16.8815.4417.44
World-R118.5219.1822.64
VideoGPA17.7518.0420.50
Ours20.3420.9724.20
VGGT-1BBase17.2315.5217.70
World-R118.8620.3024.06
VideoGPA18.2219.0922.33
Ours20.6622.0325.17
MonST3RBase17.5915.4017.86
World-R118.9519.6122.94
VideoGPA18.4618.8821.75
Ours20.9521.7823.82
DA3-GIANTBase18.0616.0318.50
World-R119.7020.0823.85
VideoGPA18.8518.9522.21
Ours21.3021.8825.32

Across all three different cameras, ours leads base, World-R1 and VideoGPA in all 27 comparisons.

Another pipeline. We report the results based on different pair of reconstructor and caremas, 4DGT and VGGT-1B cameras. Results show that ours leads base, World-R1 and VideoGPA on every backbone.

Ours − row, 4DGT PSNR, VGGT-1B camerasSFCFLL
vs base+0.91+2.24+4.68
vs World-R1+0.63+1.11+2.51
vs VideoGPA+1.11+2.34+2.86

Reward-blind MLLM judge. We use Gemini-3.5-Flash to judge whether our video is better than the base on two conditions at once: (A) it keeps roughly as much genuine object motion as the base; and (B) on that moving content, objects hold their identity, shape and structure better, with less morphing, warping, popping or flicker. Our win rate against base is 82.2% / 73.9% / 74.2% (SF / CF / LL), the highest of all methods on every backbone.

[1]Duan, Haoyi, et al. "Worldscore: A unified evaluation benchmark for world generation." Proceedings of the IEEE/CVF international conference on computer vision. 2025.

AuthorsOpenReview ↗

Response to Reviewer RdQv Part II

Weakness 1: Can reconstruction error reflect the gemoetry consistency?

Using 3D reconstruction to measure the geometric consistency of generated videos is an established practice. The two 3D-consistency baselines we compare against, VideoGPA [1] and World-R1 [2], both derive their training signal from 3D reconstruction of the generated video. Geometry Forcing [3] use the reprojection error of a DROID-SLAM reconstruction, MEt3R [4] uses DUSt3R reconstructions to measure multi-view consistency, and CamCo [5] use COLMAP reconstruction success and reprojection error. Our reward follows this line but uses a 4D reconstructor, so that genuine object motion is not penalized as inconsistency. Our key insight is that 4D reconstruction models are trained on large-scale natural videos, so their learned prior reflects how real scenes look and move. The more natural a video's content, the better it fits this prior, the more predictable it is to reconstruct, and the lower its reconstruction error.

To link reconstruction error directly to perceived quality, we evaluate with three judges that are independent of our reward:

  • VideoReward. A learned human-preference model rates ours above base overall on 66.2% / 76.0% / 84.4% of prompts (SF / CF / LL).
  • MLLM judge. Ours beats base on 82.2% / 73.9% / 74.2% of prompts (SF / CF / LL; Table 1).
  • Human study. Raters prefer ours over base in 60.0% with ties counted ½, and judge it no worse in 78%. Against the 3D-consistency baselines, ours is preferred in 80% of pairs against VideoGPA and 76% against World-R1.

[1] Du, Hongyang, et al. "Videogpa: Distilling geometry priors for 3d-consistent video generation." arXiv preprint arXiv:2601.23286 (2026). [2] Wang, Weijie, et al. "World-r1: Reinforcing 3d constraints for text-to-video generation." arXiv preprint arXiv:2604.24764 (2026). [3] Wu, Haoyu, et al. "Geometry forcing: Marrying video diffusion and 3d representation for consistent world modeling." International Conference on Learning Representations. Vol. 2026. 2026. [4] Asim, Mohammad, et al. "Met3r: Measuring multi-view consistency in generated images." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025. [5] Xu, Dejia, et al. "Camco: Camera-controllable 3d-consistent image-to-video generation." arXiv preprint arXiv:2406.02509 (2024).

Weakness 2 and Question 2: 4D representation vs motion regularization not isolated

Thanks for your valuable advice! It is very insightful. To isolate the effect of the motion term and validate the 4d design, we replaced the 4D MoVieS critic with the static 3D critic built by DA3-GIANT in World-R1 S_recon term while keeping all the other components identical. We evaluate on the 500 Table 1 prompts and report the LLM Judge motion score, consistenct score, overall VideoReward score and the Track failure rate described in Weakness 1 and Question 1.

BackboneCriticLLM Motion ↑LLM Consistency ↑VideoReward Overall vs base (%) ↑Track failure rate ↓
SFstatic 3D0.680.5347.20.89
SF4D (ours)0.830.8266.20.67
CFstatic 3D0.490.5354.20.77
CF4D (ours)0.770.7476.00.47
LLstatic 3D0.480.6077.00.94
LL4D (ours)0.710.7484.40.48

As shown in the table, replacing the 4d consistency critic with a static one solely harms the motion, consistency score and general videoreward judge. This is due to the fact that the 3d static critic is trained on the static scenes, so every moving object inside would be viewed as failure and punished.

AuthorsOpenReview ↗

Response to Reviewer RdQv Part III

Question 3: Longer horizons beyond 10 s

Thanks for pointing it out! We generated LongLive rollouts of 20, 30 and 60 s on the 500 prompts used in Table 1 and report the last 81-frame clip as follows.

HorizonMethodPSNR ↑LLM Motion ↑LLM Consistency ↑VideoReward vs base (%) ↑
20 sbase16.79———
World-R123.460.430.5477.6
VideoGPA21.470.380.3974.0
Ours24.720.670.7282.6
30 sbase16.53———
World-R123.550.440.5680.0
VideoGPA21.760.370.4282.0
Ours24.740.660.7282.4
60 sbase16.51———
World-R123.420.450.5780.2
VideoGPA21.440.400.4377.8
Ours24.710.650.7285.6
  • Base accumulates error and ours does not. From the first to the last clip, base PSNR loses 0.72 / 0.87 / 0.99 dB at 20 / 30 / 60 s, while ours gains 0.47 / 0.50 / 0.46 dB.
  • Ours leads both baselines on the last clip at every horizon. Ours' PSNR is higher than World-R1 by +1.26 / +1.19 / +1.29 dB and higher than VideoGPA by +3.25 / +2.98 / +3.27 dB . Ours also has the highest LLM Motion, LLM Consistency and VideoReward win rate against base at every horizon.

Question 4: Does a single fixed m_nat generalize across slow and fast motion?

Thanks for your question! It is very interesting.

Why we choose fixed target in our design StreamVGGT normalizes each video's reconstruction to a canonical scene size, so m is the displacement relative to the scene's own extent. A single m_nat therefore asks every video for the same relative amount of motion, which is comparable across close-ups and wide shots and across slow and fast scenes. In addition, the gate is soft: it is a Gaussian with σ = 0.010 around m_nat = 0.020, so it discourages only the extremes and give little presure to the nearby reigon.

The fixed target does not suppress high-motion videos. We split the 500 Table 1 prompts at the median of the base video's motion into a high-motion and a low-motion half, and compare the paper's Gemini LLM Motion score between the two halves:

LLM Motion ↑SFCFLLPooled
High-motion half0.8360.7650.6940.765
Low-motion half0.8300.7620.7150.769

On every backbone, ours keeps the base motion as well on high-motion prompts as on low-motion ones. There is no evidence that the fixed target might harm the high motion videos.

Reviewer_MVhKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Stream4D uses reinforcement learning to improve scene consistency in streaming autoregressive video generation while mitigating motion collapse. Its reward combines dynamic 4D reconstruction consistency, a motion prior, and an image-preference score. Experiments on three backbones report improved reconstruction metrics and overall preference, though these gains do not establish physically accurate geometry.

Critical strengths
  • Well-motivated reward design. The paper addresses motion suppression under static reconstruction rewards by combining dynamic reconstruction consistency with an explicit motion prior.

  • Complementary empirical validation. Experiments across three autoregressive backbones, reward-component ablations, an alternative reconstructor, and human evaluation provide evidence beyond improvements in the training reward alone.

Critical weaknesses
  • Limited methodological novelty. The optimization framework, preference reward, and reward normalization are inherited from prior work. The contribution primarily lies in adapting dynamic reconstruction and motion regularization, whose distinct advantages require clearer isolation.

  • Incomplete validation of perceptual consistency. The provided examples still exhibit questionable object interactions and apparent subject–background motion inconsistencies, such as the fish–cat interaction and the walking character. These artifacts raise concerns about whether improved reconstruction scores translate into perceptually coherent dynamics; their prevalence is not established by the current evaluation.

  • Insufficient validation of robustness and motion quality. Reconstruction-based evaluation remains partly coupled to the reward pipeline, while the small human study does not establish improved natural motion over the base model. Potential bias against complex but valid scenes remains insufficiently examined.

Questions for authors
  • How much of the improvement comes specifically from dynamic reconstruction? A controlled comparison replacing only the 4D reconstruction reward with a static 3D counterpart, while keeping the motion and perceptual rewards fixed, would clarify its independent contribution.

  • Does the reward penalize complex but visually valid scenes that are difficult to reconstruct? Analyzing existing cases where reconstruction scores disagree with independent human judgments would help identify this potential bias. Please also discuss the human motion preference of 43% against the base model.

9 comments
AuthorsOpenReview ↗

Response to Reviewer MVhK Part I

We thank the reviewer for the positive assessment of the reward design and of the empirical validation. We answer your questions as follows:

Weakness 1: Methodological novelty

Thanks for pointing this out. Our contribution comes from three parts:

  1. Showing that enforcing geometric consistency with a 4D reconstruction reward works for autoregressive video diffusion models, whereas the baselines primarily target bidirectional video diffusion models.
  2. We show that our reward improves 60 s long video generation while preserving motion intensity, which is a key problem in streaming video generation. More details are in our response to Reviewer RdQv, Question 3.
  3. We show that static 3D reconstruction critics reward frozen video in streaming video generation RL. We propose a 4D reconstruction critic with a gated motion prior that removes this shortcut and reward the motion quality.

Meanwhile, we also isolate the 4D critic from the motion prior in Question 1. With the motion prior held fixed, replacing the 4D critic with a static one brings motion collapse back, validating our design.

Weakness 2: Validation of perceptual consistency

We acknowledge that the fish–cat interaction in our result is not fully natural. Yet, we would like to draw the reviewer's attention to the corresponding base videos in the supplementary material, where the artifacts are more severe:

  • Cat running with a fish (second example in the supplementary). The base model's motion is overly strong and unnatural, with noticeable jittering. The remaining artifact in ours is an inconsistency that our method reduces but does not fully resolve.
  • Bee-like armored alien walking. In the base video, the character's motion is inconsistent with the background: the character appears to walk forward but drifts to the right, creating a noticeable floating effect. In our video, this drift is absent.

Beyond individual examples, we measured the consistency statistically:

  • MLLM judge. Ours beats base on 82.2% / 73.9% / 74.2% of prompts (SF / CF / LL), as shown in Table 1.
  • Human study. Raters preferred ours over base overall in 60.0% with ties counted ½ and judged it no worse in 78% of pairs. Against the 3D consistency baselines, ours is preferred overall in 80% of pairs against VideoGPA and 76% against World-R1.

Weakness 3: Robustness and motion quality

Thanks for raising this up! We added three evaluations that do not share the reward pipeline.

Point tracking. Following the photometric-consistency metric of WorldScore [1], which measures the forward–backward round-trip error of SEA-RAFT optical flow between consecutive frames, we chain the flow into long-range tracks. We evaluate it on the 500 motion-prominent test prompts of Table 1. From the first 81 frames of each video we take every second frame, start tracks from a uniform grid in the first frame (one point every 8 px), and chain the frame-to-frame flow forward over all 40 steps and then back. A point counts as a failure if it does not return to within 3 px of where it started; points that leave the image are excluded. The lower the failure rate, the more the motion agrees with the prior that the tracking model learned from natural videos. This cycle-failure rate drops on every backbone: 0.77 → 0.67 (SF), 0.84 → 0.47 (CF) and 0.69 → 0.48 (LL), validating the motion quality of our method.

Different cameras. To decouple the camera from our reward pipeline, we re-ran the MoVieS evaluation with three camera estimators that are different of StreamVGGT (VGGT-1B, MonST3R and DA3-GIANT), and evaluated it on the same 500 prompts used in Table 1:

CamerasMethod (MoVieS 4D-PSNR ↑, dB)SFCFLL
StreamVGGT (paper)Base16.8815.4417.44
World-R118.5219.1822.64
VideoGPA17.7518.0420.50
Ours20.3420.9724.20
VGGT-1BBase17.2315.5217.70
World-R118.8620.3024.06
VideoGPA18.2219.0922.33
Ours20.6622.0325.17
MonST3RBase17.5915.4017.86
World-R118.9519.6122.94
VideoGPA18.4618.8821.75
Ours20.9521.7823.82
DA3-GIANTBase18.0616.0318.50
World-R119.7020.0823.85
VideoGPA18.8518.9522.21
Ours21.3021.8825.32

Across all three different cameras, ours leads base, World-R1 and VideoGPA in all 27 comparisons.

[1]Duan, Haoyi, et al. "Worldscore: A unified evaluation benchmark for world generation." Proceedings of the IEEE/CVF international conference on computer vision. 2025.

AuthorsOpenReview ↗

Response to Reviewer MVhK Part II

Another pipeline. We report the results based on different pair of reconstructor and caremas, 4DGT and VGGT-1B cameras. Results show that ours leads base, World-R1 and VideoGPA on every backbone.

Ours − row, 4DGT PSNR, VGGT-1B camerasSFCFLL
vs base+0.91+2.24+4.68
vs World-R1+0.63+1.11+2.51
vs VideoGPA+1.11+2.34+2.86

Reward-blind MLLM judge. We use Gemini-3.5-Flash to judge whether our video is better than the base on two conditions at once: (A) it keeps roughly as much genuine object motion as the base; and (B) on that moving content, objects hold their identity, shape and structure better, with less morphing, warping, popping or flicker. Our win rate against base is 82.2% / 73.9% / 74.2% (SF / CF / LL), the highest of all methods on every backbone.

Question 1: How much of the improvement comes specifically from dynamic reconstruction? Thanks for your valuable advice! It is very insightful. To isolate the effect of the motion term and validate the 4d design, we replaced the 4D MoVieS critic with the static 3D critic built by DA3-GIANT in World-R1 S_recon term while keeping all the other components identical. We evaluate on the 500 Table 1 prompts and report the LLM Judge motion score, consistenct score, overall VideoReward score and the Track failure rate described in W3.

BackboneCriticLLM Motion ↑LLM Consistency ↑VideoReward Overall vs base (%) ↑Track failure rate ↓
SFstatic 3D0.680.5347.20.89
SF4D (ours)0.830.8266.20.67
CFstatic 3D0.490.5354.20.77
CF4D (ours)0.770.7476.00.47
LLstatic 3D0.480.6077.00.94
LL4D (ours)0.710.7484.40.48

As shown in the table, replacing the 4d consistency critic with a static one solely harms the motion, consistency score and general videoreward judge. This is due to the fact that the 3d static critic is trained on the static scenes, so every moving object inside would be viewed as failure and punished.

Question 2: Bias against complex but valid scenes, and the 43% human motion preference

1. Complex but valid scenes are reconstructed better, not penalized. We evaluate the 500 motion-prominent test prompts of Table 1 in groups of physically hard content: crowds, fluids/fire/cloth/hair and occlusion. We define complex-but-valid as prompts that combine at least two of these factors. The per category results are as follows.

GroupnMethod4D-PSNR ↑<br>SF / CF / LLLLM Motion ↑<br>SF / CF / LLLLM Consist. % ↑<br>SF / CF / LL
All prompts500Base16.88 / 15.44 / 17.44——
World-R118.52 / 19.18 / 22.640.832 / 0.676 / 0.49875.9 / 69.1 / 54.0
VideoGPA17.75 / 18.04 / 20.500.737 / 0.706 / 0.30660.9 / 60.4 / 32.9
Ours20.34 / 20.97 / 24.200.833 / 0.765 / 0.70682.2 / 73.9 / 74.2
Complex but valid26Base16.85 / 15.52 / 17.14——
World-R118.20 / 18.60 / 21.640.856 / 0.663 / 0.51065.4 / 72.1 / 51.0
VideoGPA16.71 / 17.50 / 19.200.625 / 0.702 / 0.47158.7 / 62.5 / 35.6
Ours20.42 / 20.28 / 23.200.875 / 0.798 / 0.71284.6 / 68.3 / 61.5
Crowds (≥3 agents)36Base17.61 / 16.25 / 17.74——
World-R118.65 / 19.42 / 21.780.819 / 0.701 / 0.52866.7 / 74.3 / 43.1
VideoGPA17.87 / 17.87 / 19.820.583 / 0.715 / 0.45847.9 / 61.8 / 34.0
Ours20.47 / 20.80 / 23.420.875 / 0.812 / 0.70186.8 / 57.6 / 59.0
Fluids, fire, cloth, hair82Base17.27 / 15.68 / 17.60——
World-R119.12 / 18.97 / 22.800.817 / 0.640 / 0.51877.4 / 68.9 / 54.9
VideoGPA17.89 / 17.78 / 20.420.622 / 0.704 / 0.39650.9 / 59.5 / 39.3
Ours20.90 / 20.71 / 24.400.832 / 0.790 / 0.72379.9 / 79.9 / 72.6
Occlusion40Base17.00 / 15.92 / 16.55——
World-R118.02 / 19.55 / 21.580.856 / 0.675 / 0.58865.6 / 63.7 / 62.5
VideoGPA16.85 / 17.96 / 19.300.762 / 0.681 / 0.50060.0 / 55.0 / 48.1
Ours20.09 / 21.31 / 22.580.894 / 0.681 / 0.80682.5 / 61.3 / 76.9

As shown in the table, 4D-PSNR is best in every group on every backbone: on complex-but-valid scenes, ours gains +3.57 / +4.76 / +6.06 dB over base and +2.22 / +1.68 / +1.56 dB over World-R1. The MLLM judge also ranks ours highest on consistency in every group on SF and LL.

AuthorsOpenReview ↗

Response to Reviewer MVhK Part III

2. The main failure is lost motion. and there is no siginficant failure mode in the complex but vivid group. We took every variant–base pair whose MoVieS 4D-PSNR differs by more than 0.5 dB: 500 prompts × 3 backbones × {ours, World-R1, VideoGPA vs base}. A disagreement is a pair where the MLLM judge prefers the lower-scoring video. The table reports the conflict rate:

Conflict rate (%) ↓Complex-but-valid<br>SF / CF / LLPooledOther prompts<br>SF / CF / LLAvg
World-R118.8 / 11.5 / 38.523.513.5 / 20.4 / 33.023.2
VideoGPA30.0 / 25.0 / 80.048.028.0 / 26.6 / 55.938.5
Ours8.0 / 19.2 / 30.819.510.0 / 15.5 / 14.413.4

Ours has the fewest avg conflicts both on complex-but-vivid scenes and on the other prompts. It shows that our method are not bad at complex and vivid scenes.

Furthermore, we found the disagreements are mainly due to lost motion. As the motion judge marks 59% / 75% / 95% of the disagreeing pairs as having reduced or lost motion (SF / CF / LL). In the agreeing pairs the share is only 23% / 34% / 52%.

3. The 43% human motion preference is not a significant loss.

The per-pair records give the following results with 95% CIs:

Ours vsMotionOverall
base43.0% [31, 55]60.0% [49, 71]
VideoGPA87% [79, 94]80% [71, 88]
World-R172% [62, 81]76% [68, 84]

We cannot conclude the 43% preference rate is a clear loss.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.