- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Stream4D explores RL-based post-training for streaming autoregressive video diffusion models (evaluated on Self-Forcing, Causal-Forcing, and LongLive). Because standard static-scene 3D reconstruction rewards can inadvertently reward frozen or nearly static videos, the authors combine a dynamic 4D reconstruction reward (via MoVieS) with a motion prior (enforcing smoothness and local rigidity) and a perceptual reward. Quantitative and automated evaluations cover both a motion-prominent set and a random held-out subset. The human study is conducted on 50 prompts from the motion-prominent set using LongLive.
Critical strengths
-
Pointing out that static 3D reconstruction rewards encourage motion suppression in video RL is a very practical observation, and the reward ablation studies clearly show why optimizing geometric consistency in isolation leads to degenerate, frozen outputs.
-
The method is tested across three distinct autoregressive backbones rather than a single model, and the authors go beyond in-domain reward metrics by including 4DGT cross-checks, an external video reward model, position-debiased VLM evaluations, and human studies. Including a randomly sampled held-out prompt set alongside the main motion-heavy split also helps check against overfitting to high-motion prompts, and the appendix documents training hyperparameters and compute costs well.
Critical weaknesses
-
While the paper attributes its gains to replacing a static reconstruction critic with a dynamic 4D critic, the experiments do not cleanly isolate this swap from the newly added motion reward. Because Stream4D modifies the reconstruction critic and introduces an explicit motion prior at the same time, we are missing the most direct control: keeping the exact same motion and perceptual rewards, normalization, and RL hyperparameters, and only swapping the static vs. dynamic reconstruction critic. Without this ablation, it is hard to tell how much of the improvement comes from the 4D critic itself versus the motion regularization.
-
The primary reconstruction evaluation uses the same MoVieS model as the training reward and is therefore aligned with the optimized objective. The 4DGT cross-check provides useful additional evidence, although both pipelines share the camera estimator. These results support improved reconstructability, but do not by themselves establish independent geometric correctness.
-
Several details regarding how World-R1 and VideoGPA were adapted from bidirectional diffusion to the shared distilled autoregressive trainer need clarification. While running them in a unified autoregressive setup makes sense, the paper should explicitly list which components, conditioning signals, and reward weights were kept, modified, or dropped, and how hyperparameters were tuned (noting that the World-R1 dynamic-only phase is included). Also, on the random prompt subset, VideoGPA appears to use a different weight configuration than the best-performing variant discussed in the main motion-prominent split. The submitted supplementary ZIP contains video examples and webpage assets, but no training or adapted-baseline source code or raw judgment records. The public project page also lists code as forthcoming. Releasing the exact implementations, prompt lists, adapters, and archived judgments would enable independent verification.
Questions for authors
-
If you keep the motion prior, perceptual reward, and all training settings fixed, how much difference does switching solely between a static reconstruction critic and the dynamic MoVieS critic make? Also, how are coordinate frames, scale ambiguity, camera motion, and occlusions handled when computing the motion prior?
-
When adapting World-R1 and VideoGPA to the autoregressive trainer, which exact components were modified, how were reward weights tuned on a separate validation split, and why does the random subset use a different VideoGPA variant?
-
Table 2 compares human motion naturalness with an automated motion-preservation judgment, so the two columns need not agree. Could you discuss the observed difference in this light and provide the archived per-pair records and uncertainty for the human comparisons?
5 comments
Response to Reviewer M9TA Part II
Weakness 3 and Question 2: Baseline adaptation details, VideoGPA variant on the random subset, and code/record release Adaptation. Both baselines were designed for bidirectional T2V models. We therefore reimplemented both on our three AR bases with the same Astrolabe online trainer, optimizer settings like LoRA configs and RL configs, training steps and training prompts. Only the rewards and each method's key design components are taken from the original work.
World-R1 We replace FLOW-GRPO-Fast with Astrolabe while retaining the original reward structure. Consistent with our method, we apply GDPO to normalize each reward component before summation, whereas the original implementation does not perform this normalization. We remove the S_traj reward because we donot have the camera descirption in the prompt. To accommodate our total budget of 150 training steps, we shorten each training cycle from 100 main-phase steps and 50 dynamic-phase steps to 20 and 10 steps, respectively. Finally, instead of computing the HPSv3 reward on a single randomly selected frame in their original recipe, we score all frames using HSPv2 and average the top 30% of frame scores to improve stability. VideoGPA (Table S2, standard). We use exactly the same reward, −3DCS, as the original method and adapt its DPO training alogrithm on preference pairs to our online group-relative RL algorithm. VideoGPA (Table 1, enhanced). We observe that the standard recipe in Table S2 quickly leads to nearly static videos. To address this, we incorporate the same aesthetic anchor and dynamic-phase schedule used in our World-R1 implementation. We apologize for not clearly describing these modifications in the paper.
We report the the results of standard and enhanced recipe for Table S2 tables as follows:
Random-500 table with both VideoGPA variants
| Backbone | Method | MoVieS PSNR ↑ | SSIM ↑ | LPIPS ↓ | 4DGT PSNR ↑ | SSIM ↑ | LPIPS ↓ | LLM Motion ↑ | LLM Consist. ↑ | VR VQ ↑ | VR MQ ↑ | VR TA ↑ | VR Ovr ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SF | Base | 18.19 | 0.831 | 0.239 | 16.23 | 0.546 | 0.439 | — | — | — | — | — | — |
| World-R1 | 19.35 | 0.885 | 0.182 | 16.60 | 0.560 | 0.412 | 0.773 | 68.2 | 61.4 | 49.4 | 54.2 | 56.8 | |
| VideoGPA enhanced | 18.86 | 0.896 | 0.178 | 15.88 | 0.522 | 0.433 | 0.68 | 53.4 | 50.6 | 43.4 | 58.2 | 50.2 | |
| VideoGPA standard | 24.88 | 0.908 | 0.143 | 19.87 | 0.621 | 0.396 | 0.477 | 38.3 | 63.0 | 63.4 | 65.0 | 67.4 | |
| Ours | 21.23 | 0.895 | 0.177 | 17.70 | 0.657 | 0.411 | 0.816 | 75.1 | 59.8 | 56.6 | 54.4 | 59.8 | |
| CF | Base | 16.30 | 0.802 | 0.252 | 14.94 | 0.512 | 0.455 | — | — | — | — | — | — |
| World-R1 | 19.67 | 0.894 | 0.162 | 16.34 | 0.575 | 0.417 | 0.637 | 61.1 | 72.6 | 60.6 | 57.0 | 65.8 | |
| VideoGPA enhanced | 18.38 | 0.882 | 0.186 | 14.90 | 0.506 | 0.496 | 0.67 | 49.7 | 69.2 | 58.2 | 61.8 | 65.4 | |
| VideoGPA standard | 22.94 | 0.843 | 0.147 | 19.33 | 0.605 | 0.377 | 0.484 | 36.5 | 67.6 | 54.8 | 53.4 | 62.2 | |
| Ours | 20.99 | 0.890 | 0.156 | 17.49 | 0.650 | 0.404 | 0.716 | 64.9 | 71.4 | 65.8 | 52.8 | 68.6 | |
| LL | Base | 18.11 | 0.863 | 0.217 | 15.45 | 0.509 | 0.447 | — | — | — | — | — | — |
| World-R1 | 23.01 | 0.940 | 0.137 | 17.53 | 0.630 | 0.438 | 0.457 | 46.8 | 74.8 | 68.6 | 63.4 | 75.8 | |
| VideoGPA enhanced | 21.18 | 0.939 | 0.145 | 17.21 | 0.607 | 0.391 | 0.45 | 35.8 | 63.2 | 52.0 | 56.2 | 58.2 | |
| VideoGPA standard | 22.99 | 0.943 | 0.130 | 19.24 | 0.668 | 0.339 | 0.136 | 9.8 | 66.0 | 63.6 | 54.6 | 67.0 | |
| Ours | 24.60 | 0.910 | 0.147 | 19.59 | 0.646 | 0.417 | 0.701 | 68.5 | 74.8 | 76.6 | 65.4 | 78.2 |
- The standard recipe is the stronger reconstructor but the weaker video model. On the random subset it reaches the good reconstruction scores by freezing the scene. Stream4D achieves higher motion-preservation and joint motion–consistency scores than both VideoGPA configurations on every backbone in both sets.
Response to Reviewer M9TA Part III
Question 1: motion-prior handling of coordinate frames, scale, camera motion and occlusion
Motion prior. The motion prior is computed from MoVieS's per-pixel 3D trajectories.
- Coordinate frame and camera motion. StreamVGGT first estimates per-frame cameras, which MoVieS takes as input. MoVieS then predicts each pixel's 3D trajectory in the shared world frame, and m is the frame-to-frame displacement along these trajectories. Under a moving camera, a static background stays fixed in the world frame and contributes near-zero motion, so m measures object motion.
- Scale. StreamVGGT normalizes each video's reconstruction to a canonical scene size, so m measures displacement relative to the scene's own extent. This makes m comparable across close-ups and wide shots without metric scale.
- Occlusion and unreliable regions. The motion and smoothness terms are weighted by MoVieS's per-pixel confidence, so occluded, disoccluded and low-confidence pixels contribute little.
Question 3: Human motion naturalness vs automated motion preservation We agree that the two columns measure different things.
- The MLLM judge measures preservation. It asks whether the amount of motion is kept, reduced or lost.
- Human raters judge naturalness. They pick which clip's motion looks more natural, which also weighs smoothness, physical plausibility and visual quality.
Read this way, the two columns are consistent: Against base, the motion is preserved. Against the geometry-RL baselines, ours outperforms in both motion preservation and quality.
Question 3: per-pair records and uncertainty The 95% CIs come from a 10,000-sample bootstrap over trials. The results are as follows:
| Ours vs | Motion W / T / L | Motion win% | Overall W / T / L | Overall win% |
|---|---|---|---|---|
| base | 15 / 13 / 22 | 43.0 [31, 55] | 21 / 18 / 11 | 60.0 [49, 71] |
| World-R1 | 28 / 16 / 6 | 72.0 [62, 81] | 28 / 20 / 2 | 76.0 [68, 84] |
| VideoGPA | 39 / 9 / 2 | 87.0 [79, 94] | 33 / 14 / 3 | 80.0 [71, 88] |
As shown form the table, our method beats World-R1 and VideoGPA by a large margin.
Response to Reviewer M9TA Part I
We thank the reviewer for the detailed and constructive review. We answer your questions as follows:
Weakness 1 and Quesion 1: The 4D critic swap is not isolated from the new motion reward
Thanks for your valuable advice! It is very insightful. To isolate the effect of the motion term and validate the 4d design, we replaced the 4D MoVieS critic with the static 3D critic built by DA3-GIANT in World-R1 S_recon term while keeping all the other components identical. We evaluate on the 500 Table 1 prompts and report the LLM Judge motion score, consistenct score, overall VideoReward score and the Track failure rate described in Weakness 1 and Question 1.
As shown in the table, replacing the 4d consistency critic with a static one solely harms the motion, consistency score and general videoreward judge. This is due to the fact that the 3d static critic is trained on the static scenes, so every moving object inside would be viewed as failure and punished.
Weakness 2: Primary evaluation uses the training reward's MoVieS; 4DGT shares the camera estimator
Thanks for raising this up! We added three evaluations that do not share the reward pipeline.
Point tracking. Following the photometric-consistency metric of WorldScore [1], which measures the forward–backward round-trip error of SEA-RAFT optical flow between consecutive frames, we chain the flow into long-range tracks. We evaluate it on the 500 motion-prominent test prompts of Table 1. From the first 81 frames of each video we take every second frame, start tracks from a uniform grid in the first frame (one point every 8 px), and chain the frame-to-frame flow forward over all 40 steps and then back. A point counts as a failure if it does not return to within 3 px of where it started; points that leave the image are excluded. The lower the failure rate, the more the motion agrees with the prior that the tracking model learned from natural videos. This cycle-failure rate drops on every backbone: 0.77 → 0.67 (SF), 0.84 → 0.47 (CF) and 0.69 → 0.48 (LL), validating the motion quality of our method.
Different cameras. To decouple the camera from our reward pipeline, we re-ran the MoVieS evaluation with three camera estimators that are different of StreamVGGT (VGGT-1B, MonST3R and DA3-GIANT), and evaluated it on the same 500 prompts used in Table 1:
Across all three different cameras, ours leads base, World-R1 and VideoGPA in all 27 comparisons.
Another pipeline. We report the results based on different pair of reconstructor and caremas, 4DGT and VGGT-1B cameras. Results show that ours leads base, World-R1 and VideoGPA on every backbone.
Reward-blind MLLM judge. We use Gemini-3.5-Flash to judge whether our video is better than the base on two conditions at once: (A) it keeps roughly as much genuine object motion as the base; and (B) on that moving content, objects hold their identity, shape and structure better, with less morphing, warping, popping or flicker. Our win rate against base is 82.2% / 73.9% / 74.2% (SF / CF / LL), the highest of all methods on every backbone.
[1]Duan, Haoyi, et al. "Worldscore: A unified evaluation benchmark for world generation." Proceedings of the IEEE/CVF international conference on computer vision. 2025.