- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Envision4D is a feed-forward framework for forecasting dynamic driving scenes from unposed RGB context images. It builds on a frozen VGGT backbone, inserts trainable temporal-attention modules, and jointly predicts future camera poses and target-conditioned Gaussian motion. Context-frame Gaussians are warped to future timestamps and rendered using the predicted cameras. Training combines image reconstruction, VGGT-derived camera/depth pseudo-supervision, and a progressive training strategy.
The central contribution is the integration of future-camera prediction and time-conditioned motion into a pose-free Gaussian forecasting pipeline. Experiments on Waymo and nuScenes demonstrate improved short-horizon rendering over adapted reconstruction baselines, principally STORM and DGGT.
Critical strengths
- Jointly predicting scene evolution and future cameras removes the requirement for supplied future ego poses at inference. Producing an explicit, renderable scene representation is also valuable for inspecting geometry and motion.
- I can see its significant improvement compared with the STORM.
Critical weaknesses
- Most results cover only 0.2 seconds, with a maximum horizon of 0.6 seconds. Allowing a separate displacement-normalized velocity for each target timestamp increases expressiveness, but does not itself establish coherent trajectories across time. Per-timestamp errors, turning/braking subsets, and comparisons against simple acceleration or polynomial-motion models would better support this claim. I would also like to know how large the error is when reconstructing frames based on different time intervals.
- The comparison selection needs stronger justification. Flux4D is explicitly excluded because it requires LiDAR, although its original paper reports a LiDAR-free inference variant using monocular depth. Differences in training supervision and camera requirements remain relevant, but the stated exclusion is incomplete. VGGT-World is compared only on camera accuracy, despite being particularly relevant to future-depth and geometry prediction. The paper discusses DynamicVGGT and NeoVerse as architectural alternatives but does not compare against them experimentally. Both extend VGGT to dynamic reconstruction and are relevant to the claimed advantage of in-layer temporal attention. Although Table 5 provides a useful comparison against a post-layer attention variant, this does not replace evaluation against the complete prior methods. Their different supervision and reconstruction settings require careful adaptation, but a matched comparison—or a concrete justification for exclusion—would strengthen the evidence for the proposed architecture.
Questions for authors
- How much do the VGGT camera and depth pseudo-labels contribute? What if some ground-truth values are used here?
- Can you compare with DynamicVGGT or NeoVerse’s reconstruction module under the same context-frame and future-prediction protocol, explicitly accounting for differences in supervision and camera inputs? If such adaptation is infeasible, please explain the specific obstacles and clarify how closely the post-layer attention baseline in Table 5 reproduces their motion-modeling designs.
- I hope the authors can try using synthetic data generated by video generation models to investigate whether the method can be extended to a wider range of application scenarios.
3 comments
Thank you for the detailed rebuttal and additional experiments. The authors have addressed most of my concerns.
The new ablation also clarifies the contribution of VGGT-derived pseudo-supervision. Removing this supervision causes only a relatively modest performance decrease, suggesting that these pseudo-label losses are not a major source of the method’s improvement. This limits my confidence in the claimed contribution of this component and prevents me from increasing my score further.
Nevertheless, the revision substantially strengthens the empirical evidence, and the proposed framework remains technically interesting and practically valuable. I therefore continue to recommend acceptance.
Thank you for your constructive comments and the time you have dedicated to our paper! We are glad that our response has addressed most of your concerns, and we sincerely appreciate your recognition of our work.
More evaluations on future modeling
We thank the reviewer for the constructive suggestions. Using our (T_c=2, T_f=6) model, we provide per-timestamp results, turning/braking subset evaluations, and comparisons with a learned acceleration baseline under matched architecture and training settings. Further details are provided in Appendix A.2 (lines 779–781 and 850–855, Table A.4).
The acceleration baseline performs comparably at short horizons but degrades more rapidly over time. Our consistent gains on turning and braking subsets highlight the benefits of target-dependent motion modeling under non-constant motion. More qualitative visualizations are provided in Appendix A.2 (Fig. A.1).
About pseudo-labels and GT values
We ablate the contribution of VGGT pseudo-supervision:
Our model maintains strong performance even without VGGT pseudo-supervision.
We further evaluate the performance if we use GT camera poses like STORM:
Using GT camera poses substantially improves performance by reducing prediction uncertainty and the burden of camera estimation, highlighting the challenge of jointly predicting future camera poses and 4D Gaussians.
Comparison selection
We thank the reviewer for pointing out these omissions. We will clarify that Flux4D relies on LiDAR-based geometry initialization during training but supports LiDAR-free inference. Since its official code is unavailable, we cannot reproduce it under our future extrapolation setting and instead include its reported results in the upper part of Table 1 for reference. For VGGT-World, we obtain a depth estimation error of 7.26, compared with 3.98 for our method. The performance gap may stem from VGGT-World relying solely on VGGT pseudo-supervision, whereas our method additionally incorporates 4DGS rendering supervision for joint geometric optimization. We have added these comparisons in the revised paper.
Comparison with NeoVerse
The post-layer attention variant in Table 5 was designed to isolate the effect of attention placement while keeping our nonlinear velocity modeling (CML) unchanged. To further compare with NeoVerse, we replace this post-layer temporal attention with its complete Bidirectional Motion-Encoding module. Further implementation details are provided in Appendix A.2 (lines 860–863 and 886–887).
The adapted NeoVerse baseline achieves 26.43/0.798/0.172 (PSNR/SSIM/LPIPS), showing no consistent improvement over our original post-layer variant (26.98/0.788/0.170) in Table 5 despite its more sophisticated architecture. In contrast, our in-layer design achieves substantially better performance with simple temporal attention, suggesting that integrating attention within the backbone can be more effective.
Using synthetic data
We thank the reviewer for this suggestion. To explore broader applicability, we evaluate Envision4D on unseen OmniDreams-generated driving videos under rainy and snowy conditions. Our method achieves 25.41/0.7826/0.2369 (PSNR/SSIM/LPIPS) for 2-frame prediction and 22.86/0.7304/0.2925 for 6-frame prediction. These results, together with the visualizations in Appendix A.2 (Fig. A.2), suggest robustness to unseen synthetic scenarios and motivate further exploration of video-generated data.