acceptodds
Under review as a conference paper at ICLR 2027

Envision4D: Envisioning Visual Futures via Feed-forward 4D Gaussian Splatting for Autonomous Driving

Abstract

Forecasting the future evolution of dynamic scenes is crucial in autonomous driving. However, existing feed-forward Gaussian Splatting methods are primarily designed for scene interpolation. When extended to future extrapolation, they suffer from ghosting artifacts under large displacements and are constrained by simplified motion assumptions or strict future priors. To overcome these challenges, we propose Envision4D, a feed-forward framework for pose-free future extrapolation with self-supervised motion learning. Specifically, we introduce a Future Pose Prediction module that infers future camera parameters via an iterative refinement process. Furthermore, to capture non-linear dynamics, we propose In-layer Temporal Attention and employ Conditioned Motion Lifting, which transforms the highly uncertain extrapolation process into robust relational mappings. Finally, a Progressive Training Strategy is utilized to stabilize unsupervised motion learning against error accumulation. Extensive experiments demonstrate that Envision4D achieves state-of-the-art performance, significantly outperforming existing methods in future view synthesis.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Autonomous Driving, 4DGS

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SitMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Envision4D is a feed-forward framework for forecasting dynamic driving scenes from unposed RGB context images. It builds on a frozen VGGT backbone, inserts trainable temporal-attention modules, and jointly predicts future camera poses and target-conditioned Gaussian motion. Context-frame Gaussians are warped to future timestamps and rendered using the predicted cameras. Training combines image reconstruction, VGGT-derived camera/depth pseudo-supervision, and a progressive training strategy.

The central contribution is the integration of future-camera prediction and time-conditioned motion into a pose-free Gaussian forecasting pipeline. Experiments on Waymo and nuScenes demonstrate improved short-horizon rendering over adapted reconstruction baselines, principally STORM and DGGT.

Critical strengths
  1. Jointly predicting scene evolution and future cameras removes the requirement for supplied future ego poses at inference. Producing an explicit, renderable scene representation is also valuable for inspecting geometry and motion.
  2. I can see its significant improvement compared with the STORM.
Critical weaknesses
  1. Most results cover only 0.2 seconds, with a maximum horizon of 0.6 seconds. Allowing a separate displacement-normalized velocity for each target timestamp increases expressiveness, but does not itself establish coherent trajectories across time. Per-timestamp errors, turning/braking subsets, and comparisons against simple acceleration or polynomial-motion models would better support this claim. I would also like to know how large the error is when reconstructing frames based on different time intervals.
  2. The comparison selection needs stronger justification. Flux4D is explicitly excluded because it requires LiDAR, although its original paper reports a LiDAR-free inference variant using monocular depth. Differences in training supervision and camera requirements remain relevant, but the stated exclusion is incomplete. VGGT-World is compared only on camera accuracy, despite being particularly relevant to future-depth and geometry prediction. The paper discusses DynamicVGGT and NeoVerse as architectural alternatives but does not compare against them experimentally. Both extend VGGT to dynamic reconstruction and are relevant to the claimed advantage of in-layer temporal attention. Although Table 5 provides a useful comparison against a post-layer attention variant, this does not replace evaluation against the complete prior methods. Their different supervision and reconstruction settings require careful adaptation, but a matched comparison—or a concrete justification for exclusion—would strengthen the evidence for the proposed architecture.
Questions for authors
  1. How much do the VGGT camera and depth pseudo-labels contribute? What if some ground-truth values are used here?
  2. Can you compare with DynamicVGGT or NeoVerse’s reconstruction module under the same context-frame and future-prediction protocol, explicitly accounting for differences in supervision and camera inputs? If such adaptation is infeasible, please explain the specific obstacles and clarify how closely the post-layer attention baseline in Table 5 reproduces their motion-modeling designs.
  3. I hope the authors can try using synthetic data generated by video generation models to investigate whether the method can be extended to a wider range of application scenarios.
3 comments

More evaluations on future modeling

We thank the reviewer for the constructive suggestions. Using our (T_c=2, T_f=6) model, we provide per-timestamp results, turning/braking subset evaluations, and comparisons with a learned acceleration baseline under matched architecture and training settings. Further details are provided in Appendix A.2 (lines 779–781 and 850–855, Table A.4).

HorizonMethodPSNR ↑SSIM ↑LPIPS ↓
+0.1sAcceleration27.830.8290.143
Ours28.220.8200.131
+0.2sAcceleration27.410.8040.162
Ours27.560.8040.166
+0.3sAcceleration25.480.7580.208
Ours26.300.7740.184
+0.4sAcceleration23.320.7060.279
Ours25.340.7480.208
+0.5sAcceleration21.270.6500.362
Ours25.020.7420.217
+0.6sAcceleration19.230.5820.449
Ours24.790.7350.245
OverallAcceleration24.090.7220.267
Ours26.210.7710.192
SubsetFuture FramesMethodPSNR ↑SSIM ↑LPIPS ↓
Turning2Acceleration25.280.7500.206
Ours25.850.7610.188
Turning6Acceleration21.570.6580.343
Ours22.970.6890.274
Braking2Acceleration27.430.8200.154
Ours27.550.8150.144
Braking6Acceleration24.200.7210.260
Ours25.740.7610.187

The acceleration baseline performs comparably at short horizons but degrades more rapidly over time. Our consistent gains on turning and braking subsets highlight the benefits of target-dependent motion modeling under non-constant motion. More qualitative visualizations are provided in Appendix A.2 (Fig. A.1).

About pseudo-labels and GT values

We ablate the contribution of VGGT pseudo-supervision:

Pseudo supervisionPSNR ↑SSIM ↑LPIPS ↓
Without27.470.8000.165
With (Full Model)27.810.8160.151

Our model maintains strong performance even without VGGT pseudo-supervision.

We further evaluate the performance if we use GT camera poses like STORM:

Camera ConfigurationPSNR ↑SSIM ↑LPIPS ↓
Predicted Camera27.810.8160.151
GT Camera28.740.8440.130

Using GT camera poses substantially improves performance by reducing prediction uncertainty and the burden of camera estimation, highlighting the challenge of jointly predicting future camera poses and 4D Gaussians.

Comparison selection

We thank the reviewer for pointing out these omissions. We will clarify that Flux4D relies on LiDAR-based geometry initialization during training but supports LiDAR-free inference. Since its official code is unavailable, we cannot reproduce it under our future extrapolation setting and instead include its reported results in the upper part of Table 1 for reference. For VGGT-World, we obtain a depth estimation error of 7.26, compared with 3.98 for our method. The performance gap may stem from VGGT-World relying solely on VGGT pseudo-supervision, whereas our method additionally incorporates 4DGS rendering supervision for joint geometric optimization. We have added these comparisons in the revised paper.

Comparison with NeoVerse

The post-layer attention variant in Table 5 was designed to isolate the effect of attention placement while keeping our nonlinear velocity modeling (CML) unchanged. To further compare with NeoVerse, we replace this post-layer temporal attention with its complete Bidirectional Motion-Encoding module. Further implementation details are provided in Appendix A.2 (lines 860–863 and 886–887).

The adapted NeoVerse baseline achieves 26.43/0.798/0.172 (PSNR/SSIM/LPIPS), showing no consistent improvement over our original post-layer variant (26.98/0.788/0.170) in Table 5 despite its more sophisticated architecture. In contrast, our in-layer design achieves substantially better performance with simple temporal attention, suggesting that integrating attention within the backbone can be more effective.

Using synthetic data

We thank the reviewer for this suggestion. To explore broader applicability, we evaluate Envision4D on unseen OmniDreams-generated driving videos under rainy and snowy conditions. Our method achieves 25.41/0.7826/0.2369 (PSNR/SSIM/LPIPS) for 2-frame prediction and 22.86/0.7304/0.2925 for 6-frame prediction. These results, together with the visualizations in Appendix A.2 (Fig. A.2), suggest robustness to unseen synthetic scenarios and motivate further exploration of video-generated data.

Thank you for the detailed rebuttal and additional experiments. The authors have addressed most of my concerns.

The new ablation also clarifies the contribution of VGGT-derived pseudo-supervision. Removing this supervision causes only a relatively modest performance decrease, suggesting that these pseudo-label losses are not a major source of the method’s improvement. This limits my confidence in the claimed contribution of this component and prevents me from increasing my score further.

Nevertheless, the revision substantially strengthens the empirical evidence, and the proposed framework remains technically interesting and practically valuable. I therefore continue to recommend acceptance.

Thank you for your constructive comments and the time you have dedicated to our paper! We are glad that our response has addressed most of your concerns, and we sincerely appreciate your recognition of our work.

Reviewer_dqmQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Envision4D, a feed-forward framework for future-view extrapolation from unposed context images. It jointly predicts future camera parameters and time-conditioned Gaussian motion, without requiring explicit optical-flow, tracking, or dynamic-mask supervision. The approach augments a frozen VGGT backbone with in-layer temporal attention, refines initialized future camera tokens, and conditions source-image features on target-specific temporal and ego-motion features to predict Gaussian displacements. Training combines geometric warm-up, self-exclusive rendering, progressive weighting of future-frame losses, and camera/depth pseudo-supervision from a frozen VGGT teacher that observes the complete training sequence. Experiments on Waymo and nuScenes report improvements over the evaluated reconstruction baselines in future-image quality, together with improved dynamic-region rendering and scene-flow estimation. The default forecasting horizon is 0.2 seconds, with additional image-extrapolation experiments up to 0.6 seconds.

Critical strengths
  1. The paper addresses a meaningful combination of reconstruction and forecasting requirements. Jointly predicting future camera motion and scene motion removes the need to provide target camera poses at inference time, while retaining an explicit, renderable scene representation. Avoiding explicit motion annotations is also a useful property. These features make the proposed setting more flexible than extrapolation that assumes known future camera trajectories, and constitute a substantive practical strength of the work.

  2. The reported improvements are meaningful and extend beyond whole-image reconstruction metrics. Relative to STORM, Envision4D improves future-image PSNR from 26.19 to 27.81 on Waymo and from 24.69 to 26.86 on nuScenes. It also improves dynamic-region PSNR from 21.23 to 24.46 and reduces scene-flow EPE3D from 0.206 m to 0.161 m on Waymo. These results provide evidence that the system improves motion-sensitive reconstruction rather than only static-background appearance. The matched comparison between in-layer and post-layer temporal attention is also useful evidence for the proposed backbone adaptation.

Critical weaknesses
  1. The distinctive contribution of the proposed motion-conditioning mechanism is not sufficiently isolated. Conditioned Motion Lifting is implemented through element-wise modulation of source features by target-specific features, followed by a DPT motion decoder. This is a reasonable design, but the experiments do not yet establish what is gained specifically from the proposed temporal/ego-motion conditioning beyond a simpler time-conditioned predictor. In Table 5, the constant-velocity versus CML comparison is conducted without temporal attention. The table does not include the full in-layer-attention model with only CML replaced by constant velocity, nor a control that retains target-time conditioning but removes the learned ego-motion conditioning. Consequently, although the ablations support the usefulness of several design choices, they do not sufficiently establish the independent contribution of the central motion-modeling component within the complete system. This matters because the methodological advance otherwise appears relatively incremental, consisting primarily of backbone adaptation, conditional feature modulation, and training strategies.

  2. Important supervision and geometric-evaluation details remain insufficiently specified. Equation 9 uses camera and depth pseudo-targets produced by VGGT from the complete context-and-future training sequence. This is a legitimate training setup, not evidence of test-time leakage. However, the contribution of this supervision is not independently ablated, and the statement that baselines are retrained under the same experimental setup does not clarify whether they receive comparable geometric pseudo-supervision. This leaves uncertainty about the attribution of the end-to-end improvements.

Questions for authors

Q1. Motion-conditioning controls. With the full in-layer temporal-attention backbone, future-pose module, supervision, and training schedule held fixed, how does CML compare with a constant-velocity predictor and with a target-time-conditioned predictor that does not use the learned ego-motion conditioning?

Q2. Future-motion validation. Can the scene-flow results for 1->2 and 2->3 be reported separately? Within the existing forecasting range, can the authors provide per-target-time dynamic-region results and multi-step motion errors, particularly for cases with non-constant motion?

Q3. Supervision and evaluation protocol. Which geometric pseudo-targets are used for each baseline? Please also specify the D-RMSE reference, prediction-to-ground-truth correspondence for scene flow, coordinate/scale alignment, and ego-motion handling. A controlled ablation of the geometric pseudo-supervision would help distinguish its contribution from that of the proposed architecture.

3 comments

Motion-conditioning controls

Thanks for your advice. As suggested, we conduct motion-related ablations with all other modules and training schedule fixed:

Motion PredictorPSNR ↑SSIM ↑LPIPS ↓
Constant-Velocity Predictor27.190.8030.165
Target-Time-Conditioned Predictor27.350.8100.162
Full Model (CML)27.810.8160.151

These results suggest that effective future-motion prediction requires more than constant-velocity assumptions or simple target-time conditioning. By jointly incorporating temporal and ego-motion cues, CML learns more accurate target-dependent motion dynamics, consistently outperforming both simplified alternatives within the complete architecture.

About pseudo supervision

Importantly, STORM and DGGT already benefit from more direct geometric supervision or external motion priors. STORM uses GT camera poses even during evaluation and GT depth supervision. DGGT relies on frozen VGGT geometry estimation, and pretrained 3D tracking with dynamic-mask supervision.

To isolate the contribution of VGGT pseudo-supervision, we further ablate it in our model while keeping other settings unchanged:

Pseudo supervisionPSNR ↑SSIM ↑LPIPS ↓
Without27.470.8000.165
With (Full Model)27.810.8160.151

Even without geometric pseudo-supervision, our method still outperforms both baselines without relying on GT camera poses or external dynamic-mask supervision, indicating that the performance gains cannot be solely attributed to geometric pseudo-supervision.

Future-motion validation

We report scene-flow errors separately for the 1→2 and 2→3 intervals, together with per-target-time dynamic-region reconstruction results, on both the full test set and the non-constant-motion subset (turning and braking). As Waymo provides scene-flow annotations only for consecutive frames, direct evaluation of multi-step motion errors is limited.

IntervalSubsetEPE3D ↓ACC5 ↑ACC10 ↑Dyn. PSNR ↑
1→2All0.157185.3791.5325.01
Non-constant Motion0.182584.4690.6324.46
2→3All0.164984.8991.3723.92
Non-constant Motion0.186484.3890.4323.52

The results show comparable motion accuracy across consecutive intervals, with only moderate performance degradation in non-constant-motion scenarios. We also add qualitative results in Appendix A.2 (Fig. A.1), illustrating our model's ability to handle nonlinear motion in turning and braking scenarios.

Evaluation protocol

We use Waymo's LiDAR-derived GT depth and scene-flow annotations, computing D-RMSE and scene-flow metrics at corresponding pixels with valid GT depth. To resolve scale ambiguity, we estimate the alignment using only the context frames and apply it consistently to all future predictions. After scale alignment, predicted and GT scene flow are compared in the same coordinate frame, without additional ego-motion compensation. We have clarified these details in the revised manuscript.

Reviewer_dqmQOpenReview ↗

Reply to the Author

Thank you to the author for the response. After the additional experiments and explanations addressed my concerns, I have decided to raise my rating to “weak acceptance.”

AuthorsOpenReview ↗

Thanks for Your Review

Thank you for your constructive comments on our paper. We also sincerely appreciate your recognition of our revisions.

Reviewer_Ecu9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Envision4D predicts the future pose of the camera and motion of the reconstruction of the scene. Unlike previous feedforward 3dgs methods, the goal here is not interpolation but extrapolation. It also does not rely on constant velocity to predict the future but uses a trained model, allowing more accurate camera pose prediction and object motion. The training does not rely on any motion label. The model takes 2 frames and uses a temporal attention to predict motion tokens that decode into velocity per pixel to apply to the 3D Gaussians for each target time. Another module predicts the future camera pose by adding an offset to the last context camera token, and self attention blocks refine them. Supervision is done on the rendering of the Gaussians and predicted poses and depth from a frozen vggt which sees context and future frames.

Critical strengths
  1. Novel feedforward 3dgs method that predicts future frames from unposed images and without motion labels. It beats retrained previous baselines (STORM, DGGT) on Waymo and nuScenes.
  2. Ablations that show the improvement in performance brought by each module (temporal attention inside VGGT vs after, velocity per target time vs constant velocity, learned vs linear future pose).
Critical weaknesses
  1. Baseline retraining: the retraining of STORM and DGGT is different from their original setup (ex: STORM used 4 posed frames over 1.5s vs 2 frames over 0.1s) and not detailed. This could impact actual performance of these models, making the comparison unreliable. Also missing simple baselines like copy last frame and zero motion for comparison.
  2. Horizon is very short (0.2-0.6s). For most traffic on such a short horizon the velocity might be almost constant. It would be interesting to observe longer horizon which would have more obvious non-linear trajectories.
Questions for authors
  1. Could you provide more details about the retraining of STORM and DGGT?
  2. STORM is noted as not pose-free, are the GT poses given to STORM during evaluation?
  3. Could baselines like copy last frame, linear motion and no motion be added?
  4. Some results on longer horizon could be interesting.
1 comment

Details about baseline retraining

We retrained STORM and DGGT under a unified extrapolation protocol with matched training iterations, while preserving their original architectures and supervision. Specifically, STORM uses GT camera poses during both training and evaluation, together with its original GT-depth and sky supervision. DGGT also retains its sky and dynamic mask losses.

To further address concerns about the modified input setting, we evaluate STORM under its original 4-frame, 1.5s context setting, using the same two future target frames for evaluation.

MethodInputResults
STORM (Retrained)2 frames / 0.2s26.19 / 0.798
STORM (Original Context)4 frames / 1.5s25.03 / 0.743

More baseline comparison

We further compare against three simple baselines built upon our full model:

MethodPSNR ↑SSIM ↑LPIPS ↓
Copy Last Frame24.000.7290.243
Zero Motion25.490.7730.189
Linear Motion27.190.8030.165
Our Full Model27.810.8160.151

These results demonstrate that our non-linear motion modeling provides consistent improvements over simpler motion assumptions.

Results on longer horizon

We extend the prediction horizon to 1.2s and 1.8s to evaluate longer-term forecasting performance:

Prediction HorizonMethodPSNR ↑SSIM ↑LPIPS ↓
0.6 sSTORM24.410.7530.346
0.6 sOurs26.210.7710.192
1.2 sSTORM20.980.6550.496
1.2 sOurs24.120.7220.259
1.8 sSTORM19.070.6180.526
1.8 sOurs22.740.6890.308

Our method, with nonlinear velocity modeling, demonstrates better long-horizon stability than STORM, with a smaller PSNR drop from 0.6s to 1.8s (3.47 vs. 5.34 dB) and a 3.67 dB improvement over STORM at 1.8s.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers for their detailed reviews and constructive suggestions!

We have updated the manuscript based on the reviewers' suggestions. All revisions in the updated manuscript are highlighted in red:

Add more experiments

  • Add simple baseline comparisons and motion-conditioning ablations (Reviewers Ecu9, dqmQ, Appendix A.2, lines 755–777, Table A.3).
  • Add longer-horizon forecasting results up to 1.8s (Reviewer Ecu9, Appendix A.2, lines 748–753, Table A.2).
  • Add per-interval future-motion validation (Reviewer dqmQ, Appendix A.2, lines 856–859, Table A.5).
  • Add ablations on VGGT pseudo-supervision and GT camera poses (Reviewers dqmQ, SitM, Appendix A.2, lines 888–893, Table A.7).
  • Add acceleration-based motion comparisons, per-timestamp evaluations, and turning/braking subset results (Reviewer SitM, Appendix A.2, lines 779–781 and 850–855, Table A.4 and Figure A.1).
  • Add comparisons with Flux4D and VGGT-World (Reviewer SitM, Table 1).
  • Add comparisons with NeoVerse (Reviewer SitM, Appendix A.2, lines 860–863 and 886–887, Table A.6).
  • Add evaluations on synthetic driving videos generated by OmniDreams (Reviewer SitM, Appendix A.2, lines 894–897, Figure A.2).

Add more discussion and details

  • Clarify the retraining settings and supervision of STORM and DGGT (Reviewers Ecu9, dqmQ, Appendix A.1, lines 724–727).
  • Clarify the geometric evaluation protocol, including GT references, scale alignment, and ego-motion handling (Reviewer dqmQ, Appendix A.1, lines 710–713).

We sincerely thank all reviewers for their time and valuable feedback, and look forward to further discussions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.