acceptodds
Under review as a conference paper at ICLR 2027

X-Splatter: Feed-Forward Metric 4D Gaussian Splatting for Surround-View Driving Scenes

Abstract

Feed-forward scene reconstruction supports scalable simulation and world modeling for autonomous driving. We present X-Splatter, a surround-view, feed-forward 4D Gaussian Splatting (4DGS) framework that processes all surround cameras jointly on a VGGT-Ω backbone, with cross-camera attention throughout the network. It introduces two key components: Depth Anchoring Fine-tuning (DAF), which shifts geometry from relative to metric space, and Static–Dynamic Decomposition Learning (SDDL), which combines dynamic gating, static-promoting regularization, and intermediate-frame supervision to alleviate static–dynamic conflation and enhance motion learning. Following LiDAR-supervised depth adaptation, X-Splatter uses self-supervised motion learning, without bounding boxes, tracklets, or optical-flow labels. Trained and evaluated on all surrounding views, including the side views that share the least content with the front camera, X-Splatter achieves the best full-image and dynamic-region novel-view synthesis and the lowest metric-depth RMSE among the evaluated baselines on the Waymo Open Dataset. It also achieves the lowest cross-view 3D error, indicating better geometric agreement across adjacent cameras. The same model generalizes zero-shot to nuScenes and Argoverse 2, outperforming the evaluated baselines in rendering and metric-depth accuracy on both.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
feed-forward reconstruction, 4D Gaussian splatting, dynamic scene reconstruction, surround-view driving scenes, metric depth, novel view synthesis, autonomous driving simulation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_2KoHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose X-Splatter, a new feed-forward 4D gaussian prediction framework that reconstructs both static scenes and moving objects. And then X-Splatter can replay the scene along new ego trajectories. To achieve this, the authors introduce Depth Anchoring Fine-tuning for metric depth estimation and Static–Dynamic Decomposition Learning to separate static and dynamic components and learn object motion. And the results show that X-Splatter achieves SOTA performance on the Waymo dataset.

Critical strengths

I think image-based dynamic scene reconstruction is an important research topic that supports many downstream tasks. Using mixing weights to separate static and dynamic contributions and predicting the motion dynamic gaussians is a reasonable design for static-dynamic decomposition. The experimental results in Table 1 and Table 2 show the effectiveness of X-Splatter.

Critical weaknesses

I think the novelty of Depth Anchoring Fine-tuning is limited. It mainly uses LiDAR info to align the teacher model’s relative depth and construct supervision targets. In addition, does X-Splatter in Table 1 use all five cameras as input while being evaluated only on the front camera? Could the authors provide results using only the three front-facing cameras as input, allowing comparison under the same input settings as STORM and DGGT?

I also suggest reporting per-camera metrics for each method across all five cameras to provide a clearer comparison of performance across viewpoints and show more visualization results in five cameras.

Questions for authors

Please refer to the Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 2KoH

Thank you for the positive review.

DAF novelty. DAF's contribution is not LiDAR alignment by itself but a recipe that makes a pretrained multi-view backbone metric at one fixed scale, with no per-scene alignment at inference. Table 8 measures its steps, from 5.05 m for the released backbone (even with a global scale fit) to 3.55 m, and Table 7 shows the fixed scale is not a tuned constant.

Inputs in Table 1. Yes: the X-Splatter rows of Table 1 take all five cameras as input and score the front camera, the same as the "5 / 1" rows of Table 2(b). With only the three front cameras, its front-camera PSNR is 29.32 dB instead of 29.66 dB, so the side cameras improve even the front view. Results with only the three front cameras as input are in the general response.

Per-camera metrics and visualizations. Per-camera results for all methods are in the general response: X-Splatter leads on every camera. We add a five-camera qualitative comparison to the appendix of the revised PDF.

Reviewer_twSAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

X-Splatter is a feed-forward framework for metric 4D GS reconstruction from calibrated surround-view driving videos. The method jointly processes cameras and source timestamps using the recent VGGT-Ω backbone. DAF first adapts the geometry model to metric depth using sparse LiDAR measurements and down-weighted, scale-aligned teacher targets. The adapted geometry is then frozen during appearance and motion learning.SDDL splits each GS into static and dynamic opacity contributions, pools static contributions across cameras and timestamps, and warps dynamic contributions using predicted forward and backward displacements. Training combines source-frame reconstruction, static-promoting regularization, and supervision at held-out timestamps. The default motion-learning stage does not require object boxes, tracklets, or optical-flow labels, although DAF uses LiDAR supervision.

Critical strengths
  • Metric geometry, surround-camera coverage, and dynamic reconstruction are jointly important for rendering driving scenes under calibrated ego motion. It is useful to adress a useful combination of requirements for driving reconstruction.

  • On the five-camera Waymo protocol, the default model achieves 27.45 dB full-image PSNR, 23.44 dB dynamic-region PSNR, and 3.55 m metric-depth RMSE, compared with STORM’s 22.66 dB, 20.82 dB, and 6.67 m, respectively.

  • The paper reports substantial improvements over the evaluated STORM and DGGT checkpoints on Waymo, including rendering quality, metric depth, and cross-camera geometry.

Critical weaknesses
  • Sec. 4.1 states that the evaluated STORM and DGGT checkpoints were trained on three front-facing Waymo cameras and were not retrained for the five-camera setting. X-Splatter, in contrast, is trained for the full surround-camera configuration. As a result, the all-camera comparison combines method differences with a substantial change in the baselines’ input distribution. Do the proposed metric adaptation, frozen geometry, soft opacity decomposition, and surround-camera processing improve on a strong reconstruction pipeline under comparable conditions?

  • Table 5 shows a useful improvement from held-out-time supervision: dynamic PSNR increases from 22.00 to 22.61 dB and full-image PSNR from 26.18 to 26.97 dB, which establishs the usefulness of held-out-time supervision more clearly than the claimed SDDL mechanism.

  • The teacher-removal experiment has an normalization issue that needs clarified. Equation (10) defines the supervised set as (S={p>0}) and normalizes the geometry loss by (|S|). Under this definition, setting the teacher weight to zero removes teacher-filled pixels from (S), increasing the coefficient on the remaining LiDAR residuals and potentially changing the inverse-depth weighting floor. Please clarify the implementation and compare teacher removal with matched LiDAR coefficients and normalization before attributing the transfer difference specifically to the teacher’s dense supervision.

  • The introduction motivates replay under new ego trajectories and scalable reconstruction of driving archives. However, the quantitative novel-view evaluation uses held-out timestamps along recorded trajectories, as the limitations sec. says.

  • Please identify the exact backbone checkpoint, encoder, and sky-mask implementation. which is particularly relevant because sky masks affect both training and evaluation.

  • Equation (4) describes interpolation between source timestamps, while the 20-frame evaluation window also includes targets beyond the final source frame at index 15. Please specify which dynamic contributions are used and how they are warped for these targets, and report interpolation and extrapolation results separately.

Questions for authors

See above

1 comment
AuthorsOpenReview ↗

Response to Reviewer twSA

Thank you for the detailed review.

1. Comparable conditions. See the general response: with only the three front cameras, X-Splatter is above STORM's own published result for this setting (27.39 vs 26.38 dB full, 23.33 vs 22.10 dB dynamic).

2. SDDL. Agreed: Table 5 identifies held-out-time supervision as SDDL's largest measured contributor, as Sec. 4.5 states ("this objective accounts for the gain"). It acts on the dynamic copies that gating creates, so the two are used together. Splitting the validation scenes into thirds by their share of dynamic pixels, the static-promoting term's full-image cost shrinks from 0.36 to 0.09 dB and its dynamic-region gain grows from +0.03 to +0.14 dB from the least to the most dynamic third.

3. Teacher-removal normalization. Good catch: Eq. 10 normalizes by ∣S∣|S|, so removing teacher pixels also rescales the LiDAR term and the inverse-depth floor. We will make the normalization explicit in Appendix B and present Table 9 as a comparison of the two recipes as trained, without attributing the difference to the teacher alone.

4. Novel trajectories. Held-out timestamps on the recorded trajectory are the standard quantitative protocol (as in STORM), because ground truth exists only there. Figure 1 shows a rendering from a viewpoint off the recorded trajectory.

5. Backbone, encoder and sky mask. The backbone is the released VGGT-Ω 1B checkpoint (Appendix B); its patch encoder is a DINOv3-architecture ViT (patch 16, Appendix C; Sec. 2 should read DINOv3, which we will correct). Sky masks come from the U²-Net sky-segmentation model distributed with VGGT-Ω (skyseg.onnx), thresholded at 32/255 on the per-image normalized response. With the same mask for every method, sky is excluded from all full-image metrics; it covers 25% / 20% / 15% / 6% / 5% of pixels on the front / front-left / front-right / side-left / side-right cameras.

6. Targets after the last source frame. Frames 16–19 use only frame 15's dynamic Gaussians, moved by (t−15)/5(t-15)/5 of the predicted forward displacement. Results separately (three front cameras; input cameras as listed):

MethodFull PSNR↑ interp / extrapDyn. PSNR↑ interp / extrapDepth RMSE (m)↓ interp / extrap
STORM (5 cameras in)22.25 / 22.3021.21 / 20.756.83 / 6.99
DGGT (1 camera at a time)22.70 / 19.9020.24 / 18.2211.74 / 13.43‡
X-Splatter (5 cameras in)28.06 / 26.7324.03 / 22.543.47 / 3.88

‡After DGGT's pose alignment. X-Splatter leads on both interpolated and extrapolated frames.

Reviewer_wwwkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the task of feed-forward 4D novel-view reconstruction through feed-forward Gaussian Splatting in a surround-view setting. The authors propose Depth Anchoring Fine-tuning (DAF), which aligns the teacher's relative depth to the depth scale of LiDAR-observed pixels to generate pixel-wise depth information at metric scale and uses it as supervision. Subsequently, the geometry learned through Depth Anchoring Fine-tuning is frozen to prevent incorrect corrections that may arise from confusion between motion and geometry. The authors also propose Static–Dynamic Decomposition Learning (SDDL), which assigns a gate to each Gaussian to control the proportions of static and dynamic components, and employs Static-promoting Regularization to prevent static objects from being represented as dynamic representations. Finally, through Intermediate-frame Supervision, the authors introduce additional reconstruction supervision at intermediate time steps to improve motion learning regarding "which direction to move.”

Critical strengths

The paper is easy to read, and the methods are clearly connected to the four problems presented in the Introduction.

Critical weaknesses

[Major 1] Static-promoting Regularization appears to have different effects depending on the degree of dynamicity of a scene. In particular, scenes with many moving objects require greater use of Dynamic Gaussians, so the regularization may negatively affect performance. The current loss formulation uses a fixed-scale lambda value, which encourages the model to suppress the use of Dynamic Gaussians regardless of the characteristics of the scene. Therefore, two questions arise. (1) An analysis is needed to determine whether regularization with a fixed weight is consistently effective across scenes with different degrees of dynamicity. As an example experiment, I recommend categorizing scenes according to their degree of dynamicity and comparing performance as the lambda value varies. (2) It would be worth considering the possibility of Adaptive Regularization, which reduces the regularization strength in scenes with substantial dynamic content. Comparing this approach with the proposed fixed regularization method could help examine the scalability of the methodology.

[Major 2] X-Splatter's DAF converts relative teacher depth into metric depth based on LiDAR information and uses it as supervision. However, its performance may vary depending on the quality of LiDAR data in the training dataset. As LiDAR points become sparser in a scene, Metric Scale Estimation may become unstable, potentially negatively affecting training performance. Therefore, I recommend conducting an ablation experiment to demonstrate the robustness of the proposed method to LiDAR sparsity.

[Major 3] According to the experimental setup in Section 4.1, the authors state that “Both checkpoints were trained on the three front-facing Waymo cameras and not retrained for five-camera input.” Therefore, it appears that STORM and DGGT, the key comparison baselines, were evaluated using checkpoints trained with three front-facing cameras. In contrast, X-Splatter was trained from the beginning using five surround-view cameras. Therefore, for a fair performance comparison of the methods, I suggest training X-Splatter using the same three front-facing cameras as the baselines and conducting a comparative evaluation under the same camera configuration.

Questions for authors

The questions are summarized below. Please refer to the Critical Weaknesses section for detailed explanations.

Major 1: Is Static-promoting Regularization consistently effective across scenes with different degrees of dynamicity, and could Adaptive Regularization further improve performance?

Major 2: How robust is DAF to varying levels of LiDAR sparsity?

Major 3: Could the authors provide a fair comparison by training and evaluating X-Splatter and the baselines under the same three-camera configuration?

2 comments
AuthorsOpenReview ↗

Response to Reviewer wwwk

Thank you for the review and the helpful suggestions.

Major 1. Static-promoting regularization across scene dynamics. We ran the requested analysis. We split the 202 validation scenes into thirds by their share of dynamic (ground-truth box) pixels and evaluated the sparsity-on and sparsity-off ablation runs of Table 5 (rows 4–5, both without held-out-time supervision) on each third:

Scene motion (dynamic-pixel share)Δ Full PSNRΔ Dyn. PSNRΔ Depth RMSE (m)
Low (0–1.1%)−0.36+0.03−0.01
Medium (1.1–4.8%)−0.20+0.09−0.04
High (5.0–26%)−0.09+0.14−0.06

Δ = sparsity on − off; pooled by image count, the thirds reproduce Table 5. The fixed weight does not penalize highly dynamic scenes: there its full-image cost is smallest and its dynamic-region and depth gains are largest. A motion-adaptive weight is a natural extension, which we will mention.

Major 2. LiDAR sparsity. DAF is built for sparse supervision. On Waymo, LiDAR covers only about 5% of the supervised pixels and nothing beyond 75 m, so most of each image is supervised by the down-weighted, LiDAR-anchored teacher. The per-frame anchor is a weighted median over thousands of returns, and frames with fewer than 32 returns contribute no teacher targets. LiDAR is never used at inference, and on nuScenes and Argoverse 2 the model gives the lowest metric depth error of the evaluated methods with no alignment (Table 4).

Major 3. Same camera configuration. See the general response: with only the three front cameras, X-Splatter is competitive with the baselines' reported results on the full image and better on moving objects, and adding the side cameras improves even the front views.

Thank you for your thorough response to my review. I will maintain my current rating.

Reviewer_voitOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents X-Splatter, a feed-forward framework for metric 4D Gaussian reconstruction from calibrated surround-view video and known ego poses. Depth Anchoring Fine-tuning (DAF) adapts a VGGT-Ω geometry backbone using sparse LiDAR measurements and low-weight, LiDAR-aligned teacher depth. The adapted geometry is then frozen during appearance and motion learning. Static–Dynamic Decomposition Learning (SDDL) splits Gaussian opacity between shared static and motion-warped dynamic contributions, excludes a source frame's own dynamic Gaussians when reconstructing that frame, and combines static-promoting regularization with held-out-time supervision. On five-camera Waymo evaluation, the model achieves 27.45 dB full-image PSNR, 23.44 dB dynamic-region PSNR, and 3.55 m metric-depth RMSE.

Critical strengths

The system addresses a consequential combination of geometric and temporal requirements. Adapting geometry before freezing it for motion learning is a coherent way to reduce geometry–motion ambiguity, while shared static Gaussians exploit observations across cameras and time.

The system achieves substantial practical gains without motion annotations in the default training configuration.

The geometric evaluation is more informative than image reconstruction alone. Table 3 measures whether adjacent cameras place the same static surface at the same 3D location and also reports depth error against LiDAR, avoiding the interpretation that agreement alone implies correctness.

Critical weaknesses

The experiments do not sufficiently isolate the contribution of the proposed integration from inherited capabilities and training differences. The "w/o DAF" experiment substitutes the released depth head without retraining while retaining the adapted backbone and downstream heads (Appendix F), so it measures a head replacement and its compatibility with adapted features rather than the complete DAF training strategy. The joint-processing ablation removes both backbone cross-camera processing and the added motion-head cross-camera attention, leaving the incremental benefit of the latter unresolved. Table 5 also lacks a full-model-minus-sparsity variant. Finally, STORM and DGGT are evaluated from checkpoints trained on three front-facing cameras without retraining for the five-camera setting (Sec. 4.1).

The evidence for learning a physically meaningful decomposition and motion is indirect. Image reconstruction can improve without establishing accurate velocities or static–dynamic assignments. The paper reports no direct motion or decomposition metrics, such as End-Point Error in 3D (EPE3D), Acc5, Acc10, and angular error reported by STORM.

Questions for authors

Can you separately disable the motion-head cross-camera attention while retaining joint processing in the frozen backbone? Can you also compare the current DAF-then-freeze strategy with a matched alternative, such as continuing to optimize the DAF-initialized geometry during 4D training? These comparisons would help distinguish the added motion fusion and staged training from the inherited geometry prior.

Can you provide a full-model-minus-sparsity ablation?

Is a comparison with a baseline adapted to all five cameras available? If not, can you report a matched evaluation restricted to the three front-facing cameras used to train the released baseline checkpoints, alongside per-camera results? This would help quantify the role of the camera-configuration mismatch in the surround-view gains.

Can you evaluate scene-flow accuracy on Waymo using the protocol and metrics reported by STORM (e.g., EPE3D, Acc5,Acc10, and angular error), including an ablation of SDDL?

1 comment
AuthorsOpenReview ↗

Response to Reviewer voit

Thank you for the careful review and the positive assessment.

Q1a. Motion-head cross-camera attention. Removing only this attention (Table 5 protocol) changes the results by at most 0.05 dB and 0.02 m (27.01 / 22.66 dB and 3.60 m, against 26.97 / 22.61 dB and 3.62 m). It keeps every stage operating on all cameras jointly for consistency of the design and is not a claimed contribution; the gain comes from joint processing in the backbone (without any cross-camera processing: 26.63 / 22.32 dB and 4.31 m, Table 5).

Q1b. DAF-then-freeze. Freezing is a deliberate design choice: once DAF fixes the metric scale, the frozen geometry cannot trade depth for motion during 4D training, the ambiguity described in Sec. 1. The resulting metric depth is the most accurate among the evaluated methods on Waymo (Table 2) and on nuScenes and Argoverse 2 without alignment (Table 4). The "w/o DAF" row isolates the adapted depth head inside the trained pipeline, as stated in Appendix F; the full DAF recipe is evaluated against the released backbone in Table 8 (5.05 → 3.55 m).

Q2. Sparsity. Table 5 and Appendix G consistently characterize the sparsity term as a regularizer that trades full-image PSNR (0.2–0.6 dB) for dynamic-region quality. Splitting the 202 validation scenes into thirds by their share of dynamic pixels (same run pair as Table 5), the term costs 0.36 dB of full-image PSNR in the least dynamic third but only 0.09 dB in the most dynamic one, where its dynamic-region gain is largest (+0.14 dB): the trade is most favourable where decomposition matters most. The largest measured SDDL gain comes from held-out-time supervision, which the full model includes.

Q3. Camera configuration and per-camera results. See the general response.

Q4. Motion accuracy. X-Splatter targets reconstruction and rendering, so we evaluate motion through rendering moving objects at unobserved times. On extrapolated frames, where moving objects must be placed by the predicted motion of the last input frame, X-Splatter's dynamic-region PSNR on the three front cameras is 22.54 dB, against 20.75 dB for STORM (given all five cameras) and 18.22 dB for DGGT.

Comments on the paper

1 comment

We thank the reviewers for their positive assessment and constructive comments.

We would like to kindly clarify the aim of X-Splatter. It is not designed to outperform existing methods given the same input, but to enable surround-view reconstruction: it processes all cameras of the rig jointly, while the dynamic feed-forward baselines are trained on at most the three front cameras. Using the whole rig makes the reconstruction complete around the vehicle and improves its quality.

The reviewers' question about the baselines' three-camera setting also lets us show what the surround views add, and we thank them for it. We evaluated X-Splatter with only the three front cameras as input and with all five, scoring both on the three front cameras (Waymo, the clips and held-out frames of Table 2):

MethodInput camerasFull PSNR↑Dyn. PSNR↑Depth RMSE (m)↓
STORM*326.3822.10–
DGGT*327.4120.99–
X-Splatter327.3923.333.85
X-Splatter527.7323.663.57

*As reported (our Table 1).

Two results follow. With fewer cameras than it was trained with, X-Splatter is competitive with the baselines in their own setting on the full image (27.39 dB, against 26.38 dB for STORM and 27.41 dB for DGGT as reported) and better on moving objects (23.33 dB, against 22.10 and 20.99 dB). Adding the two side cameras then improves even the front views: PSNR rises from 27.39 to 27.73 dB (23.33 to 23.66 dB on moving objects) and depth error falls from 3.85 to 3.57 m.

Per-camera results (Waymo, full / dynamic PSNR; input cameras as listed):

CameraSTORM (5 cameras in)DGGT (1 camera at a time)X-Splatter (5 cameras in)
Front21.72 / 21.1925.16 / 21.0729.66 / 24.49
Front-left22.35 / 20.8020.65 / 18.6326.87 / 22.80
Front-right22.71 / 21.3320.16 / 18.8726.63 / 23.32
Side-left23.30 / 19.9918.03 / 17.1727.14 / 22.68
Side-right23.02 / 20.2817.41 / 16.9526.97 / 23.34

X-Splatter leads on every camera, and its side views are as good as its front-side views. We add both tables and a five-camera qualitative comparison to the appendix of the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.