acceptodds
Under review as a conference paper at ICLR 2027

S2D-Splat: Static-First Gaussian Reconstruction of Dynamic Driving Scenes with Layer-Wise Feature Guidance

Abstract

Moving objects introduce cross-frame inconsistencies that complicate camera estimation and static geometry recovery in driving scenes. Recent feed-forward Gaussian methods explicitly separate static and dynamic primitives, but often estimate camera parameters and geometry from shared features that jointly encode both components, leaving static reconstruction susceptible to dynamic interference. To address this challenge, we propose S2D-Splat, a feed-forward framework for static-first Gaussian reconstruction with layer-wise feature guidance. Our framework separates static and dynamic features before camera and geometry estimation, enabling the static branch to recover camera parameters and static 3D Gaussians with reduced dynamic interference. Intermediate static features are cached at each layer and reused as fixed conditioning at corresponding layers of the dynamic branch. This asymmetric information flow preserves the established static representations while providing stable geometric context for reconstructing dynamic 3D Gaussians. An instance-level motion decoder further queries the fused static and dynamic representations to estimate object trajectories, supporting motion-aligned Gaussian fusion and temporal interpolation. Experiments on Waymo and zero-shot evaluations on nuScenes and the MM-AU accident dataset demonstrate that S2D-Splat outperforms existing approaches on most evaluated metrics and generalizes to highly dynamic, safety-critical driving scenarios.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Autonomous Driving, 3D Gaussian Splatting, Feed-Forward Scene Reconstruction, Motion Prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RUinOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce a method for feed-forward driving scene reconstruction, proposing separation of static and dynamic features before camera and geometry estimation and reducing dynamic interference. Static branch works independently and is guiding dynamic branch, preserving the established static representations while supplying geometric context for dynamic reconstruction. Additionally, authors develop instance-level motion decoder for estimating object trajectories. Compared to other methods, S2D-Splat achieves state-of-the-art performance.

Critical strengths
  1. Authors show that splitting static and dynamic parts earlier in backbone improves model geometry, better parts separation and reconstruction quality.
  2. Method achieves state-of-the-art performance on reconstruction and novel view synthesis.
Critical weaknesses
  1. Model depends on GT bounding boxes and uses sharper masks from Sam2, while baselines split on dynamic and static with learned approach (DGGT predicts dynamic masks, StreetForward learns dynamic probabilities), which makes it unclear whether improvement comes from that or from proposed split before geometry estimation. Additionally authors used VGGT-Omega backbone, while baselines run on earlier models (VGGT and Pi3).
Questions for authors
  1. Mask ablation
  • Sam2 is running on GT boxes, but in practice this boxes are not available. How would model perform with automatic mask generation using Sam2?
  • As tokens are coarse, how sensitive model is for boundaries of masks being used for dynamic-static split? Could you evaluate the model with 2d boxes used instead of Sam2 masks? Please also provide additional visual results, as in Fig. 3 objects are either far or blurry and it would be interesting to see on examples with big cars in front of us and their boundaries (as in Fig.4).
  1. As Instance-level motion decoder is one of central contributions claimed in paper, evaluate 3d tracking and compare to DGGT and StreetForward.
  2. Does model better split background and objects and static part is not carrying dynamic information? Could you provide qualitative comparison with baselines for background and objects rendered independently?
3 comments
AuthorsOpenReview ↗

Response to Reviewer RUin (1/2)

We sincerely thank the reviewer for the detailed evaluation. We appreciate the recognition of early static-dynamic separation and the state-of-the-art reconstruction and novel-view synthesis results, as well as the questions about prompt and backbone fairness, mask boundaries, 3D tracking, and qualitative separation. The responses below address these points.

Q1. Mask ablation

  • Sam2 is running on GT boxes, but in practice this boxes are not available. How would model perform with automatic mask generation using Sam2?
  • As tokens are coarse, how sensitive model is for boundaries of masks being used for dynamic-static split? Could you evaluate the model with 2d boxes used instead of Sam2 masks? Please also provide additional visual results, as in Fig. 3 objects are either far or blurry and it would be interesting to see on examples with big cars in front of us and their boundaries (as in Fig.4).

Prompt availability. For benchmark evaluation, test-set 3D boxes are projected to 2D and used as SAM2 prompts. In practice, a 2D detector can generate candidate boxes, but a human currently selects the key moving instances before SAM2 propagates their masks. We have not yet quantified fully automatic detector-generated prompts; we agree this limits scalability, and automatic prompt generation and selection are future work (discussed in Sec. 5).

Boundary sensitivity. We added a test-time ablation that replaces SAM2 masks with 2D box regions, labeling a patch as dynamic when its center falls inside a box; the trained model is otherwise unchanged. Performance remains close to the SAM2-mask setting (Table 5 in Sec. 4):

MethodPSNRSSIMD-RMSE
2D-Box Mask25.6600.8092.387
S2D-Splat (SAM2 mask)25.9530.8142.152

This indicates robustness to coarser mask boundaries: patch tokens are already coarse, and the boundary branch can recover static cues from mixed boundary tokens. The ablation tests boundary coarseness, not missed-object errors or automatic instance-selection quality.

Additional visuals. We added a close-vehicle decomposition example (Appendix D, Fig. 4) comparing GT, DGGT, StreetForward, and S2D-Splat, with full/static/dynamic renders showing large nearby vehicles and their boundaries. S2D-Splat isolates the vehicles in the dynamic render with no obvious residuals in the static render.

AuthorsOpenReview ↗

Response to Reviewer RUin (2/2)

Q2. As Instance-level motion decoder is one of central contributions claimed in paper, evaluate 3d tracking and compare to DGGT and StreetForward.

We evaluate 3D tracking with APD$_{3D}$ on Waymo: points densely sampled on each dynamic instance are displaced with the predicted motion between adjacent frames and compared with the ground-truth rigid motion derived from box trajectories (thresholds: 0.1/0.3/0.5/1.0 m). All methods are evaluated under the same protocol (Table 2 in Sec. 4):

MethodAPD$_{3D}$
DGGT13.14%
StreetForward11.47%
S2D-Splat19.20%

S2D-Splat outperforms both baselines by a clear margin. DGGT and StreetForward estimate motion without an explicit instance-level motion decoder (StreetForward predicts per-point velocities), whereas our decoder queries the fused static-dynamic representation to predict each instance's rigid motion.

Q3. Does model better split background and objects and static part is not carrying dynamic information? Could you provide qualitative comparison with baselines for background and objects rendered independently?

Yes. The static component of S2D-Splat carries less dynamic information, as reflected by the static-region depth accuracy on Waymo (Table 2 in Sec. 4):

MethodStatic D-RMSE
DGGT3.745
StreetForward2.913
S2D-Splat2.057

Lower static-region D-RMSE indicates that the static reconstruction is cleaner and less affected by dynamic interference.

Qualitatively, we added a comparison of full, static, and dynamic renders for GT, DGGT, StreetForward, and S2D-Splat (Appendix D, Fig. 4). S2D-Splat retains the background in the static render and isolates the vehicles in the dynamic render with no obvious residual vehicle artifacts, while StreetForward and DGGT show visible dynamic residuals in their static renders. However, we acknowledge that the model does not achieve perfect static-dynamic separation, and we have revised the corresponding claim in the manuscript.

Thank you for the clarifications, additional experiments and visualizations. Your response addressed most of my concerns. I will maintain my original score.

Reviewer_hSxQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a feed-forward dynamic 3DGS reconstruction method for urban driving scenarios. The proposed method starts with a static vs. dynamic scene part labeling and then follows a two stage pipeline in which first the static scene part is reconstructed and subsequently the dynamic parts are reconstructed too.

Critical strengths

The authors propose a plausible, well-motivated pipeline for dynamic urban scene reconstruction which yields competitive results.

Critical weaknesses
  • While the proposed multi-stage approach makes the method more modular, it also has the disadvantage that early incorrect dynamic segmentation decisions cannot be reverted in later stages. The system hence relies fully on the quality of this motion segmentation, which therefore deserves its own evaluation.

  • Limited evaluation: While the evaluation contains standard neural reconstruction metrics like PSNR, SSIM and D-RMSE, (1) the quality of the static vs. dynamic segmentation is not separately assessed. (2) the quality of the estimated motion field is not assessed. It would be interesting to visualize the motion field. A quantitative evaluation of the estimated scene flow would also be insightful.

  • A discussion of limitations and some failure cases would be a valuable edition to the paper.

Questions for authors

Since the framework is focused on static-first reconstruction, the quality of this first step has crucial impact on the overall reconstruction result. Is there any way to revert/over-rule decomposing decisions from the first stage?

1 comment
AuthorsOpenReview ↗

Response to Reviewer hSxQ

We thank the reviewer for the careful evaluation. We appreciate the recognition of the well-motivated static-first pipeline and competitive performance, as well as the constructive comments on error propagation, segmentation quality, motion evaluation, and limitations. Our responses are provided below.

Q1. Since the framework is focused on static-first reconstruction, the quality of this first step has crucial impact on the overall reconstruction result. Is there any way to revert/over-rule decomposing decisions from the first stage?

The current pipeline does not feed later reconstruction errors back to revise the initial instance masks. Missed instances or substantially incorrect masks can therefore affect both reconstruction stages. The qualitative comparison of full, static, and dynamic renders in Appendix D, Fig. 4 shows that the static--dynamic mask partition is of good quality. In addition, the boundary branch can recover static cues from mixed boundary tokens, and our test-time 2D-box ablation shows that performance remains close under a coarser mask partition; however, neither result demonstrates that arbitrary segmentation errors can be corrected. We acknowledge this limitation and discuss it in Sec. 5.

Quality of the static--dynamic separation. We added a test-time ablation that replaces SAM2 masks with 2D box regions, labeling a patch as dynamic when its center falls inside a box; the trained model is otherwise unchanged. The results remain close to the SAM2-mask setting (Table 5 in Sec. 4):

MethodPSNRSSIMD-RMSE
2D-Box Mask25.6600.8092.387
S2D-Splat (SAM2 mask)25.9530.8142.152

We also added a qualitative comparison of full, static, and dynamic renders against GT, DGGT, and StreetForward (Appendix D, Fig. 4). In the illustrated scene, S2D-Splat reduces visible dynamic-object residuals in the static render and isolates the vehicles in the dynamic render, while DGGT and StreetForward show visible residuals. We do not yet provide a standalone segmentation metric or a quantitative measure of leakage between the two rendered components.

Camera, static geometry, and motion metrics. We added camera-pose AUC, static-region D-RMSE, and APD$_{3D}$ comparisons on Waymo (Table 2 in Sec. 4). AUC@3$^{\circ}/AUC@30/AUC@30^{\circ}$ is the standard camera-pose AUC (higher is better):

MethodAUC@3$^{\circ}$AUC@30$^{\circ}$Static D-RMSEAPD$_{3D}$
DGGT0.6190.9713.74513.14%
StreetForward0.6880.9832.91311.47%
S2D-Splat0.7140.9842.05719.20%

For motion, we evaluate dense 3D motion: points densely sampled on each dynamic instance are displaced with the predicted motion between adjacent frames and compared with the ground-truth rigid motion derived from box trajectories.

  • Metric: APD$_{3D}$, the average fraction of points whose 3D endpoint error falls below 0.1/0.3/0.5/1.0 m thresholds (Table 2 in Sec. 4).
  • Qualitative results: predicted motion tracks and flow against the rigid ground truth (Appendix D, Fig. 5); motion interpolation on nuScenes (Appendix D, Fig. 6).
Reviewer_4YSkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents S2D-Splat, a feedforward framework for reconstructing dynamic driving scenes from unposed images. It separates static and dynamic features before camera and geometry estimation, conditions dynamic reconstruction on cached static tokens at each layer, and predicts instance poses for motion-aligned Gaussian fusion. The reported improvements on Waymo and zero-shot evaluations are promising. My main concerns are the lack of direct measurements of the claimed camera and static-geometry improvements, and incomplete specification of test-time auxiliary inputs and evaluation alignment. These concerns motivate my current recommendation of weak rejection.

Critical strengths

The layer-wise token replacement provides a concrete mechanism for preserving the static branch's representations while allowing dynamic tokens to access their context. Table 4 provides evidence that this conditioning is useful: removing guidance reduces PSNR from 25.953 to 25.521 and increases D-RMSE from 2.152 to 2.539.

The reported improvements extend to dynamic regions and cross-dataset evaluation. On Waymo, dynamic-region PSNR increases from StreetForward's 20.252 to 22.043, while D-RMSE decreases from 5.439 to 4.065. The nuScenes novel-view results also improve, providing evidence of transfer beyond the training dataset.

Critical weaknesses

The incremental value of layer-wise cached conditioning needs stronger evidence. MUT3R suppresses dynamic interference in early attention layers, while DrivingScene learns a static prior before training dynamic residuals with the static backbone frozen. D4RT introduces independent point-query decoding, and PointForward uses 3D boxes and canonical instance representations for motion propagation. These differ from the proposed architecture and establish precedents for its broader design principles. The unresolved question is whether conditioning at every layer improves over a simpler static-conditioning alternative under matched settings. The current guidance ablation compares cached conditioning with its removal, leaving this distinction untested.

The central explanation is not directly validated. The paper attributes the gains to reduced dynamic interference in camera and static geometry estimation, but reports neither camera pose errors nor separate static-region geometry metrics. As the paper notes, the decoupling ablation measures the combined design: it changes the separator, reconstruction branches, and prediction heads together. It therefore does not isolate the effect of suppressing dynamic keys on camera estimation or static geometry. Moreover, masking dynamic keys in Eq. (1) cannot guarantee that static tokens contain no dynamic information: the DINOv3 encoder can already mix spatial context, and the boundary branch incorporates features derived from mixed tokens. The absolute isolation statements in Section 3.1 are stronger than the architecture establishes, even if the mechanism reduces interference in practice.

Important evaluation details remain unclear. Appendix B specifies one box prompt per dynamic instance in one inference frame and explains that training boxes come from projected annotated 3D boxes. The source of test-time prompts and the selection of moving instances remain unspecified. Although all baselines are evaluated with the same frame configuration, their initialization, adaptation, auxiliary inputs, and motion supervision are not described sufficiently to assess comparability. Appendix A uses sky masks for rendering, but the construction of target-view masks is unclear, as are the source of target camera parameters and their alignment with the predicted scene. Together with the missing implementation details for unobserved-time poses and Gaussian fusion, these gaps make the evaluation pipeline difficult to reconstruct.

The motion-decoder comparison provides limited evidence of object-pose accuracy. The decoder ablation uses point tracking and instance clustering, with motion estimated from tracked centroids, but reports no object translation/rotation errors or comparison with tracked points followed by rigid-transform fitting. Such a comparison would better test the advantage of predicting object-level poses. Separately, the MM-AU results support zero-shot observed-view reconstruction of pre-collision accident-dataset clips. They do not evaluate collision or post-impact dynamics, or held-out-view synthesis on MM-AU; conclusions about safety-critical generalization should be restricted to the tested setting.

Questions for authors

Does conditioning on cached static tokens at every layer improve over simpler static conditioning? Can the authors compare it with an input-only fusion baseline under otherwise matched settings?

Does the proposed separation improve camera accuracy and static geometry? Please provide camera translation/rotation errors and static-region depth metrics, including an ablation that retains both branches and heads but removes only dynamic-key masking.

How are test-time box prompts obtained and moving instances selected? How are target-view sky masks and target camera parameters obtained, and how are the target cameras aligned with the predicted scene? Please clarify baseline initialization, training, auxiliary inputs, and motion supervision.

Beyond the world-space pose conversion described in Section 3.3, please specify pose construction at unobserved timestamps, rotation parameterization and interpolation, and the fusion and deduplication of motion-aligned Gaussians. Can object-level motion errors and dynamic-region results establish the decoder's advantage over point tracking followed by rigid-transform fitting?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 4YSk (1/2)

We sincerely thank the reviewer for the careful evaluation. We appreciate the recognition of layer-wise token replacement and the promising Waymo and cross-dataset results, as well as the constructive questions about direct evaluation details. Our point-by-point responses follow.

Q1. Does conditioning on cached static tokens at every layer improve over simpler static conditioning? Can the authors compare it with an input-only fusion baseline under otherwise matched settings?

We added an input-only fusion baseline that keeps the separator, two-stage pipeline, heads, and training recipe unchanged, but injects cached static tokens only at the input of the dynamic branch. The results are:

MethodPSNRSSIMD-RMSE
w/o Guidance25.5210.7952.539
Input-only Fusion25.6370.8062.297
S2D-Splat25.9530.8142.152

Input-only conditioning improves over removing guidance, while layer-wise injection further improves the results. The gap arises because layer-wise reinjection allows dynamic tokens at every layer to attend to the clean static tokens cached from the static branch, which provide a reliable geometric and camera reference during dynamic feature refinement, whereas in the input-only baseline the injected static tokens gradually drift away from the accurate static background in subsequent layers (Sec. 4.3).

Q2. Does the proposed separation improve camera accuracy and static geometry? Please provide camera translation/rotation errors and static-region depth metrics, including an ablation that retains both branches and heads but removes only dynamic-key masking.

We added camera-pose AUC and static-region D-RMSE comparisons on Waymo. AUC@3$^{\circ}/AUC@30/AUC@30^{\circ}$ is the standard camera-pose AUC: the area under the curve of the fraction of image pairs whose relative rotation and translation errors fall below the corresponding angular thresholds; higher is better. We also added a controlled ablation that retains both branches and the dedicated heads while removing only the −∞-\infty mask on dynamic key positions.

MethodAUC@3$^{\circ}$AUC@30$^{\circ}$Static D-RMSEAPD$_{3D}$
DGGT0.6190.9713.74513.14%
StreetForward0.6880.9832.91311.47%
S2D-Splat0.7140.9842.05719.20%
AblationPSNRSSIMD-RMSE
w/o Dyn-Sup (no dynamic-key mask)25.3940.7892.664
S2D-Splat25.9530.8142.152

These results support the effectiveness of dynamic-key suppression and show improved camera and static-depth metrics against the compared dynamic baselines (Table 5 in Sec. 4). We agree that the masking operation does not guarantee zero dynamic information in static features. We have revised the manuscript accordingly: dynamic-key masking substantially reduces, but does not completely eliminate, dynamic interference in the static representation.

AuthorsOpenReview ↗

Response to Reviewer 4YSk (2/2)

Q3. How are test-time box prompts obtained and moving instances selected? How are target-view sky masks and target camera parameters obtained, and how are the target cameras aligned with the predicted scene? Please clarify baseline initialization, training, auxiliary inputs, and motion supervision.

These details are now specified in Appendix C.

For the reported benchmark metrics, test-set 3D boxes are projected to the image plane to provide the 2D prompts used by SAM2. The reconstruction model consumes these 2D prompts, not 3D boxes. When 3D annotations are unavailable, our practical alternative uses a 2D detector to generate candidate boxes, followed by human selection of key moving instances; SAM2 then propagates the prompted mask across the clip. We acknowledge that this alternative is not fully automatic and that the benchmark prompt protocol uses test annotations to obtain the 2D prompts.

For Waymo target-view rendering, each target view is one of the four intermediate frames between two adjacent reconstructed keyframes. As part of offline preprocessing, we run SegFormer on every RGB frame used in the benchmark, including both the eight reconstruction keyframes and the intermediate target frames, and cache the resulting sky masks by frame index. The keyframe masks are used when constructing the sky model, while each target-view mask is used only for sky compositing when rendering that target novel view. The rendering camera is not the ground-truth target pose; it is interpolated from the predicted cameras of the two bracketing keyframes, with translation and intrinsics linearly interpolated and rotation obtained by SLERP after projecting the endpoint rotations to valid rotations. Because both the keyframe cameras and Gaussians are in the same predicted coordinate system, the interpolated target camera is already aligned with the predicted scene and requires no additional rigid alignment. When rendering a novel viewpoint that is not present in the dataset and therefore has no pre-generated target-view sky mask, we follow the sky-Gaussian fusion rule and composite the sky render where the accumulated scene opacity is low. We also clarify that baselines use the same eight-frame, single-camera RGB input setting; released pretrained weights are used where available, and methods requiring adaptation are fine-tuned with their own recipes without architectural changes; motion supervision follows each baseline's respective method.

Q4. Beyond the world-space pose conversion described in Section 3.3, please specify pose construction at unobserved timestamps, rotation parameterization and interpolation, and the fusion and deduplication of motion-aligned Gaussians. Can object-level motion errors and dynamic-region results establish the decoder's advantage over point tracking followed by rigid-transform fitting?

Appendix B now specifies camera construction at unobserved timestamps, including linear interpolation of translation and intrinsics and quaternion SLERP for rotation. For dynamic Gaussian fusion, predicted instance motion applies translation and rotation to the Gaussians from the two bracketing frames. A nearest-frame-first visibility gate renders the farther-frame Gaussians only where the nearer render has insufficient accumulated opacity, reducing duplicate surfaces and holes.

On Waymo, we evaluate dynamic motion with APD$_{3D}$:

  • Protocol: points densely sampled on each dynamic instance are displaced with the predicted motion between adjacent frames, and the resulting 3D positions are compared with the ground-truth rigid motion derived from box trajectories.
  • Metric: the average fraction of points whose 3D endpoint error falls below the 0.1/0.3/0.5/1.0 m thresholds; higher is better.

The results are:

MethodAPD$_{3D}$ (%)
DGGT13.14
StreetForward11.47
S2D-Splat19.20

The w/o Decoder ablation replaces the learned decoder with instance clustering and TAPIP3D tracking. Its reconstruction results are:

VariantPSNRSSIMD-RMSE
w/o Decoder25.8410.8082.269
S2D-Splat25.9530.8142.152

Together, these results establish the decoder's advantage: the learned decoder directly predicts instance-level motion, whereas replacing it with point tracking degrades all reconstruction metrics.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the reviewers for their time and effort in reviewing our manuscript, as well as for their insightful feedback and recognition of the strengths of our work:

  • Static-First Decoupling: S2D-Splat separates static and dynamic components before camera and geometry estimation, reducing dynamic interference and providing a coherent basis for reconstructing complex driving scenes. (4YSk, hSxQ, RUin)
  • Static-to-Dynamic Guidance: The reconstructed static representation guides dynamic reconstruction through cached static tokens and an independent static branch, preserving stable geometric context while recovering dynamic objects. (4YSk, RUin)
  • Strong Empirical Results: S2D-Splat achieves competitive reconstruction and novel-view synthesis performance, with improvements on Waymo, and cross-dataset evaluation. (4YSk, hSxQ, RUin)

We have revised the main paper according to your comments, with all revisions highlighted in blue. Specifically, we added three direct evaluations: camera-pose AUC, static-region D-RMSE, and 3D motion tracking. We also added three controlled ablations: dynamic-key masking, input-only fusion, and motion-decoder replacement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.