acceptodds
Under review as a conference paper at ICLR 2027

Structured Observation Modeling for Feed-Forward 4D Driving Reconstruction

Abstract

Feed-forward 4D driving reconstruction aims to recover coherent dynamic scenes from sparse multi-camera temporal observations, while existing methods still struggle with coherent reconstruction from such sparse observations. Through a data-centric analysis, we find that farther target views receive sparser and more distributed geometric support across cameras and time, while driving datasets exhibit heterogeneous reconstruction conditions, motivating structured observation modeling and heterogeneous training, respectively. In this paper, we propose SOM4D, a feed-forward framework designed to better model sparse multi-view temporal observations for 4D driving scene reconstruction. Specifically, we introduce Pose-Time Relational Adaptation (PTRA) to explicitly model camera–time relations within a frozen visual-geometry backbone. We further introduce Hierarchical Observation Consolidation (HOC) to integrate multi-level features and shared scene context while preserving dense spatial details. To improve explicit scene reconstruction, we develop a Corrective Gaussian Head (CGH) that decodes dynamic Gaussians and progressively corrects them through geometry-aware scene correction and recurrent source-render feedback. Additionally, we train SOM4D on mixed-data to broaden the training conditions and assess the benefit of heterogeneous training on Waymo. Extensive experiments under both Waymo-only and mixed-data settings validate the proposed model design and the benefit of heterogeneous training.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Autonomous Driving, Gaussian Splatting, Feed-forward Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zVEXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SOM4D, a feed-forward framework for 4D driving scene reconstruction based on VGGT-Omega. It introduces three modules:
1 Pose-Time Relational Adaptation (PTRA) to model camera and time relations, 2 Hierarchical Observation Consolidation (HOC) combine multi-level features, 3 Corrective Gaussian Head (CGH) refine dynamic Gaussians using source observations. The paper also studies mixed-data training with Waymo, nuScenes and Argoverse 2. Experiments mainly on Waymo show improvements in rendering quality and depth accuracy, with ablation studies evaluating each module.

Critical strengths

None

Critical weaknesses

The technical novelty appears limited.

The method builds on a strong VGGT-Omega backbone and combines relatively familiar attention, feature fusion, and refinement designs. Although Table 3 shows improvements, removing PTRA also removes its feature stream, so the ablation does not isolate the benefit of the proposed relational design.

A simpler baseline would help clarify the contribution beyond the pretrained model. This limits my confidence in the significance of the architectural contributions.

Questions for authors

1 Does PTRA improve performance because of the camera–time relations, or simply because it adds more attention layers? Can you compare it with the same module without the relational biases?

2 How fast is the method, and how much GPU memory does it use compared with STORM and SLARM?

3 The shared context only improves full-image PSNR by 0.01 dB in Table 3. How much extra computation does it add, and is the small gain worth the cost?

Overall, I think this is a well-developed and solid piece of work. I will consider raising my score after reading the other reviewers' comments and the authors' response.

6 comments

Response to W1 & Q1: Relational biases and contributions beyond Omega.

Thank you for raising these questions. As described in Sec. 3.3, PTRA aims to adapt frozen Omega features to sparse multi-view temporal observations through structured camera–time relational modeling. Learned relative biases are one component of this adaptation.

To examine the role of relational biases beyond additional attention layers, we compare PTRA with the same attention module but without learned relative biases. To evaluate the contribution beyond the pretrained backbone, we also construct a simpler baseline using the same frozen Omega + Native DPT + Basic Gaussian Head. The results are as follows (Waymo NVS PSNR, dB):

ConfigurationNon-sky NVSDynamic NVS
Full SOM4D26.8622.51
Without Relative Bias26.8222.47
Same Omega + Native DPT + Basic Gaussian Head25.4121.11

For relational modeling, removing the biases while retaining the attention layers yields 26.82/22.47 dB, compared with 26.86/22.51 dB for Full—a reduction of 0.04 dB in each metric. The submitted Table 3 further shows that removing the complete PTRA pathway and its feature stream reduces non-sky/dynamic NVS by 0.67/0.97 dB, supporting the contribution of the integrated camera–time adaptation.

For the backbone attribution, the simpler baseline achieves 25.41/21.11 dB with the same frozen Omega, substantially below Full SOM4D, which improves non-sky/dynamic NVS by 1.45/1.40 dB, respectively. This demonstrates that the proposed adaptation and reconstruction architecture provides measurable benefits beyond the pretrained backbone, consistent with our paper's contribution. (Revised in Sec. 4.4 and Appendices E.1–E.2.)

Response to Q2: Runtime and GPU memory.

Thank you for asking. We report this comparison in Appendix D.2, Table 13, summarized below. Compared with STORM, SOM4D’s mean forward latency is 99.6 ms higher, and its peak GPU memory is 4.85 GiB higher. Compared with SLARM, the corresponding increases are 115.8 ms and 3.43 GiB.

MethodMean forward latency (ms)Peak GPU memory (GiB)
STORM127.03.23
SLARM110.84.65
SOM4D226.68.08

While using more resources, SOM4D improves non-sky/dynamic NVS by 0.49/0.70 dB and REC by 1.10 dB, and reduces non-sky depth RMSE from 4.95 to 4.11 m, relative to our matched Waymo-only SLARM rerun (Table 1). For reconstruction from fixed sparse observations, we consider this a reasonable quality–resource trade-off. Reducing latency and memory remains an important direction for future work.


Response to Q3: Shared-context computation and benefit.

Thank you for raising this question. Under the same counting convention, Shared Context increases model FLOPs by 1.09% and peak GPU memory by approximately 0.04 GiB. The submitted joint context/subset ablation reports +0.10 dB dynamic NVS (Table 3). Given the limited additional computation and memory, we consider this an acceptable trade-off in the evaluated setting. Future work will further streamline context aggregation and readback to improve efficiency. (Added in Appendix E.4.)

Dear Reviewer zVEX,

We sincerely appreciate your thoughtful feedback and the time you have devoted to reviewing our work. We have provided further experimental evidence and uploaded a revised manuscript with changes highlighted in blue.

As the November 18 (AoE) deadline approaches, we would welcome any remaining questions. We hope our additional results help address your concerns and would be grateful if you could reconsider your evaluation.

Thank you again for your time and consideration.

Reviewer_vXW2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SOM4D, a feed-forward method for reconstructing dynamic driving scenes from sparse multi-camera and multi-time observations. It introduces PTRA for camera-time relation modeling, HOC for multi-level feature fusion, and CGH for Gaussian decoding and correction. The paper also studies geometric support in three driving datasets and trains a model on Waymo, nuScenes, and Argoverse 2.

Critical strengths
  • The problem is important and practical, and the full reconstruction pipeline is well designed.
  • SOM4D shows consistent gains over the reproduced STORM and SLARM baselines, especially in depth accuracy and harder target views.
  • The Waymo-only and Mix3 comparisons are reasonably controlled.
  • The experiments and appendix are detailed, and the authors are clear about some limitations of their data analysis.
Critical weaknesses
  1. Novelty claim

    PTRA is designed to model camera-time relations within a frozen VGGT backbone. However, its cross-view and temporal attention are closely related to prior works such as 4D-VGGT[1] and DynamicVGGT[2]. The paper needs to explain its technical difference more clearly. Separate ablations for the view branch, temporal branch, and relative attention biases would also help.

  2. Missing comparisons

    The direct comparison mainly covers STORM and SLARM. Despite 4D-VGGT and DynamicVGGT mentioned above, DGGT follows a very similar Waymo input protocol and should be compared or discussed. The reproduced SLARM results are lower than its published results, and this difference should be explained.

  3. Role of the data-centric analysis

    The proposed support metric is well defined, but it is still a static geometric coverage proxy. It does not cover dynamic surfaces or directly show that PTRA/HOC benefits come from the diagnosed observation structure. The analysis is useful as motivation, but may be too weak to stand as a major contribution by itself.

  4. Limited Mix3 conclusions

    Multi-dataset training is not new, and the paper only evaluates one equal mixture of three datasets. There is no ratio study or evaluation on a fully unseen domain. The current experiments support this particular training mixture, but not a general claim about broader domain diversity.

  • Haonan Wang et al. 4D-VGGT: A SpatioTemporal Foundation Model for Dynamic Scene Geometry Estimation. ECCV, 2026.
  • Zhuolin He et al. DynamicVGGT: Learning Dynamic Point Maps for 4D Scene Reconstruction in Autonomous Driving. CVPR, 2026.
Questions for authors
  • What is the main technical difference between PTRA and the relation/temporal modules in 4D-VGGT and DynamicVGGT?
  • Why are DGGT not included in the experiments, and why are the reproduced SLARM results lower than the published ones?
  • Does the support score predict reconstruction error after controlling for dataset and target distance?
9 comments

Response to W1 & Q1: PTRA technical distinction and ablations.

Thank you for highlighting these related approaches. 4D-VGGT uses cross-view global fusion and local temporal windows with GRU-based aggregation, while DynamicVGGT computes motion-aware temporal attention independently at each patch position and camera view. PTRA inserts residual patch-token adapters at four stages of a frozen Omega backbone, incorporating relative calibrated camera geometry and temporal offsets. Its same-time cross-camera and same-camera cross-time branches exchange information across spatial patches within each group. Our main distinction lies in how the view and temporal branches are constructed and how they are inserted into the frozen backbone.

Related-work discussion revised in Sec. 2.1; controls added in Appendix E.1 (Table 15; NVS PSNR, dB):

PTRA configurationNon-skyDynamic
View-only26.8022.41
Time-only26.8322.45
Without Relative Bias26.8222.47
Full26.8622.51

Full performs best in both metrics. The no-bias variant retains both attention branches and camera/time embeddings; removing only the relative biases reduces non-sky/dynamic PSNR by 0.04/0.04 dB. Both single-branch variants also perform below Full, supporting the combined view–time configuration.

Response to W2 & Q2: DGGT and SLARM comparability.

Thank you for asking. DGGT addresses unposed 4D reconstruction, jointly estimating camera poses and scene geometry from images without known camera poses. In contrast, SOM4D addresses posed reconstruction, where camera calibration and poses are provided to reconstruct the dynamic scene from multi-view temporal observations. These are different task settings with different input assumptions. We therefore restrict our direct numerical comparisons to posed reconstruction methods under matched protocols, rather than directly ranking unposed and posed approaches. We discuss DGGT as relevant work under the unposed setting.

For SLARM, official pretrained weights are unavailable, so we retrained Waymo-only and Mix3 variants using the same data, inputs, 100k updates, BS32, BF16, and evaluation protocol as SOM4D. Our reproduction uses offline full attention, fresh initialization, and no semantic distillation. Published SLARM-F also uses full attention, but is trained with 200k updates, BS64, semantic alignment, and an additional 3k classification stage. The published 27.49 dB and our 26.67 dB therefore reflect different training configurations. Our reported comparison is conducted under a matched training budget and evaluation protocol. (Revised in Appendix C.3.)

Response to W3 & Q3: Geometric support, reconstruction error, and module gains.

Thank you. We analyze static geometric support to understand sparse observation conditions and inform PTRA/HOC design.

Using one fixed Mix3 checkpoint, we analyzed 203,808 target-view evaluations from 502 scenes across Waymo, nuScenes and Argoverse 2. Controlling for dataset, camera and dataset-specific nonlinear nearest-source distance, +10 pp support is associated with +0.765 dB non-sky PSNR (95% scene-bootstrap CI [+0.652, +0.888]). Adding support reduces five-fold scene-held-out per-target non-sky PSNR prediction RMSE from 3.115 to 3.062 dB, showing predictive value beyond dataset and distance.

Separately, matched 50k Waymo-only ablations show PTRA improves non-sky NVS in all 12 distance/support bins (+0.56–0.78 dB), with larger low-support than high-support point gains within each distance quartile. Dense HOC also improves non-sky NVS in all 12 bins (+0.09–0.31 dB) over w/o HOC, with the same backbone/PTRA/Gaussian head and shared context/subset supervision off in both. Overall dynamic NVS gains are +0.87/+0.23 dB for PTRA/HOC. These results connect the geometric diagnosis to reconstruction error and module benefits across observation conditions. (Added in Appendix E.3.)

Reviewer_cb7sOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies feed-forward 4D driving scene reconstruction from sparse, calibrated multi-camera temporal observations. The proposed SOM4D introduces Pose-Time Relational Adaptation (PTRA) into a frozen visual-geometry backbone and uses Hierarchical Observation Consolidation (HOC) to integrate multi-level features and shared scene context. It then predicts pixel-aligned dynamic Gaussians and updates their attributes through geometry-aware correction and source-render feedback. Gaussian positions evolve according to cubic temporal trajectories. The paper also analyzes geometric support under different observation conditions and investigates mixed training on Waymo, nuScenes, and Argoverse 2. Experiments report improvements in appearance and depth reconstruction over reproduced baselines, together with component ablations.

Critical strengths
  1. The method improves both appearance and geometric reconstruction under the reported common evaluation protocol. With Waymo-only training, SOM4D improves full-image and dynamic-region novel-view PSNR over the reproduced SLARM baseline by 0.49 dB and 0.70 dB, respectively, while reducing full-image depth RMSE from 4.95 to 4.11. These results demonstrate practical benefits for reconstruction quality.
  2. Several proposed components have meaningful ablation support. Under the common 50k training schedule, removing PTRA reduces full-image and dynamic-region novel-view PSNR by 0.67 dB and 0.97 dB, respectively. Removing the full corrective component reduces both metrics by 0.47 dB. These results support the effectiveness of relational adaptation and prediction correction within the proposed system.
Critical weaknesses
  1. The necessity and distinct contribution of some central design choices are not sufficiently established. SOM4D shares major representation choices with SLARM, including pixel-aligned Gaussian generation and cubic motion modeling. Its additional contributions therefore lie primarily in relational adaptation, hierarchical aggregation, and prediction correction. Although the ablations support several components, the evidence for some emphasized designs is weaker. For example, removing shared context reduces full-image and dynamic-region novel-view PSNR by only 0.01 dB and 0.10 dB, respectively, with almost unchanged depth error. The authors acknowledge that most of HOC’s benefit comes from the dense feature hierarchy, but the distinctive role of shared context remains unclear. Furthermore, the ablations remove pathway-specific losses together with the corresponding modules. They consequently establish the contribution of each pathway together with its associated supervision, while providing less evidence about the necessity of the particular architectural mechanism.
  2. The comparisons do not yet sufficiently establish the sources of the performance gains or their trade-off with computational cost. Table 13 shows that, despite having fewer trainable parameters, SOM4D has substantially more total parameters. Under the same profiling conditions, mean latency increases from 110.8 ms for SLARM to 226.6 ms, and peak memory increases from 4.65 GiB to 8.08 GiB. The paper transparently reports these costs. However, the larger pretrained backbone and the additional architectural components change together, making their respective contributions difficult to distinguish in the overall comparison. The attention mode, semantic supervision, initialization, and training budget used for the reproduced SLARM baseline also need clarification. These details matter for assessing the magnitude of the methodological contribution and the strength of the comparison.

Overall, I recognize the reported reconstruction improvements and the evidence supporting several components. However, I remain unconvinced about the significance of the incremental contribution, the necessity of some proposed designs, and the completeness of the baseline comparison. My current recommendation is weak rejection. I am open to revising my assessment and increasing my score if the authors clarify these issues and provide convincing explanations or supporting evidence.

Questions for authors
  1. Given that SOM4D and SLARM share major Gaussian representation and motion-modeling choices, what specific limitations of existing methods are addressed by PTRA, HOC, and prediction correction, respectively? How does the existing evidence support these particular designs? If available, results using the same Omega backbone with simpler aggregation or prediction heads would help distinguish the benefits of the backbone from those of the additional architecture. This is primarily a request to clarify the contribution and provide any existing supporting evidence.
  2. Removing shared context reduces full-image PSNR by only 0.01 dB. Under which scenes, metrics, or observation conditions does this component provide its main benefit? Please also specify which pathway-specific losses are removed in each ablation and clarify the extent to which the results distinguish architectural contributions from those of the associated training signals.
  3. Does the reproduced SLARM use full attention or window-based causal attention? Is semantic distillation retained? Please specify its initialization, training budget, and evaluation configuration. Since the reproduced results differ from those reported in the original paper, please explain the relevant configuration or protocol differences so that comparability can be assessed.
  4. The appendix describes a separately predicted and refined dynamic score. How does this score enter the final motion or rendering computation? Is it used to explicitly distinguish static and dynamic Gaussians? What supervises this score, and what signals are used for the auxiliary supervision associated with dynamic regions?
9 comments

Response to W1 & W2 & Q1: Contributions beyond SLARM and Omega.

Thank you for this question. Our contribution is organizing sparse multi-view temporal observations by feature adaptation, hierarchical integration and explicit scene correction—not new Gaussian or motion models.

PTRA adapts pretrained features in camera/time groups; HOC fuses levels while retaining dense detail for Gaussian decoding; CGH corrects residual errors via geometry and source-render feedback.

Table 3 supports all three paths; feedback removal alone costs 0.40 dB non-sky NVS. Revised in Sec. 4.4/Table 3; controls added in Appendix E.2:

VariantNon-sky NVSNon-sky RECDynamic NVSDynamic REC
Frozen Omega + native DPT/basic GS head25.41 (−1.45)27.24 (−1.65)21.11 (−1.40)24.00 (−1.33)
w/o PTRA + feature stream26.19 (−0.67)28.20 (−0.69)21.54 (−0.97)24.69 (−0.64)
w/o HOC26.69 (−0.17)28.64 (−0.25)22.29 (−0.22)25.11 (−0.22)
w/o CGH26.39 (−0.47)28.13 (−0.76)22.04 (−0.47)24.57 (−0.76)
w/o source-render feedback26.46 (−0.40)28.22 (−0.67)22.24 (−0.27)24.76 (−0.57)
Full26.8628.8922.5125.33

From the table, Full gains 1.45/1.65 dB NVS/REC over the same-Omega native DPT+basic GS/MS3 baseline; dense HOC vs native DPT gains 0.173 dB NVS with context/subset off in both; CGH vs basic head gains 0.438/0.742 dB NVS/REC with PointWorld auxiliary off in both.

Response to W1 & Q2: Shared context and supervision attribution.

Thank you. HOC mainly benefits from dense multi-level consolidation (Sec. 4.4); shared context adds complementary cross-observation aggregation. The submitted Table 3 reports joint context/subset gains of +0.01/+0.10 dB non-sky/dynamic NVS.

To isolate the contribution of shared context, we additionally compare models with subset consistency disabled in both, using the same 50k Waymo training schedule and fixed 1,817-clip evaluation. Adding shared context improves static NVS by +0.022 dB (95% CI [+0.010, +0.033]), with +0.007 dB non-sky NVS and −0.061 dB dynamic NVS. The observed positive effect is primarily in static regions.

Common reconstruction losses are unchanged; pathway-specific changes follow.

AblationSupervision removed relative to Full SOM4D
w/o PTRANone
w/o Camera/Time EmbeddingsNone
w/o HOCSubset Consistency Loss
w/o Shared ContextSubset Consistency Loss
w/o Subset ConsistencySubset Consistency Loss
w/o CGHAuxiliary supervision Loss
w/o Source-render FeedbackNone

To separate architecture from supervision: With context and subset consistency disabled in both models, Dense HOC improves non-sky/dynamic NVS by +0.173/+0.226 dB over Native DPT. With the PointWorld auxiliary loss disabled in both models, CGH improves NVS/REC by +0.438/+0.742 dB over a Basic Gaussian Head. Enabling only this auxiliary loss with the complete architecture fixed improves NVS/REC by +0.018/+0.023 dB.

Response to W2 & Q3: SLARM reproduction and comparability.

Thank you for this question. Since official SLARM weights are unavailable, we retrained Waymo-only and Mix3 variants using SOM4D-matched data/inputs, 100k updates, BS32 and BF16 for controlled comparisons. We use offline full attention (not windowed/causal), fresh initialization, no semantic distillation, 5k warmup followed by constant 4e−4 LR. We use the same evaluation protocol as SOM4D. SLARM-F uses full attention, but 200k updates/BS64, semantic alignment and a 3k classification stage. Thus, the published 27.49 dB and our 26.67 dB reflect different training setups. Our SLARM result supports matched-budget method/Mix3 comparisons. (Revised in Appendix C.3.)


Response to Q4: Dynamic score and supervision.

Thank you for this question. Gaussian displacement is computed from the predicted cubic trajectory coefficients. The dynamic score is not used as a motion gate or thresholded to partition Gaussians into static and dynamic sets. Instead, it is composited as a continuous attribute during rendering, contributes to cross-view preservation weighting, and is passed as part of the Gaussian state to source-render feedback correction.

The source-view dynamic logits are supervised by aligned source dynamic masks using binary cross-entropy. The separate auxiliary supervision consists of observation-feature reconstruction and distillation of the pre-correction dynamic probability and depth confidence. (Revised in Appendices B.2 and C.1.)

Comments on the paper

3 comments

General Response (1/2) — Shared Concerns and Evidence

Dear Area Chair and Reviewers,

We thank the reviewers for their careful assessment and constructive suggestions. We appreciate their recognition of the practical sparse-observation setting, reconstruction improvements, and existing ablations. Below we consolidate the shared concerns and new evidence. The reviewer-specific responses provide full configurations and explanations.

Here, W denotes the ordered concern under Critical Weaknesses and Q denotes a Question for Authors. For vXW2, the Mix3 concern is W4, although no separate Q4 was listed.

GR1. Relational adaptation and contributions beyond the backbone (cb7s W1/W2/Q1; vXW2 W1/Q1; zVEX W1/Q1). SOM4D addresses sparse camera–time observations through PTRA relational adaptation, HOC dense consolidation, and CGH source-based correction. PTRA inserts residual adapters at four stages of frozen Omega and differs from 4D-VGGT/DynamicVGGT in the organization of camera/time patch interactions and the frozen-backbone adaptation setting (Secs. 2.1, 3.3).

The submitted pathway ablations and the requested controls give the following Waymo NVS PSNR (dB):

ConfigurationNon-sky NVSDynamic NVS
Full SOM4D26.8622.51
w/o PTRA and feature stream26.1921.54
w/o HOC26.6922.29
w/o CGH26.3922.04
View-only PTRA26.8022.41
Time-only PTRA26.8322.45
Without Relative Bias26.8222.47
Same frozen Omega + Native DPT + Basic GS Head25.4121.11

The complete PTRA pathway contributes +0.67/+0.97 dB non-sky/dynamic NVS over its removal. With attention retained, removing only learned relative biases changes both metrics by −0.04 dB. The simpler same-Omega system reaches 25.41/21.11 dB versus Full at 26.86/22.51 dB, supporting measurable adaptation/reconstruction gains beyond the pretrained backbone. These comparisons distinguish the complete pathway from a single bias term or the backbone alone (Table 3; Appendix E.1–E.2, Tables 15–16).

GR2. Dense hierarchy, shared context, and supervision attribution (cb7s W1/Q2; zVEX Q3). Our HOC contribution is primarily its dense multi-level consolidation, with shared context providing complementary cross-observation aggregation (Sec. 4.4). To separate pathways from their associated auxiliary losses, we provide matched controlled comparisons:

Controlled comparisonMetricGain (dB)
Dense HOC vs. Native DPT (Context/Subset off in both)Non-sky NVS+0.173
Dense HOC vs. Native DPT (Context/Subset off in both)Dynamic NVS+0.226
CGH vs. Basic Head (PointWorld auxiliary off in both)Non-sky NVS+0.438
CGH vs. Basic Head (PointWorld auxiliary off in both)Non-sky REC+0.742

For context itself, disabling Subset Consistency in both variants yields +0.022 dB static, +0.007 dB non-sky, and −0.061 dB dynamic NVS when context is added. This is separate from the submitted joint context/subset ablation (+0.01/+0.10 dB non-sky/dynamic). Context adds 1.09% model FLOPs and approximately 0.04 GiB peak GPU memory under the measured protocol. The complete loss-to-ablation mapping is in Appendix C.1; the controls and profiling are in Appendix E.2/E.4.

GR3. Observation geometry, error prediction, and heterogeneous Mix3 (vXW2 W3/Q3, W4). Across 203,808 target-view evaluations from 502 scenes, and after controlling for dataset, camera, and dataset-specific nonlinear source distance, +10 percentage points of geometric support corresponds to +0.765 dB non-sky PSNR (95% scene-bootstrap CI [+0.652,+0.888]). Adding support lowers five-fold scene-held-out PSNR-prediction RMSE from 3.115 to 3.062 dB. PTRA and dense HOC each improve non-sky NVS across all 12 distance/support bins (Appendix E.3, Table 17).

With fixed equal-probability Mix3, we hold architecture, supervision, total updates, and evaluation fixed within each posed method. Despite two-thirds fewer Waymo training samples, Mix3 improves Waymo non-sky NVS for SOM4D (27.16→27.32 dB), STORM (25.91→26.12 dB), and SLARM (26.67→26.97 dB) (Table 1). Appendix Tables 9–10 separately report Waymo-only zero-shot transfer to nuScenes/Argoverse 2 and Mix3 evaluations of those three training domains.

General Response (2/2) — Reviewer-Specific Concerns and Revisions

GR4. Baseline comparability and quality–resource trade-off (cb7s W2/Q3; vXW2 W2/Q2; zVEX Q2). Our SLARM rerun uses offline full attention, fresh initialization, and no semantic distillation. Because official SLARM weights are unavailable, we retrained Waymo-only and Mix3 baselines with matched data/inputs, 100k updates, BS32, BF16, and evaluation. Published SLARM-F uses 200k updates/BS64, semantic alignment, and an additional classification stage. Its 27.49 dB and our 26.67 dB therefore correspond to different training configurations (Appendix C.3). DGGT addresses reconstruction from unposed images, whereas SOM4D reconstructs from posed, calibrated observations (Sec. 2.2).

Appendix D.2, Table 13 reports mean forward latency and peak GPU memory for STORM: 127.0 ms/3.23 GiB; SLARM: 110.8 ms/4.65 GiB; SOM4D: 226.6 ms/8.08 GiB. Against the matched Waymo-only SLARM rerun, SOM4D improves non-sky/dynamic NVS by +0.49/+0.70 dB, source-view REC PSNR by +1.10 dB, and non-sky depth RMSE from 4.95 to 4.11 m (Table 1), quantifying the reconstruction-quality/resource trade-off.

Concerns of each reviewer and our responses

Reviewer cb7s

  • W1/Q1 — Individual pathway contributions: Table 3 documents PTRA/HOC/CGH and feedback ablations; the same-Omega simple baseline and the HOC/CGH controls strengthen pathway attribution (GR1–GR2; Sec. 4.4, Appendix E.2).
  • W1/Q2 — Shared context, conditions, and losses: context-only metrics identify a small static-region gain; the full ablation-to-loss mapping and supervision-controlled HOC/CGH results separate the relevant effects (GR2; Appendix C.1/E.2).
  • W2/Q3 — SLARM reproduction: attention mode, initialization, semantic supervision, training budget, and the difference from published SLARM-F are explained (GR4; Appendix C.3).
  • Q4 — Dynamic score: the score is a continuously rendered state used in preservation weighting and source-render feedback, without gating cubic motion; source dynamic masks and prediction-derived auxiliary targets provide supervision (Appendices B.2/C.1).

Reviewer vXW2

  • W1/Q1 — PTRA versus 4D-VGGT/DynamicVGGT: the distinct frozen-backbone camera/time patch adaptation is explained together with view-only, time-only, and no-bias controls (GR1; Sec. 2.1, Appendix E.1/Table 15).
  • W2/Q2 — DGGT and SLARM: posed versus unposed input assumptions and the matched-budget SLARM rerun are clarified (GR4; Sec. 2.2, Appendix C.3).
  • W3/Q3 — Data-centric analysis: support predicts PSNR beyond dataset and target distance, while 12-bin PTRA/HOC analyses connect reconstruction gains to observation conditions (GR3; Appendix E.3/Table 17).
  • W4 — Mix3 scope: the fixed, budget-matched mixture is evaluated in three posed reconstruction methods; Waymo-only zero-shot transfer and Mix3 domain evaluations are reported separately (GR3; Table 1, Appendix D.1/Tables 9–10).

Reviewer zVEX

  • W1/Q1 — Relations, attention capacity, and Omega: the bias-only ablation keeps attention layers fixed, and a simpler same-Omega system quantifies gains beyond the backbone (GR1; Sec. 4.4, Appendix E.1–E.2).
  • Q2 — Inference time and memory: the already reported three-method profiling is summarized alongside reconstruction-quality improvements (GR4; Appendix D.2/Table 13).
  • Q3 — Shared-context cost and benefit: context-only NVS metrics are distinguished from the submitted joint context/subset result; FLOPs and memory overhead are reported (GR2; Appendix E.2/E.4).

Revisions in the updated manuscript

  • Sec. 2.1 and Sec. 4.4/Table 3: clearer distinctions from related temporal/view modeling and the same-Omega basic reconstruction baseline.
  • Appendix B.2 and C.1–C.3: dynamic-score usage, pathway-specific supervision, and reproducible SLARM retraining details.
  • Appendix E.1–E.4 (Tables 15–17): view/time/bias controls, supervision-controlled ablations, support/error and module analyses, and context resource profiling.
  • Appendix D.1–D.2 (Tables 9–10, 13): zero-shot versus Mix3 cross-dataset results and existing inference-resource comparisons.

We thank the Area Chair and reviewers for their consideration. Further details are provided in our individual responses and the updated manuscript.

Area_Chair_k9UjOpenReview ↗

Official Comment by Area Chair

Dear Reviewers,

The authors have submitted their rebuttal to address your concerns. As the discussion deadline approaches, please review their response and share any remaining concerns or questions soon so the authors have enough time to respond.

Thank you, -AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.