acceptodds
Under review as a conference paper at ICLR 2027

GG4D: Geometry-Guided Dynamic Novel-View Synthesis

Abstract

Dynamic novel-view synthesis lets us freely explore a captured moving scene from any viewpoint, enabling free-viewpoint video and immersive VR and AR. To this end, feedforward methods synthesize novel views in a single forward pass without per-scene optimization. Among them, one line predicts explicit 3D representations such as 3D Gaussians, while another directly predicts target views from input images using a learned implicit renderer. Yet explicit methods are limited in fidelity by geometry errors and their fixed rendering process, and implicit renderers without shared 3D geometry struggle to maintain multi-view consistency. In this paper, we present GG4D, a feedforward framework for dynamic novel-view synthesis that combines explicit scene geometry with implicit image synthesis to achieve both multi-view consistency and high rendering fidelity. First, to aggregate geometry across frames without mixing dynamic regions from different timesteps, we propose an efficient segmentation module that separates static and dynamic regions without off-the-shelf optical flow or segmentation models. Using this separation, we construct a 4D point cloud for explicit 3D representation that guides novel-view synthesis. Second, to enable high-fidelity rendering in real time, we propose Large Chunk Attentive Test-Time Training (LaCAT), which effectively encodes input observations into a compact scene memory through attention. Our implicit renderer, built upon it, summarizes the input video once and reuses this information to synthesize high-quality novel views in real time. Extensive experiments demonstrate that \ours achieves state-of-the-art quality and multi-view consistency on both monocular and multi-view dynamic scene benchmarks, taking a step toward immersive applications.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
novel-view synthesis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LtaDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduced GG4D that addresses dynamic novel-view synthesis by combining explicit geometry with an implicit renderer. It separates static and dynamic regions using clustered features from a frozen geometric backbone, constructs a time-indexed point cloud, and introduces LaCAT to encode input observations into reusable scene memory. Experiments shows that the proposed method improved image quality and multi-view consistency on multiple benchmarks, alongside ~58 FPS rendering at 512 resolution after scene construction.

Critical strengths
  • Strong empirical performance across multiple settings. GG4D improves over the strongest feedforward baseline by approximately 3.6 dB on DyCheck and 2.6 dB on Neural3D, while achieving the lowest MEt3R across all four benchmarks. Additional camera-pose and resolution evaluations strengthen confidence in the reported comparisons.

  • The paper proposed an effective integration with practical rendering efficiency. Cluster-based segmentation improves over token-level classification, and its downstream rendering quality approaches that obtained with ground-truth masks on the evaluated NVIDIA subset. Separating scene construction from rendering also enables interactive rendering after preprocessing.

Critical weaknesses
  • I think the segmentation contribution is insufficiently differentiated from closely related work. Motion segmentation using geometric foundation-model features is already explored by VGGT4D and GeoMotion,. However, Easi3R is particularly relevant: it already uses encoder-feature clustering and cluster-level aggregation of dynamic scores to improve motion segmentation. DAS3R also trains a dynamic-mask prediction head on frozen geometric-model features. Neither is discussed in the submission. GG4D’s supervised cluster classifier and refinement network differ from these approaches, but the conceptual and empirical advantages need clearer justification. Static–dynamic decomposition itself is also established in previous work such as D²NeRF, which limits the novelty of this paper.

  • The specific advantage of LaCAT’s memory-writing design is not fully established. Reusable fast-weight scene representations have already appeared in LaCT, tttLRM, and FSM. LaCAT’s contribution is therefore the query-addressed, attention-augmented regression target. The current ablations do not compare this design against cross-view attention followed by a standard LaCT update, which could also perform attention only during scene construction. Moreover, cross-view attention improves over intra-view attention by only 0.07 dB on NVIDIA, and the ablations use a smaller renderer. These limitations leave the necessity of the proposed target construction uncertain.

  • There are wierd citations in Reference, such as line 540: FIXME Feng et al. In-place test-time training. In International Conference on Learning Representations (ICLR), 2026. arXiv:2604.06169, and line 638: FIXME Ouyang, FIXME Cai, and FIXME Hu. Fixme title. arXiv preprint arXiv:2606.21803, 2026.

Questions for authors
  • How does the proposed cluster classifier improve upon Easi3R’s clustering-based aggregation of motion evidence? Can a controlled comparison using the same backbone features and evaluation protocol isolate the benefit of the learned cluster readout?
  • Under matched training, model capacity, and scene-construction compute, how does LaCAT compare with cross-view attention followed by a standard LaCT update? Does the advantage persist with the full renderer on dynamic benchmarks?
1 comment

Thank you very much for the detailed response and the additional experiments. After reading authors’ responses and the other reviewers’ comments, Most of my concerns are solved. However my remaining concerns is still the methodological contribution. I see GG4D as a well-executed integration of prior ideas, with useful refinements in dynamic region segmentation and fast-weight memory writing. The new results demonstrate the value of these refinements, but I still think the overall contribution over prior work relatively modest. So I lean towards weak rejection.

Reviewer_d2XkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper propose GG4D, a framework for dynamic novel-view synthesis from unposed monocular or multi-view videos. The framework uses a frozen VGGT-Ω model to estimate camera parameters and depth, and separates static and dynamic regions to construct a 4D point cloud.

Alongside this explicit geometry, input observations are encoded into a reusable fast-weight scene memory using LaCAT, an extension of LaCT. During rendering, the projected point cloud provides geometric guidance, while the neural renderer reads the scene memory to synthesize the target image. Experiments report improved rendering quality and multi-view consistency across monocular and multi-view benchmarks, with real-time rendering after scene construction.

Critical strengths

Strong empirical performance GG4D achieves consistent improvements in rendering quality and multi-view consistency across the evaluated monocular and multi-view benchmarks. The gains over the strongest reported feedforward baselines are substantial, including approximately 3.6 dB mPSNR on DyCheck and 2.6 dB PSNR on Neural3D.

Efficient rendering after scene construction Reusing the constructed point cloud and fast-weight scene memory avoids repeatedly processing all input frames for each target view. The system achieves approximately 58 FPS at 512 resolution on an H200 GPU after scene construction, supporting interactive viewpoint changes.

Critical weaknesses

Complex multi-stage pipeline with limited analysis of interactions between components The framework combines geometry estimation, semantic clustering, cluster classification, mask refinement, point-cloud construction, and fast-weight memory updates. Several components are trained separately and connected through discrete processing steps, increasing implementation and training complexity. Although the component-level ablations support individual design choices, the sensitivity of the complete pipeline to intermediate decisions and errors remains less clear.

Strong dependence on a specific external geometry model GG4D relies on the frozen VGGT-Ω model for both depth/camera estimation and the features used for dynamic segmentation. Consequently, the geometry and segmentation branches both depend on the quality of this backbone. Although the authors acknowledge that geometry errors degrade rendering, robustness to such errors and portability to other geometry models are not sufficiently characterized.

Questions for authors

How sensitive is the final rendering quality to the clustering granularity and dynamic-mask thresholds? In particular, does training the refinement network with ground-truth-derived coarse masks affect its robustness to predicted masks at inference?

How do segmentation and rendering quality change under controlled depth or camera-pose perturbations? Could the framework retain its benefits with another geometry backbone, and which components would require retraining?

Reviewer_SuCJOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents GG4D, which is a feedforward approach for Novel View Synthesis (NVS) from a dynamic monocular or multi-view input. The approach first constructs a 4D point cloud from the input video. To this end, the authors propose to utilise features extracted from a geometric foundation model, followed by semantic clustering and motion (static/dynamic) classification. The authors built the scene representation on top of Large Chunk Test-Time Training (LaCT) and propose their Large Chunk Attentive Test-Time Training (LaCAT), which allows input views to update the memory state (implicit scene representation). A similar mechanism is also used to synthesise the output frames. The experimental sections provide results on both monocular and multi-view datasets, where GG4D achieves better quantitative results than compared works.

Critical strengths

I find the concept of the approach presented in the paper convincing. Namely, building the model utilising memory state as a proxy for implicit scene representation is sound. The authors adaptation of the existing LaCT to incorporate information from all input images seems to be effective by the results. Further, the use of a strong prior to build the segmentation module seems to make the training more manageable in terms of time. The results shown in the paper are very convincing in terms of quantitative tables. Additionally, the appendix experiments and ablations are extensive and meaningful.

Critical weaknesses

I wonder what really makes the approach perform that well and how much of this can be attributed to the segmentation unit. Whilst Table 4 shows 'Ours' vs GT mask, it would be great to see masks from Table 2 used in conjunction with the setup to see the sensitivity to the quality of the masks. Similarly, one could also test a weaker pose/depth prior for the purpose of evaluating the model. For datasets with GT camera poses, it would also be interesting to see GT poses used with a method such as MVSAnywhere [1] for depth prediction. That could either boost the score under a known-pose scenario or degrade it due to bad GT poses. In fact, I believe that Table 1 should show methods with GT poses where applicable. DyCheck poses are not perfect, and if VGGT-Ω poses are better, the global alignment of the trajectory causes misalignment of test views. This makes it harder to disambiguate the effect of misalignment and the underlying 3D representation on the rendering performance.

One minor weakness is the lack of any video results; while never required, it is typically hard to judge effects such as flickering without video samples. In this case, it would be particularly interesting to gauge the consistency of the produced output (given the lack of explicit underlying 3D representation).

[1] S. Izquierdo, M. Sayed, M. Firman, G. Garcia-Hernando, D. Turmukhambetov, J. Civera, O. M. Aodha, G. Brostow, J. Watson, "MVSAnywhere: Zero-Shot Multi-View Stereo", CVPR, 2025.

Questions for authors
  • The appendix lists a relatively small number of datasets used for training. Do the authors have any intuition about what enables this rather scarce training setup to perform well? Would that be a strong VGGT-Ω prior?
  • Figure 2 suggests that the approach struggles to express detail; e.g. on the DyCheck sample, the face is missing, so is the hand, and the T-shirt texture seems a bit washed out. Does it correlate with the prior quality, or maybe the network is not expressive enough for the harder samples?
  • How would the method handle long sequences?
  • Have the authors observed any issues in static parts of the scene (e.g. doubling/ghosting) caused by accumulating a static point cloud via the union of all timesteps?
  • Not a question, but some bibliography entries are missing (left as 'FIXME')

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.