acceptodds
Withdrawn from ICLR 2027

GeoDYN: 4D Geometry-Grounded Video Diffusion for Dynamic View Synthesis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Dynamic novel view synthesis requires both geometric consistency with the observed video and the ability to synthesize unobserved content. Existing geometry-conditioned methods render a reconstructed 4D prior into pixel-space conditions, so reconstruction errors propagate directly into the video diffusion backbone, forcing it to decide on its own what to trust. We argue that how the prior is delivered matters as much as its quality, and introduce GeoDYN, which uses a 4D Gaussian prior to transport source video features to target camera views and timesteps at multiple DiT blocks of the backbone, instead of passing rendered appearance as input. Each GeoDYN block further refines the transported features, preserving source context while suppressing unreliable geometry instead of propagating its errors. In a controlled comparison with matched 1.3B backbones, GeoDYN reduces LPIPS by 34% and FID by 48% relative to a pixel-rendering baseline on the Cam×Time benchmark. Despite using 9× fewer parameters, GeoDYN achieves target-view reconstruction quality comparable to a 14B pixel-rendering model while requiring roughly half the inference time per step.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Generation for Novel View Synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1f9cOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes GeoDYN, a framework is to use 4DGS to establish geometric correspondences and transport learned features across viewpoints and time, instead of directly conditioning video diffusion on potentially corrupted 4DGS-rendered images.

Critical strengths
  1. This paper proposes a new approach that uses input video features warped through 4DGS as conditioning for video diffusion, rather than directly using 4DGS-rendered images. The experimental results demonstrate the effectiveness of this approach.

  2. The motivation is interesting, as 4DGS rendering can introduce ghosting and noticeable reconstruction artifacts. I think this is an important problem worth investigating.

Critical weaknesses
  1. The paper lacks a controlled ablation study to demonstrate that feature warping is better than using 4DGS-rendered images as conditioning. I suggest replacing the warped features with VAE-encoded features from 4DGS renderings while keeping the same network architecture, training data, and fine-tuning settings. This would provide a more direct comparison between the two conditioning strategies.

  2. The feature warping process is not clearly explained, particularly in Figure 3. Based on my understanding of Equation (2), (F_{\mathrm{src}}^i(\tau,p)) represents features from the source frame at timestamp (\tau). If so, how does the method maintain view consistency under large viewpoint changes, where some regions are not visible in the corresponding source frame? It is unclear how the proposed pipeline handles this limitation.

  3. The results of the full model in the ablation study do not match those reported in the main comparison table. I would expect the full model to achieve the same performance in both tables. Could the authors clarify the difference in experimental settings or explain this discrepancy?

Questions for authors
  1. Diffusion model features are not necessarily 3D consistent, which is one reason why most existing methods use rendered images or point cloud rather than features as conditioning. There are any new issues for using features instead of rendering images?

  2. I wonder why the better performance in uncovered regions. From my view, the reason is the conditioning in your pipeline is less reliable (Due to the 3D inconsistency feature), causing the model to rely more on diffusion priors rather than reconstruction information. Could you explain the improvement with more details?

  3. How is the uncovered region defined in this paper? Does it mean that the region is invisible from all source views? If so, I think evaluating PSNR in these regions is questionable. It should be impossible to accurately recover the ground-truth image in completely unseen regions unless there is some information leakage.

Reviewer_iABmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GeoDYN for camera- and time-controlled novel-view synthesis from monocular videos, addressing artifacts propagated from imperfect 4D geometry priors. The method transports source-video features through 4D Gaussians, combining high-resolution feature transport, source space–time correspondence, feature and opacity refinement, and adaptive fusion into a video diffusion model. Experiments show improvements over a backbone-matched pixel-conditioning baseline, particularly in regions with incomplete geometry, while comparisons with other methods reveal trade-offs in reconstruction quality, camera accuracy, and efficiency.

Critical strengths
  • The paper addresses the practical problem of reconstruction artifacts propagating into geometry-conditioned video generation through a clearly motivated feature-transport framework.

  • Backbone-matched comparisons show improvements over the pixel-conditioning baseline, while coverage-based analysis helps identify where the method provides the largest benefits.

  • The paper is clearly organized, with a helpful pipeline illustration and detailed implementation and evaluation information in the appendix.

Critical weaknesses
  1. Insufficient comparison with a dynamic extension of GeoNVS. The core feature-lifting, rendering, and adaptive-fusion framework closely follows GeoNVS. However, the main comparison is against pixel conditioning, leaving the contribution beyond an existing feature-based approach unclear. A reasonably adapted GeoNVS-style baseline using the same 4D geometry prior and a comparable training setup would help establish whether the proposed extensions are necessary. This would require appropriate temporal adaptation, rather than directly applying the original static-scene checkpoint.

  2. The effectiveness of the dynamic-specific designs is not sufficiently isolated. The ablations do not compare 4D refinement against per-frame 3D refinement, or full space–time correspondence against spatial-only correspondence. Consequently, they cannot establish how much benefit comes specifically from temporal modeling. Adding correspondence improves reconstruction and camera metrics but worsens FVD, suggesting a trade-off that deserves further analysis. The independent benefit of opacity refinement is also small: removing it changes LPIPS from 0.389 to 0.390 and FID from 53.8 to 54.0.

  3. The performance gap to SpaceTimePilot needs a more substantive explanation. SpaceTimePilot uses the same Wan2.1-1.3B backbone scale, yet achieves better PSNR, SSIM, LPIPS, and camera accuracy across all three temporal-control categories in Table 5. Although GeoDYN performs better on several metrics in the real monocular benchmarks, the paper only briefly acknowledges its weaker temporal-control results. A clearer analysis is needed to explain when the additional geometric processing provides an advantage and why it underperforms on these ground-truth-based evaluations.

  4. Qualitative fidelity and temporal consistency remain insufficiently demonstrated. The presented examples still exhibit inaccuracies in subject pose, appearance, and scene details; several examples in Figure 6 visibly differ from the ground truth. Furthermore, the provided materials contain no playable video results, making it difficult to assess motion continuity, evolving deformations, and consistency during occlusion or retiming. The reported FVD and VBench metrics provide useful evidence, but do not fully resolve these concerns. Video comparisons and a focused analysis of failure cases would strengthen the evaluation.

  5. The pixel-versus-feature comparison does not fully isolate the conditioning representation. Although the backbone, geometry prior, training data, and training duration are matched, GeoDYN also changes the fusion architecture and uses more trainable parameters: 499.63M versus 393.5M for NeoVerse-1.3B. The results therefore demonstrate the effectiveness of the complete system, but do not establish how much improvement comes from feature transport alone. A comparison with matched fusion capacity would make the attribution more convincing.

Questions for authors
  1. Could you compare against a GeoNVS-style baseline adapted to dynamic scenes using the same 4D geometry prior and a comparable training setup?

  2. Can you isolate the benefits of 4D versus per-frame 3D refinement, and space–time versus spatial-only correspondence?

  3. What explains the consistent gap to SpaceTimePilot in Table 5, given the same backbone scale, and under which conditions is GeoDYN preferable?

  4. Are video comparisons available to assess motion continuity, subject fidelity, and consistency under occlusion and retiming?

  5. How much of the improvement over NeoVerse-1.3B comes from feature transport versus the different fusion architecture and additional trainable parameters?

Reviewer_yLhhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents GeoDYN, a geometry-grounded video diffusion framework for dynamic novel view synthesis. Instead of directly rendering reconstructed 4D Gaussians into pixel-space conditioning signals, GeoDYN transports intermediate diffusion features through the 4D Gaussian representation and integrates the resulting geometry-aligned features into the diffusion backbone. The method further introduces high-resolution feature transport, Gaussian feature and opacity refinement, and source-correspondence cues to improve geometry conditioning. Experiments on three benchmarks demonstrate improved performance over a matched pixel-space conditioning baseline and competitive results in novel-view synthesis and camera-controlled video generation.

Critical strengths
  1. The paper addresses a relevant limitation of pixel-space geometry conditioning and presents a coherent alternative based on feature transport.
  2. The matched comparison against NeoVerse using the same 1.3B backbone, reconstruction prior, and training schedule provides useful evidence for the proposed design.
  3. The evaluation covers reconstruction fidelity, perceptual quality, camera control, efficiency, and component ablations across multiple benchmarks.
Critical weaknesses
  1. The paper argues that Gaussian refinement suppresses unreliable geometry, but the current evidence is largely indirect. Controlled experiments with corrupted geometry priors would more convincingly demonstrate that the method corrects or suppresses reconstruction errors.

  2. The matched baseline may be undertrained rather than capacity-limited, which is the load-bearing confound. Every headline number rests on a self-re-implementatio. One observation suggests the 1.3B baseline may not be a fair representative of the pixel-rendering paradigm: In Fig. 2a, NeoVerse (1.3B) scores 13.49 dB on uncovered pixels, which below BulletTime's 16.48 dB, a model with no geometry prior at all. Being actively harmed relative to using no geometry is a strong signal of an unconverged or mis-tuned controller.

  3. The weighted-average correspondence representation may become ambiguous under occlusion, or conflicting Gaussian projections. The paper does not adequately evaluate the reliability of these correspondences in challenging dynamic scenes.

  4. Removing learned opacity changes nothing measurable (0.01 dB PSNR, 0.2 FID, identical SSIM) and in fact improves translation error. Meanwhile "w/o learned feature" and "w/o refinement" are identical to three decimal places on four of six metrics, i.e. once feature refinement is removed, opacity refinement contributes nothing. The table supports exactly one conclusion: feature refinement matters and opacity refinement does not.

Questions for authors
  1. Why is SpaceTimePilot excluded from Tab. 2 and Fig. 2 when Cam x Time results for it exist in Tab. 5? Please add it, including the coverage-stratified analysis. How do you reconcile a 2.00B geometry-free model outperforming GeoDYN by ~2.3 dB PSNR with the claim that geometry priors are the right way to get camera-time control at small scale?

  2. As the GS rendering is looped in the diffusion process, what's the runtime of the system? Does the GS model slow down the training process significantly?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.