acceptodds
Withdrawn from ICLR 2027

RoGe: Novel View Synthesis via End-to-End Implicit Reconstruction and Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Novel view synthesis from sparse inputs requires both geometric grounding from the observed views and generative priors of unobserved regions, motivating recent hybrid methods that combine reconstruction and generation. However, existing methods bridge the two with rendered images or explicit 3D representations such as point maps or 3D Gaussians. Generation is thus conditioned on a lossy and imperfect projection of the scene, inheriting its errors, and reconstruction receives no signal from generation to correct them. We present RoGe, an end-to-end unified reconstruction and generation framework that removes this explicit bridge. It targets roaming within a scene anchored by sparse views: given a few posed images and a camera trajectory, it synthesizes a temporally coherent video along that trajectory. From the sparse input views, RoGe builds an implicit scene representation with a feed-forward reconstruction model, and queries it with camera rays to obtain per-view geometric features. These features are injected into a video diffusion model as conditioning, without any explicit 3D intermediate. Both modules are trained jointly, so the generation objective directly shapes its own geometric conditioning. We conduct extensive experiments, where RoGe outperforms reconstruction-based, generation-based, and hybrid baselines in terms of image-level quality and video-level temporal and geometric consistency. Ablations confirm that ray-queried implicit features outperform both raw reconstruction tokens and rendered RGB as conditioning, and that joint training brings further gains.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Novel View Synthesis, Feed-Forward Reconstruction, Video Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wKmVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RoGe, a unified reconstruction-generation framework for novel view synthesis (NVS) from sparse posed images. The method builds an implicit scene representation using a camera-conditioned VGGT, and queries its scene tokens with target camera rays to obtain per-view geometric features. These features are injected into a pretrained video diffusion model, together with context image latents and Plücker ray maps. The reconstruction and generation modules are jointly optimized with flow matching and photometric losses, without introducing an explicit 3D intermediate representation.

Experiments on RE10K, DL3DV, and ACID demonstrate improvements over reconstruction-based, generation-based, and hybrid NVS baselines, particularly in image quality and video-level fidelity. Additional experiments evaluate trajectory extrapolation on ScanNet, while ablations investigate the effectiveness of ray-queried geometric features and joint training.

Critical strengths
  1. Effective use of implicit geometric features for video generation.

RoGe directly conditions video generation on target-view-aligned geometric features, avoiding explicit 3D reconstruction or rendered RGB intermediates. The ablations in Table 5 demonstrate that ray-queried geometric features provide more effective conditioning than raw VGGT tokens or rendered/predicted RGB maps. This supports the effectiveness of the proposed integration.

  1. Good empirical performance.

RoGe achieves consistent improvements in image-level metrics over strong reconstruction and generation baselines across three datasets, including the unseen ACID dataset. The video-level evaluations also show competitive geometric consistency and camera controllability. These results demonstrate the practical effectiveness of the proposed framework.

Critical weaknesses
  1. Limited methodological novelty and insufficient differentiation from prior work.

The proposed framework primarily integrates existing reconstruction and generation techniques. In particular, LagerNVS already extracts target-view-aligned features from implicit VGGT scene tokens through ray–scene attention, while related work such as Gen3R (CVPR 2026) has explored tight integration between reconstruction and diffusion. The main contribution of RoGe appears to be adapting these techniques to condition a pretrained video diffusion model, rather than introducing a substantially new reconstruction-generation mechanism. Moreover, the closely related Gen3R is not discussed, making it difficult to establish the novelty and significance of the proposed integration.

  1. Insufficient evidence for the significance of the proposed integration.

Although RoGe achieves strong quantitative results, the ablations in Table 5 show that end-to-end joint training provides only a modest improvement (20.07 to 20.40 PSNR), while subsequent higher-resolution fine-tuning contributes a larger gain. This raises questions about the relative importance of joint optimization compared with the strong pretrained reconstruction and generation backbones. Overall, the experiments demonstrate improved performance but do not sufficiently establish a substantial methodological advance over existing reconstruction-generation approaches.

Questions for authors

See weaknesses.

Reviewer_5bSCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RoGe, a framework for novel-view video synthesis from sparse posed images along specified camera trajectories. RoGe uses a camera-conditioned VGGT backbone to encode the input views into an implicit scene representation. Camera-ray tokens query this representation through cross-attention to obtain view-specific geometric features, which are adapted to condition a pretrained video diffusion model without constructing an explicit intermediate 3D representation. The central contribution is the joint adaptation of the reconstruction and generation modules using a flow-matching objective and an auxiliary photometric reconstruction loss, allowing the generative objective to shape the geometric conditioning.

Critical strengths
  • The experimental results show strong synthesis quality with competitive geometric consistency and camera control. The results indicate practical value for sparse-view trajectory synthesis across complementary measures of appearance, video quality, and spatial control.
  • The method proposes an effective integration of implicit reconstruction features with video generation. RoGe adapts LagerNVS’s ray-queried scene features into spatially and temporally aligned conditions for a pretrained video generator and enables joint adaptation of the two modules. The results substantiate a useful design for connecting reconstruction and generation without an explicit intermediate 3D representation
Critical weaknesses

The central benefit attributed to generative feedback is not isolated experimentally.

  1. Table 5, rows #6–#7, improves PSNR from 20.07 to 20.40, but unfreezing the reconstruction branch simultaneously enables updates from the flow-matching and photometric objectives across VGGT, the ray encoder, and the ray–scene attention blocks. While the architecture permits generative feedback, this comparison does not establish whether that feedback, additional supervised reconstruction adaptation, or their combination explains the gain. A key control would retain photometric updates while stopping only the flow-matching gradient into reconstruction; selectively freezing VGGT would further clarify which components require adaptation.
  2. The claim that the photometric loss prevents forgetting is also unsupported by a loss ablation or before/after reconstruction measurements.
  3. Finally, the joint-versus-frozen comparison is performed only at low resolution: row #8 receives additional high-resolution refinement with the reconstruction branch frozen, but no equally refined counterpart of row #6 is reported.

The conditioning ablation does not establish that avoiding explicit 3D is responsible for the improvement.

Table 5 compares high-dimensional implicit features (#6) with RGB predictions from implicit reconstruction (#5) and RGB renderings from an explicit Gaussian representation (#4). The comparison between #4 and #6 changes both the scene representation and the information passed to the generator. In contrast, #5–#6 supports retaining features instead of decoding them to RGB; it does not distinguish implicit geometry from explicit geometry. The Gaussian variant also uses a different reconstruction head, introducing another potential source of performance differences. An explicit representation that renders learned features into the same video backbone, with comparable conditioning capacity and training settings, would more directly test the proposed distinction.

Limited evaluation sample

RE10K, DL3DV, and ACID are each evaluated on only ten selected sequences. Following SEVA’s RE10K/DL3DV selection helps comparability, but does not establish coverage of the datasets’ scene diversity

Questions for authors
  1. Clarification of scope of the coupling contribution. How does RoGe's claimed reconstruction–generation coupling differ from prior methods such as MVSplat360 and Ouroboros3D, which also allow diffusion gradients to update reconstruction networks?

  2. Request for evidence of contribution of generative gradients. What evidence isolates the benefit of flow-matching gradients reaching the reconstruction branch from the benefit of additional photometric adaptation? In Table 5, moving from #6 to #7 enables both. Could you provide a control that retains photometric updates but stops only the flow-matching gradient at the reconstruction output? Does the benefit of joint adaptation persist when the frozen and jointly adapted variants receive the same subsequent high-resolution refinement?

  3. Request for evidence of the role of the photometric objective. What evidence supports the claim that the auxiliary photometric loss preserves the pretrained reconstruction capability during joint training? Could you provide a comparison with and without this loss, together with held-out reconstruction measurements or RGB-head outputs before and after adaptation?

  4. Clarification and supporting evidence of reliability of the evaluation. How were the ten evaluation sequences in each of RE10K, DL3DV, and ACID selected, particularly for ACID, and how consistently do the reported improvements hold across individual scenes? Could you provide paired per-scene results or uncertainty estimates computed at the sequence level, especially for the #6–#7 ablation?

Reviewer_nexqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes RoGe, a sparse-view novel view synthesis framework that conditions a video diffusion model on implicit geometry rather than on explicit 3D intermediates such as point maps or Gaussians. A camera-conditioned VGGT, initialized from LagerNVS, encodes the posed context views into scene tokens. Ray tokens for each context and target view query these scene tokens through ray–scene cross-attention, producing per-view geometric features. These features are aligned to the latent grid and injected into Wan2.1-I2V-14B together with Plücker ray maps and the context latents. Both modules are trained jointly with LoRA, using flow matching plus an auxiliary photometric loss. RoGe reports the best image-level metrics on RE10K, DL3DV and ACID against reconstruction, generation and hybrid baselines.

Critical strengths
  1. The motivation is clear. Conditioning generation on rendered images or explicit 3D passes reconstruction errors downstream, and querying an implicit scene representation with target rays is a natural way to avoid that lossy bridge.

  2. The conditioning ablation (Table 5) is informative. Raw VGGT tokens hurt performance (#2 vs. #1), and conditioning VGGT on the input cameras repairs this by building features in the given frame (#3). Ray-queried features outperform both rendered and decoded RGB (#4–#6). Together these give useful evidence on what to inject into a video model.

Critical weaknesses
  1. The central “end-to-end” contribution is weakly supported. The architecture combines existing parts: LagerNVS’s camera-conditioned VGGT and ray–scene decoder, FrameCrafter’s per-frame context encoding, and Wan2.1. The joint training is the main new element, yet it adds only 0.33 dB PSNR (#6 → #7, Table 5), from a single run. That gain is not isolated from the effect of simply making the reconstruction branch trainable. There is no control that trains the reconstruction LoRA with the photometric loss alone or blocks the flow-matching gradient, and neither λ nor the photometric loss is ablated. The final 480×832 model freezes the reconstruction module entirely, and the resolution step contributes more (+1.10 dB, #7 → #8) than joint training. No evidence shows that generation actually “corrects” reconstruction errors, as the abstract claims.

  2. The evaluation is too small for the strength of the claims. Quantitative results use 10 test sequences each for RE10K and DL3DV, and 10 ACID sequences the authors selected themselves “with substantial viewpoint variation.” No confidence intervals or seeds are reported. All outputs are cropped and resized to 192×192, which compresses differences in fine detail. The ablations run only on the 10 DL3DV sequences. The ScanNet roaming setup is left unspecified: number of scenes, how the extrapolated trajectories were built, and context selection.

Questions for authors
  1. Can you add controls that train the reconstruction branch with only the photometric loss, or with the flow-matching gradient stopped, to isolate the effect of joint training? Can you report multiple seeds for Table 5, ablate λ, and show whether joint training changes the reconstruction branch’s own outputs?

  2. Can you evaluate on the full RE10K and DL3DV test splits, or at least substantially more sequences, with confidence intervals? How were the ACID sequences selected, and how were the ScanNet roaming trajectories and context views constructed?

  3. Manuscript seems to be rushed. Citations use \cite instead of \citep throughout, so references run into the text (e.g., “per-scene optimization Mildenhall et al. (2021); Kerbl et al. (2023)”). The photometric weight is named λ in Eq. 10 but λ_render in Sec. 4.1, and Difix3D+ is listed as “Difix3D” in Table 1. The conclusion contains “temporally and coherent videos,” and several sentences are missing spaces (“paragraphs.We”, “claims.All”). There is no appendix, so inference details such as sampling steps, guidance and w(σ) are absent. The Reproducibility Statement refers to “mathematical proofs aided by generative AI,” but the paper contains no proofs, which suggests the statements were templated rather than written for this submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.