acceptodds
Under review as a conference paper at ICLR 2027

ESC: Escaping 2D Diffusion Ambiguity with Geometric Constraints for Consistent Novel View Synthesis

Abstract

Generating 3D-consistent novel views from 2D diffusion models remains a fundamental challenge in computer vision. In this work, we introduce a framework that enhances geometric alignment for monocular view synthesis under drastic camera changes by jointly generating a target view and its corresponding depth map. By conditioning the diffusion pipeline on the single source image, source depth, and target camera rays, our approach forces the model to inherently reason in 3D space even under extreme viewpoint shifts. To guarantee spatial fidelity, we propose a novel per-pixel depth consistency constraint, which verifies and enforces 3D consistency across generated views. Our framework establishes a robust pipeline for geometrically consistent view synthesis, yielding superior visual accuracy and spatial alignment compared to current state-of-the-art methods. With this work, we pave the road towards robust 3D world models capable of understanding complex environments from a single view image.

Primary area:
generative models
Keywords:
novel-view synthesis, 3D vision, diffusion model, image generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ch8gOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ESC, a diffusion-based framework for single-image novel view synthesis that conditions a pretrained image-editing diffusion transformer on source depth and Plücker rays of the source and target cameras. Using LoRA adapters and a Ray ControlNet, the model jointly generates target RGB and depth. Its trainable components are optimized using flow-matching and auxiliary reconstruction, perceptual losses. The paper also introduces the PCC, a bidirectional depth-reprojection score used both for evaluation and for inference-time Best-of-N selection. Experiments on DL3DV, RealEstate10K, and Tanks & Temples report improvements in LPIPS, DINO-Sim, DreamSim, and PCC relative to the evaluated baselines.

Critical strengths
  • Strong single-sample perceptual results.\textbf{Strong single-sample perceptual results.} ESC-Base leads the compared baselines on LPIPS, DINO-Sim, and DreamSim across all three benchmarks, so the reported perceptual advantage does not depend entirely on Best-of-N selection. Results on seven Tanks & Temples scenes also provide evidence of transfer beyond the DL3DV/RealEstate10K fine-tuning mixture.
  • Useful component ablations.\textbf{Useful component ablations.} Table 6 shows that adding depth improves perceptual quality and PCC over ESC-Shadow across all three real-world benchmarks, with further gains from PCC selection. These results support the components' practical usefulness within the proposed pipeline.
  • Geometric diagnostics.\textbf{Geometric diagnostics.} Controlled synthetic experiments distinguish geometric perturbations from source-compatible changes in occluded regions. PCC increases for the former while remaining low for the latter, illustrating its potential as a complement to target-image similarity metrics. These tests support the distinction between source-compatible geometry and reproducing a particular ground-truth completion, although they do not establish universal consistency guarantees.
Critical weaknesses

Major Weaknesses

  • Insufficient comparison with related methods.\textbf{Insufficient comparison with related methods.} The paper motivates its contribution through the limitations of geometry warping followed by 2D completion, but does not adequately distinguish its approach from existing methods that already incorporate geometry into generation. In particular, MVGD (Guizilini et al., CVPR 2025) combines ray-map conditioning with novel-view image and depth generation, while SceneCompleter (Chen et al., arXiv:2506.10981) explicitly motivates joint geometry and appearance completion through the geometric limitations of 2D inpainting. Neither work is cited in the manuscript or included in the experimental comparison. Their omission leaves the novelty of RGB–depth generation and geometry-aware conditioning insufficiently established. The paper should explain what ESC contributes beyond adapting a stronger image-editing backbone and adding a geometric selection criterion.
  • Incomplete attribution of the improvements.\textbf{Incomplete attribution of the improvements.} ESC-Shadow simultaneously removes depth conditioning and depth prediction while retaining Ray ControlNet, so it does not isolate these components or establish that improved appearance–structure coupling explains the gains. The separate effects of the reconstruction, LPIPS, and DINO losses are also not ablated. Their basic visual-fidelity rationale is understandable, but the experiments do not establish their individual contributions to the reported perceptual and geometric gains. Furthermore, other stochastic NVS methods could use external depth estimation for PCC selection, but neither this alternative nor ESC-Shadow with PCC selection is evaluated. Selection uses ESC-predicted depth whereas benchmark PCC uses DA3 depth, but both rely on the same reprojection criterion. Matched-budget selection baselines and independent geometric checks would better distinguish generator-specific benefits from generic sampling-and-ranking gains.
  • Overstated consistency guarantees.\textbf{Overstated consistency guarantees.} PCC measures depth-reprojection conflicts over valid projection coverage, while the main inference procedure selects the lowest-scoring candidate without refining its RGB or geometry. Selecting the minimum does not ensure that the selected candidate is geometrically consistent, particularly when all candidates contain errors. Nor does it explicitly verify agreement between the generated RGB and depth. Although Appendix C.11 describes a multi-view extension, the main source–target selection procedure is not shown to enforce mutual consistency across independently generated target views. Moreover, log transformation and clipping reduce the influence of extreme residuals, while averaging can dilute severe localized errors. These are reasonable robustness choices, but their implications for ranking structural errors require validation. Describing PCC as strictly enforcing or guaranteeing scene consistency is therefore stronger than the demonstrated mechanism supports.
  • Mixed evidence for viewpoint control.\textbf{Mixed evidence for viewpoint control.} Table 9 shows that LagerNVS leads PSNR/SSIM across all three datasets, while competing methods lead pose AUC and translation accuracy. ESC does achieve strong rotation-error results on some benchmarks, but the overall results indicate a trade-off rather than uniformly superior viewpoint control. The perceptual and PCC improvements therefore need to be distinguished from precise compliance with the camera prompt. Blurring and pixel-alignment sensitivity do not adequately explain the weaker pose results. Figure 5 also shows substantial variation in structure, scale, and framing under identical conditioning. Without separating source-visible and newly revealed regions, it is unclear how much reflects valid completion diversity versus scene or viewpoint drift.

Minor Weaknesses

  • The depth ablation is introduced too late.\textbf{The depth ablation is introduced too late.} ESC-Shadow appears in Figure 4 before being defined, while Section 4.1 introduces only ESC-Base and ESC-PCC-BoN. Its definition and central quantitative ablation are deferred to Appendix D.2 and Table 6.
  • Figure 2 needs clearer context.\textbf{Figure 2 needs clearer context.} Although the caption defines the modifications and pixel counts, the scene objects and red error overlays are easy to confuse. The projection directions, residuals, counts, and normalized score need a clearer visual connection. An unmodified-pair reference score would make PCC = 0.0232 more interpretable.
  • Qualitative results need clearer annotations.\textbf{Qualitative results need clearer annotations.} Figures 1 and 5 should clarify camera changes and source–target overlap. Per-candidate PCC values and reprojection-error visualizations would help interpret Figure 5. The in-the-wild examples illustrate plausible generations but provide limited evidence of precise geometric consistency.
Questions for authors
  • How does ESC differ from MVGD and SceneCompleter in geometric conditioning, RGB–depth generation, and consistency verification? Can the authors provide a comparison under comparable input settings?
  • Which ablations isolate source-depth conditioning, target-depth prediction, Ray ControlNet, and the auxiliary losses? What evidence shows that improved RGB–depth agreement explains the gains, rather than the addition of depth supervision or other training changes?
  • What candidate count and total latency, including scoring, are used for each real-world benchmark? How do other stochastic baselines or ESC-Shadow perform with external depth estimation and PCC selection under the same candidate budget? Please report generation and depth-estimation costs separately.
  • How sensitive are PCC rankings to thresholding, log transformation, and clipping, especially for severe localized versus widespread minor errors? Can these rankings and consistency across multiple generated views be validated using an independent geometric reference?
  • What explains the weaker pose AUC and translation-error results? For Figure 5, can the authors provide the requested camera transformation, candidate PCC/pose errors, and a visible-versus-newly-revealed-region analysis to distinguish completion diversity from drift?
  • Is test-time source-depth conditioning computed from the source image alone or the 60-frame DA3 reconstruction? If multiple frames are used, do they include the target, and do baselines receive equivalent geometric information? Please distinguish conditioning inputs from depth used only for supervision or evaluation.
2 comments

We thank the reviewer for their thoughtful feedback.

Q1 and W1: Novelty relative to MVGD and SceneCompleter We will expand our Related Work section to include a direct comparison with MVGD and SceneCompleter. ESC differs from these works in key aspects:

  • Learned 3D-grounded generation vs. zero-shot guidance (MVGD): A fundamental difference between MVGD and our work is that MVGD is a zero-shot approach, whereas ESC explicitly fine-tunes a 2D image-editing diffusion backbone conditioned on source RGB-D and Plücker rays to jointly generate target RGB and depth. We believe that fine-tuning the model to directly learn this added depth modality is essential—by learning the joint RGB-D mapping end-to-end, ESC genuinely grounds generation in 3D scene geometry rather than relying solely on zero-shot test-time input modifications.
  • A new 3D consistency verifier (PCC): A central contribution of our work is proposing PCC—a dedicated, standalone verifier for 3D geometric consistency via bidirectional point-cloud reprojection. Because ESC intrinsically predicts target depth alongside RGB, it pairs naturally with this verifier to enforce and select for 3D consistency directly at inference time, without requiring an external depth estimator.

Q2 and W2 Attribution of components and losses [Q2, W2] To isolate the contribution of depth, our ESC-Shadow experiment (Table 6, Appendix D.2) directly ablates source-depth conditioning and target-depth prediction

By removing only the depth input and output channels, ESC-Shadow serves as a controlled RGB-only baseline that isolates the effect of joint RGB-D modeling from the backbone and training objectives. Comparing ESC-Shadow against ESC-Base across all three benchmarks (DL3DV, RealEstate10K, and Tanks and Temples) shows consistent improvements in both perceptual quality and 3D geometric consistency (PCC). This confirms that the gains are not merely a byproduct of the pretrained backbone or perceptual losses, but are directly driven by grounding the generation in source depth and jointly predicting target depth. Comparing ESC-Shadow against ESC-Base demonstrates that adding depth conditioning and joint RGB-D prediction yields consistent improvements across all benchmarks. Specifically, while the auxiliary perceptual losses primarily drive gains in 2D visual quality (as reflected in perceptual metrics), the ESC-Shadow ablation confirms that incorporating explicit depth conditioning and prediction provides additional, measurable gains in 3D geometric consistency (PCC) as well as overall synthesis quality. We will bring the definition of ESC-Shadow and Table 6 into the main text to make this ablation clear upfront.

Regarding the comment that "ESC-Shadow simultaneously removes depth conditioning and depth prediction while retaining Ray ControlNet, so it does not isolate these components...", we would like to clarify the rationale behind this design choice. Because our base model (Qwen-Edit) is a 2D image-editing model, retaining the Ray ControlNet is strictly necessary to adapt it to novel-view synthesis; without precise camera ray conditioning, the target viewpoint is unspecified and the NVS task itself becomes ill-defined. Keeping Ray ControlNet fixed in ESC-Shadow therefore provides the proper controlled baseline to specifically isolate the impact of adding depth conditioning and joint depth prediction.

Q3: Candidate Count, Latency, and Matched-Compute Selection

  • Candidate count: We use a consistent candidate count of N=20 for ESC-PCC-BoN across all three real-world benchmarks (DL3DV, RealEstate10K, and Tanks and Temples).
  • Latency breakdown: Generating a single RGB-D sample with ESC-Base takes 3 seconds (table 7), and generating N=20 candidates takes about 60 seconds. Because ESC intrinsically predicts target depth, computing the bidirectional PCC score across all 20 candidates is lightweight (no scoring overhead).
  • Matched-compute comparison & baselines: Even under a strictly matched single-sample budget (N=1), ESC-Base (w/o BoN selection) already outperforms the baselines on perceptual metrics across all benchmarks (Table 1), demonstrating that our gains do not rely solely on Best-of-N sampling. Furthermore, Table 7 shows that our model matches or exceeds the inference speed of other stochastic generative baselines.

Q4: PCC Design Choices, Sensitivity, and Validation Rationale for thresholding, clipping, and log transformation: Because PCC is fundamentally a 3D distance error metric, these design choices are necessary to ensure geometric stability rather than arbitrary tuning: Thresholding and clipping prevent distant background regions with ill-defined depth (such as the sky) from corrupting the score. Depth estimations for such regions are inherently noisy or unbounded, so clipping ensures PCC evaluates reliable, well-defined scene geometry. Log transformation maps depth distances to a logarithmic scale so that PCC does not disproportionately overweight errors on far-away pixels relative to foreground geometry. This naturally balances sensitivity between localized structural errors and widespread minor deviations.

Q5: Pose AUC and Translation-Error Results Standard camera pose estimators are trained on sharp, real-world imagery, making their behavior unpredictable when evaluated on generative artifacts—such as the severe hallucinations of GLD or the heavy blurring of LagerNVS. As a result, a blurry or hallucinated image can yield an arbitrarily strong pose score or SSIM despite lacking true structural coherence. We refer the reviewer to our dedicated pose metric vs. PCC analysis in Appendix C.12.1, which examines these limitations in detail, as well as our qualitative comparisons against LagerNVS and GLD, which show that ESC achieves noticeably better visual quality and structurally plausible content completion that standard pose metrics fail to capture.

Q6: Test-Time Source-Depth Conditioning At test time, source-depth conditioning is computed solely from the single source image—we do not use a 60-frame DA3 reconstruction as inference input, nor is any target-view information used. The statement in Appendix A.4 ("Pairs are sampled from within the 60-frame chunks") refers only to how source–target image pairs are sampled from video sequences, not to multi-frame depth conditioning at inference time. We will revise the wording in Appendix A.4 to make this distinction—and the separation between test-time conditioning inputs versus supervision/evaluation—completely clear.

Reviewer_THFuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses novel view synthesis from a single image. It adapts a large image-editing diffusion model to generate the target view together with its depth map, conditioned on the source image, the source depth and the camera rays of both views. The idea is that predicting depth alongside the image grounds the generation in 3D, especially for regions not visible in the source view.

The authors also propose "PCC", a score that checks 3D consistency by projecting the depth of each view into the other and counting the conflicts. They use it as an evaluation metric and to pick the most consistent of several generated samples at test time.

The method is evaluated on DL3DV, RealEstate10K and Tanks and Temples against several recent methods. It gets the best perceptual scores and the best PCC, especially with the sample selection, while some baselines score better on PSNR, SSIM and camera pose accuracy, which are only reported in the appendix.

Critical strengths

The first strength is that the recipe is practical and that the role of depth is ablated cleanly. Adapting a large image-editing model with camera rays and joint image and depth generation gives the best perceptual quality on all three datasets, including Tanks and Temples, which none of the methods were trained on. The ablation without depth keeps the same architecture and removes only the depth part, and adding depth improves perceptual quality and 3D consistency on every dataset. This is the clearest evidence in the paper that grounding the generation in depth actually helps.

The second strength is the effort to measure 3D consistency in an interpretable way. The proposed score separates geometry that is missing from geometry that is hallucinated, and by design it does not penalize plausible changes in regions the source view cannot see. The synthetic experiments show that it reacts to real geometric errors while ignoring these changes. The authors also report a lower bound computed with the ground-truth image. They are upfront that picking the best of several samples costs extra compute. They also show that the score from the model’s own depth agrees well with the one from an external depth estimator. Better ways to evaluate consistency in generative view synthesis are needed, so I see this as a useful contribution.

Critical weaknesses

The main claim about better 3D consistency mostly comes from the best-of-N selection, not from the model itself. With a single sample, ESC is only marginally better than the baselines on the consistency score: about 0.245 vs. 0.248 on DL3DV and 0.077 vs. 0.078 on RealEstate10K, which is tiny compared to the spread across samples. The clear improvement only appears when 20 samples are generated and the one with the best score is kept. But that selection uses the model’s own depth, which the paper shows agrees very closely with the depth used for evaluation (see reported correlation). So the method is selected on almost the same metric it is evaluated on. To the best of my understanding, the baselines don’t get the same selection, and it costs about 20 times more compute.

ESC is behind on the standard metrics which are only reported in the appendix I find this misleading, I believe these should be reported in the main paper and discussed. LagerNVS has better PSNR and SSIM on all three datasets. GLD or LagerNVS also have better camera pose accuracy everywhere, for example 0.73 vs. 0.58 on RealEstate10K and 0.50 vs. 0.33 on Tanks and Temples (AUC at 15 degrees).

The model is trained with LPIPS and DINO feature losses, and it is evaluated with LPIPS, DINO similarity and DreamSim, which build on similar features. Even the version without depth already beats every baseline on DINO similarity and DreamSim, so most of this margin seems to come from the backbone and the losses rather than from the use of geometry.

Questions for authors

My main concern is with best-of-N rsults. I would be happy to change my rating based on this concern. The others are minor points as the visual quality of the generated content is visibly better and it is clear that blurry results can influence PSNR. Only rotation errors could be discussed better.

Is my understanding correct that the best-of-N results use 20 samples, and that the selection is based on the consistency score computed from the model’s own predicted depth? If so, the evaluation score and the selection score are very closely related. Could you discuss how much the baselines would improve with the same selection?

The model is trained with LPIPS and DINO feature losses, and the perceptual evaluation uses LPIPS, DINO similarity and DreamSim. Do I understand correctly that the version without depth already beats all baselines on DINO similarity and DreamSim? If so, how much of the perceptual gain do you attribute to the training losses rather than to the depth conditioning?

Why is ESC behind GLD and LagerNVS on camera pose accuracy on all three datasets, given that the method is designed to improve geometric alignment? Also, training uses Depth Anything 3 poses and depth, while evaluation uses COLMAP poses. How are the two brought to the same scale at test time?

4 comments

We thank the reviewer for their thoughtful feedback. We are glad they found our proposed recipe practical, our ablation on the role of depth clean, and our 3D consistency score a useful contribution. We address their concerns below.

W1, Q1, and Q2: ESC is only marginally better than the baselines on the consistency score with N=1, how baseline would improve with N? We confirm that our Best-of-N selection uses N=20 samples. Applying this same Best-of-N selection to the baseline models is non-trivial and computationally prohibitive. First, some baselines are deterministic and cannot generate multiple diverse samples to select from. Second, unlike ESC, which intrinsically generates the corresponding depth maps during inference, other models would require running a separate external depth estimator (like DA3) on every single generated image before computing the PCC score. This external scoring step adds significant computational overhead. Because our approach jointly generates RGB and depth, it makes this self-contained geometric selection mechanism uniquely practical and efficient to apply.

W2 and Q4: ESC is behind on the standard metrics which are only reported in the appendix Standard camera pose estimation and pixel-wise metrics (like PSNR and SSIM) are simply not reliable measures for true 3D consistency or perceptual quality in generative novel view synthesis. Pose estimators are trained on sharp, real-world imagery, making their reaction to generative artifacts—such as the severe hallucinations of GLD or the heavy blurring of LagerNVS—unpredictable. A blurry or hallucinated image can yield an arbitrarily good pose score or SSIM, despite lacking actual structural coherence. We refer the reviewer to our qualitative examples comparing our method against LagerNVS and GLD, which clearly demonstrate that our approach yields superior visual quality and structurally plausible content completion, aspects that these standard metrics fail to capture. We will move this standard metric and pose accuracy discussion to the main text to clarify this perspective.

W3 and Q3: [gains seem] to come from the backbone and the losses rather than from the use of geometry We agree that adapting a strong pretrained image-editing backbone and utilizing LPIPS/DINO training losses establishes a strong baseline performance. However, it is crucial to note that ESC-Shadow was trained with the exact same LPIPS and DINO feature losses as our full model. Therefore, the improvements in both perceptual metrics and 3D consistency (PCC) from ESC-Shadow to ESC-Base (as shown in Table 6) can be strictly attributed to the addition of explicit depth conditioning and joint RGB-D prediction. This confirms that grounding the generation in 3D geometry provides a substantial, measurable benefit beyond what the backbone and perceptual losses provide on their own.

DA3 vs COLMAP scale alignment: During evaluation, we align the DA3-predicted scale and coordinate frame to the COLMAP evaluation frame using the alignment implementation directly from the official DA3 repository.

Dear Reviewer, the discussion deadline is approaching soon. Can you please provide feedback to the authors?

Thank you for your responses, and for clarifying the Best-of-N setup and the scale alignment. I also went over your response to Reviewer MMtG, since it covers the same concern about the Best-of-N selection. Specifically the shared metric concern which was not addressed in the responses.

Firstly, the consistency improvements appear to come mainly from the Best-of-N selection. The selection with PCC uses the same score as the evaluation metric, only computed with the model’s own depth instead of DA3. Since the model’s depth is trained on DA3 depth, it is not surprising that the two agree almost perfectly. From your response to Reviewer MMtG, I see that the selection also improves the perceptual metrics, so it does pick better samples.

Predicting depth makes this selection easy for ESC, which is good. But ESC gets 20 samples and the baselines get one, and the paper itself calls this test-time scaling and not a matched comparison. With a single sample, the consistency gain is small on DL3DV and RealEstate10K, and clear only on Tanks and Temples. I think this should be stated clearly in the main paper.

Some baselines, like VIVID and GLD, are also diffusion models. Couldn't they get the same selection using an external depth model? Is there anything stopping this except for adding one DA3 in the loop for every sample?

The other points are minor for me. It is good that the pose results will move to the main paper, but the explanation that the pose estimator is unreliable on these outputs is not shown, and on the real datasets Best-of-N does not really change pose accuracy.

My main concern about the Best-of-N results remains.

We thank the reviewer for their thoughtful follow-up and constructive feedback.

For the rebuttal, we evaluated PCC Best-of-N (BoN) selection on the Tanks and Temples (T&T) dataset for the stochastic baselines: PhotoNVS, VIVID, and GLD. The remaining baseline models are deterministic feedforward networks, for which Best-of-N sampling does not apply.

One important distinction to note is that because PhotoNVS, VIVID, and GLD do not predict target depth, their Best-of-N selection had to be performed using DA3 depth estimations—the exact same estimator used to compute the PCC evaluation metric reported in the table. In contrast, ESC performs Best-of-N selection using its own intrinsically predicted depth maps rather than the external evaluation estimator.

ModelPSNR ↑SSIM ↑LPIPS ↓DINO-Sim ↑DreamSim ↓PCC ↓AUC@15 ↑AUC@30 ↑RPE-R ↓RPE-T ↓
Tanks & Temples Dataset
PhotoNVS10.130.26920.76880.23450.59020.44670.02750.079649.7043.41
PhotoNVS-PCC-BoN10.220.26900.76070.30370.50080.40850.11930.104832.6035.52
VIVID10.380.28590.75270.37930.50400.44160.08330.231033.8931.86
VIVID-PCC-BoN10.560.28580.70220.42240.47710.40410.12280.319031.5630.36
OVIE10.580.30930.82190.06400.78810.51020.00520.022953.0153.35
GLD10.880.28910.63650.50990.40270.41840.49890.620815.9215.33
GLD-PCC-BoN11.340.29830.61150.52770.38900.36230.50930.641515.1115.04
LagerNVS13.480.46690.64200.27380.66300.49010.19370.273635.2533.84
ESC-shadow10.780.26920.63360.65490.29280.42910.16940.383722.0020.08
ESC (Ours)11.270.28820.58350.72420.23620.38260.32870.550114.6215.64
ESC-PCC-BoN (Ours)11.420.28920.55960.73390.22320.28490.33290.548214.7416.28
Oracle (GT-vs-DA3)·····0.18990.77000.83577.556.33

In Appendix C.12, we present a detailed analysis comparing PCC against perceptual and pose-based metrics; specifically, in Appendix C.12.2, we discuss why pose-based metrics can be unreliable for evaluating generative novel view synthesis. We hope this analysis helps address your concern regarding the unreliability of pose estimators. At the same time, we appreciate your suggestion and will move the pose metrics into the main text of the revised manuscript.

Thanks again.

Reviewer_MMtGOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces ESC, a geometry-aware diffusion framework for monocular novel view synthesis that aims to overcome the geometric inconsistencies often observed in 2D diffusion-based approaches. Instead of generating only RGB images, ESC jointly predicts the target RGB view and its depth map while conditioning on the source image, source depth, and source/target camera geometry represented as Plücker rays. The authors further introduce a Projection Consistency Constraint (PCC), a bidirectional depth reprojection metric designed to quantify and enforce 3D consistency between views. During inference, PCC is used in a Best-of-N sampling strategy to select the sample with the strongest geometric consistency.

Experiments show that ESC achieves substantial improvements in perceptual metrics (LPIPS, DINO-Sim, DreamSim) and generally improves the proposed PCC metric relative to several recent NVS baselines. The work argues that commonly used image-quality and pose metrics fail to assess true geometric consistency and presents PCC as both an evaluation metric and an inference-time selection criterion. The central message is that incorporating explicit geometric reasoning through joint RGB-depth generation and geometry-aware conditioning leads to more structurally coherent novel-view synthesis, especially under large viewpoint changes.

Critical strengths
  1. Novel geometric consistency metric: PCC is a meaningful contribution. Unlike standard image-based metrics, it explicitly evaluates bidirectional depth reprojection consistency and can detect hallucinated or missing geometry. The visualization and formulation are intuitive and grounded in established geometric principles.

  2. Careful discussion of metric limitations: The authors explicitly discuss the shortcomings of PSNR, SSIM, and pose-based metrics for evaluating NVS, and attempt to fill this gap with PCC. This discussion is thoughtful and highlights an important issue in the field.

Critical weaknesses
  1. The main experimental results use PCC primarily as an inference-time Best-of-N selection criterion rather than as an explicit optimization objective in Eq. (6). Although the supplementary discusses differentiable PCC formulations, it is unclear whether PCC-based training contributes to any of the reported quantitative gains. Clarifying the role of PCC during optimization versus inference would strengthen the paper.

2.Compute comparison may be unfair: ESC-PCC-BoN generates multiple diffusion samples and selects the best according to PCC. The paper acknowledges that compute scales approximately linearly with the number of samples. Consequently, comparisons with single-sample baselines are not compute-matched and may overstate practical advantages.

  1. Heavy dependence on external geometry estimators: Both training and evaluation depend strongly on DA3-generated depth and camera poses. If DA3 is inaccurate, errors may propagate into supervision, conditioning, and PCC computation. The paper does not thoroughly quantify how sensitive ESC is to DA3 errors in real-world settings.

  2. PCC validation may be somewhat circular: PCC is both proposed as a new evaluation metric and used as the selection mechanism for ESC-PCC-BoN. As a result, improvements on PCC are somewhat expected. Stronger evidence would come from independent geometry evaluations (e.g., reconstruction accuracy, multi-view consistency benchmarks, or downstream 3D tasks) that do not directly rely on PCC.

  3. Missing runtime and practicality analysis: The paper briefly discusses compute scaling but provides limited analysis of inference latency, memory usage, or practical deployment costs, especially for the Best-of-N strategy. In realistic applications, the additional sampling cost may be significant.

Questions for authors
  1. The paper introduces PCC as a geometric consistency constraint, but Eq. (6) does not include a PCC term. Appendix C.9 discusses differentiable PCC losses. Were any of the reported models trained with PCC-based supervision? If not, what prevented incorporating PCC directly into optimization, and if yes, what is the quantitative contribution of PCC training versus PCC-based inference-time selection?

  2. What value of N is used in ESC-PCC-BoN, and how does quality scale with compute? Could the authors provide a compute-quality tradeoff curve and compare against baselines given equivalent inference budgets?

  3. How sensitive is ESC to errors from DA3? Have the authors evaluated performance under deliberately perturbed depth maps or camera poses to quantify robustness to imperfect geometric priors?

4 comments

We thank the reviewer for their thoughtful feedback. We were pleased to see that the reviewer found the PCC metric a meaningful contribution and that its introduction was well motivated with the limitations of prior metrics used in the literature. The PCC metric is a strong independent contribution of the paper. Prior 2D metrics (PSNR, SSIM, LPIPS) fail to capture multi-view geometric consistency, while full 3D reconstruction pipelines are computationally expensive and introduce optimization artifacts. The PCC metric fills this gap by directly evaluating the 3D geometric alignment in pixel space (see appendix D).

We would also like to highlight a core contribution of our ESC architecture: it explicitly utilizes source RGB-D and relative camera information to jointly generate novel-view RGB and depth. By combining the predicted RGB-D with camera rays, our model effectively constructs a 3D point cloud. This is a critical capability, as it enables a 2D diffusion model to directly generate in 3D space, grounding the generative process in actual scene geometry rather than relying solely on 2D priors. We address the reviewer’s concerns below.

PCC at inference vs. training objective criteria [W1, Q1]: PCC at inference vs. training objective criteria [W1, Q1]: We clarify that our main models evaluated on real-world data were not trained with a PCC loss. Using PCC as a direct learning objective is sensitive to the quality of the 3D training data and requires accurate depth maps from both views. For real imagery, we currently rely on DA3's depth estimation, which is not always accurate enough to serve as a robust training objective (see our error analysis of DA3 using PCC in Figures 12–14). However, in the simulation experiment (Table 11), where ground truth depth from both the input and output views is available, we demonstrated that using PCC as a learning objective is possible and successfully improves generation results. To clarify the quantitative contribution of PCC training versus PCC-based inference-time selection, we refer to Table 11, which illustrates the differences between PCC-inference and PCC-training.

MetricESC-Base @100k itersESC-PCC-BoN 100k itersESC-PCC-FT @100k+30k iters
Synthesis Quality
PSNR ↑30.7331.5933.65
SSIM ↑0.95320.95780.9693
Perceptual Quality
LPIPS ↓0.05840.04600.0324
DINO-Sim ↑0.96500.96520.9746
Geometric Fidelity
PCC ↓0.12200.09150.0721
Trajectory & View Prompt Consistency
AUC@5 ↑0.44030.52830.6132
AUC@15 ↑0.70670.77580.8095
AUC@30 ↑0.80460.84620.8641
RPE-R ↓12.3411.0310.37
RPE-T ↓2.691.821.71

Furthermore, as shown in Table 1, adding PCC BoN at inference time improves overall perceptual metrics, not just the PCC score.

DatasetMetricESC-BaseESC-PCC-BoNImprovement
DL3DVLPIPS ↓0.44480.4340-0.0108
DINO-Sim ↑0.83660.8386+0.0020
DreamSim ↓0.14200.1402-0.0018
RE10KLPIPS ↓0.26130.2150-0.0463
DINO-Sim ↑0.91570.9222+0.0065
DreamSim ↓0.06690.0655-0.0014
Tanks & TemplesLPIPS ↓0.58350.5596-0.0239
DINO-Sim ↑0.72420.7339+0.0097
DreamSim ↓0.23620.2232-0.0130

Comparison fairness [W2 and Q2]: Regarding the Best-of-N selection, we used a candidate count of N=20. We address the compute fairness concern with two key insights:

  • Single-sample improvements: As demonstrated in Table [1], our base model (ESC-Base) improves upon standard novel-view synthesis perceptual metrics even without the Best-of-N selection. Thus, our method outperforms the single-sample baselines even under strictly matched compute budgets.
  • Intrinsic depth generation: The novel-view depth used to compute the PCC score for selection is intrinsically generated by our own model during inference, not provided by an external 3D estimator. Because the baseline models we compared against do not predict target depth, they cannot utilize this self-contained geometric selection mechanism.

Dependence on DA3 [W3 and Q3]: We refer the reviewer to our detailed error analysis of DA3 using PCC in Figures 12–14. In our simulation experiment (Table 11), where ground truth depth maps are available for both input and output views, we evaluate performance both with ground truth geometry and with DA3-estimated geometry to isolate the impact of estimation errors. While DA3 depth predictions can indeed contain local per-pixel noise that makes them sub-optimal as a direct pixel-level training loss, the reprojected PCC score computed via DA3 acts effectively as a robust global verifier and selection mechanism for the overall generated target view.

Validation strength [W4]: Furthermore, our results showing that using PCC for Best-of-N selection at inference time improves quantitative metrics serve as a strong proof of concept, demonstrating that PCC provides a valid reward signal and indicating its potential to be used for Direct Preference Optimization (DPO) in future work.

We present extensive quantitative and qualitative evidence demonstrating that PCC is a valid, robust evaluation metric. While using PCC for Best-of-N (BoN) selection at inference time naturally elevates PCC scores, our ESC modeling approach achieves clear performance gains even with single-sample generation. Furthermore, Figures 5, 20, 21, and 22 visually illustrate the incremental improvements provided by BoN. We observe no evidence of metric overfitting, confirming that the evaluation is not circular. When provided with precise depths, PCC rigorously identifies geometric violations; even when relying on inferred depths, it remains highly effective at discriminating higher-quality samples.

Dear Reviewer, the discussion deadline is approaching soon. Can you please provide feedback to the authors?

For the rebuttal, we evaluated PCC Best-of-N (BoN) selection on the Tanks and Temples (T&T) dataset for the stochastic baselines: PhotoNVS, VIVID, and GLD. The remaining baseline models are deterministic feedforward networks, for which Best-of-N sampling does not apply.

One important distinction to note is that because PhotoNVS, VIVID, and GLD do not predict target depth, their Best-of-N selection had to be performed using DA3 depth estimations—the exact same estimator used to compute the PCC evaluation metric reported in the table. In contrast, ESC performs Best-of-N selection using its own intrinsically predicted depth maps rather than the external evaluation estimator.

ModelPSNR ↑SSIM ↑LPIPS ↓DINO-Sim ↑DreamSim ↓PCC ↓AUC@15 ↑AUC@30 ↑RPE-R ↓RPE-T ↓
Tanks & Temples Dataset
PhotoNVS10.130.26920.76880.23450.59020.44670.02750.079649.7043.41
PhotoNVS-PCC-BoN10.220.26900.76070.30370.50080.40850.11930.104832.6035.52
VIVID10.380.28590.75270.37930.50400.44160.08330.231033.8931.86
VIVID-PCC-BoN10.560.28580.70220.42240.47710.40410.12280.319031.5630.36
OVIE10.580.30930.82190.06400.78810.51020.00520.022953.0153.35
GLD10.880.28910.63650.50990.40270.41840.49890.620815.9215.33
GLD-PCC-BoN11.340.29830.61150.52770.38900.36230.50930.641515.1115.04
LagerNVS13.480.46690.64200.27380.66300.49010.19370.273635.2533.84
ESC-shadow10.780.26920.63360.65490.29280.42910.16940.383722.0020.08
ESC (Ours)11.270.28820.58350.72420.23620.38260.32870.550114.6215.64
ESC-PCC-BoN (Ours)11.420.28920.55960.73390.22320.28490.33290.548214.7416.28
Oracle (GT-vs-DA3)·····0.18990.77000.83577.556.33

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.