- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ESC, a diffusion-based framework for single-image novel view synthesis that conditions a pretrained image-editing diffusion transformer on source depth and Plücker rays of the source and target cameras. Using LoRA adapters and a Ray ControlNet, the model jointly generates target RGB and depth. Its trainable components are optimized using flow-matching and auxiliary reconstruction, perceptual losses. The paper also introduces the PCC, a bidirectional depth-reprojection score used both for evaluation and for inference-time Best-of-N selection. Experiments on DL3DV, RealEstate10K, and Tanks & Temples report improvements in LPIPS, DINO-Sim, DreamSim, and PCC relative to the evaluated baselines.
Critical strengths
- ESC-Base leads the compared baselines on LPIPS, DINO-Sim, and DreamSim across all three benchmarks, so the reported perceptual advantage does not depend entirely on Best-of-N selection. Results on seven Tanks & Temples scenes also provide evidence of transfer beyond the DL3DV/RealEstate10K fine-tuning mixture.
- Table 6 shows that adding depth improves perceptual quality and PCC over ESC-Shadow across all three real-world benchmarks, with further gains from PCC selection. These results support the components' practical usefulness within the proposed pipeline.
- Controlled synthetic experiments distinguish geometric perturbations from source-compatible changes in occluded regions. PCC increases for the former while remaining low for the latter, illustrating its potential as a complement to target-image similarity metrics. These tests support the distinction between source-compatible geometry and reproducing a particular ground-truth completion, although they do not establish universal consistency guarantees.
Critical weaknesses
Major Weaknesses
- The paper motivates its contribution through the limitations of geometry warping followed by 2D completion, but does not adequately distinguish its approach from existing methods that already incorporate geometry into generation. In particular, MVGD (Guizilini et al., CVPR 2025) combines ray-map conditioning with novel-view image and depth generation, while SceneCompleter (Chen et al., arXiv:2506.10981) explicitly motivates joint geometry and appearance completion through the geometric limitations of 2D inpainting. Neither work is cited in the manuscript or included in the experimental comparison. Their omission leaves the novelty of RGB–depth generation and geometry-aware conditioning insufficiently established. The paper should explain what ESC contributes beyond adapting a stronger image-editing backbone and adding a geometric selection criterion.
- ESC-Shadow simultaneously removes depth conditioning and depth prediction while retaining Ray ControlNet, so it does not isolate these components or establish that improved appearance–structure coupling explains the gains. The separate effects of the reconstruction, LPIPS, and DINO losses are also not ablated. Their basic visual-fidelity rationale is understandable, but the experiments do not establish their individual contributions to the reported perceptual and geometric gains. Furthermore, other stochastic NVS methods could use external depth estimation for PCC selection, but neither this alternative nor ESC-Shadow with PCC selection is evaluated. Selection uses ESC-predicted depth whereas benchmark PCC uses DA3 depth, but both rely on the same reprojection criterion. Matched-budget selection baselines and independent geometric checks would better distinguish generator-specific benefits from generic sampling-and-ranking gains.
- PCC measures depth-reprojection conflicts over valid projection coverage, while the main inference procedure selects the lowest-scoring candidate without refining its RGB or geometry. Selecting the minimum does not ensure that the selected candidate is geometrically consistent, particularly when all candidates contain errors. Nor does it explicitly verify agreement between the generated RGB and depth. Although Appendix C.11 describes a multi-view extension, the main source–target selection procedure is not shown to enforce mutual consistency across independently generated target views. Moreover, log transformation and clipping reduce the influence of extreme residuals, while averaging can dilute severe localized errors. These are reasonable robustness choices, but their implications for ranking structural errors require validation. Describing PCC as strictly enforcing or guaranteeing scene consistency is therefore stronger than the demonstrated mechanism supports.
- Table 9 shows that LagerNVS leads PSNR/SSIM across all three datasets, while competing methods lead pose AUC and translation accuracy. ESC does achieve strong rotation-error results on some benchmarks, but the overall results indicate a trade-off rather than uniformly superior viewpoint control. The perceptual and PCC improvements therefore need to be distinguished from precise compliance with the camera prompt. Blurring and pixel-alignment sensitivity do not adequately explain the weaker pose results. Figure 5 also shows substantial variation in structure, scale, and framing under identical conditioning. Without separating source-visible and newly revealed regions, it is unclear how much reflects valid completion diversity versus scene or viewpoint drift.
Minor Weaknesses
- ESC-Shadow appears in Figure 4 before being defined, while Section 4.1 introduces only ESC-Base and ESC-PCC-BoN. Its definition and central quantitative ablation are deferred to Appendix D.2 and Table 6.
- Although the caption defines the modifications and pixel counts, the scene objects and red error overlays are easy to confuse. The projection directions, residuals, counts, and normalized score need a clearer visual connection. An unmodified-pair reference score would make PCC = 0.0232 more interpretable.
- Figures 1 and 5 should clarify camera changes and source–target overlap. Per-candidate PCC values and reprojection-error visualizations would help interpret Figure 5. The in-the-wild examples illustrate plausible generations but provide limited evidence of precise geometric consistency.
Questions for authors
- How does ESC differ from MVGD and SceneCompleter in geometric conditioning, RGB–depth generation, and consistency verification? Can the authors provide a comparison under comparable input settings?
- Which ablations isolate source-depth conditioning, target-depth prediction, Ray ControlNet, and the auxiliary losses? What evidence shows that improved RGB–depth agreement explains the gains, rather than the addition of depth supervision or other training changes?
- What candidate count and total latency, including scoring, are used for each real-world benchmark? How do other stochastic baselines or ESC-Shadow perform with external depth estimation and PCC selection under the same candidate budget? Please report generation and depth-estimation costs separately.
- How sensitive are PCC rankings to thresholding, log transformation, and clipping, especially for severe localized versus widespread minor errors? Can these rankings and consistency across multiple generated views be validated using an independent geometric reference?
- What explains the weaker pose AUC and translation-error results? For Figure 5, can the authors provide the requested camera transformation, candidate PCC/pose errors, and a visible-versus-newly-revealed-region analysis to distinguish completion diversity from drift?
- Is test-time source-depth conditioning computed from the source image alone or the 60-frame DA3 reconstruction? If multiple frames are used, do they include the target, and do baselines receive equivalent geometric information? Please distinguish conditioning inputs from depth used only for supervision or evaluation.
2 comments
Q3: Candidate Count, Latency, and Matched-Compute Selection
- Candidate count: We use a consistent candidate count of N=20 for ESC-PCC-BoN across all three real-world benchmarks (DL3DV, RealEstate10K, and Tanks and Temples).
- Latency breakdown: Generating a single RGB-D sample with ESC-Base takes 3 seconds (table 7), and generating N=20 candidates takes about 60 seconds. Because ESC intrinsically predicts target depth, computing the bidirectional PCC score across all 20 candidates is lightweight (no scoring overhead).
- Matched-compute comparison & baselines: Even under a strictly matched single-sample budget (N=1), ESC-Base (w/o BoN selection) already outperforms the baselines on perceptual metrics across all benchmarks (Table 1), demonstrating that our gains do not rely solely on Best-of-N sampling. Furthermore, Table 7 shows that our model matches or exceeds the inference speed of other stochastic generative baselines.
Q4: PCC Design Choices, Sensitivity, and Validation Rationale for thresholding, clipping, and log transformation: Because PCC is fundamentally a 3D distance error metric, these design choices are necessary to ensure geometric stability rather than arbitrary tuning: Thresholding and clipping prevent distant background regions with ill-defined depth (such as the sky) from corrupting the score. Depth estimations for such regions are inherently noisy or unbounded, so clipping ensures PCC evaluates reliable, well-defined scene geometry. Log transformation maps depth distances to a logarithmic scale so that PCC does not disproportionately overweight errors on far-away pixels relative to foreground geometry. This naturally balances sensitivity between localized structural errors and widespread minor deviations.
Q5: Pose AUC and Translation-Error Results Standard camera pose estimators are trained on sharp, real-world imagery, making their behavior unpredictable when evaluated on generative artifacts—such as the severe hallucinations of GLD or the heavy blurring of LagerNVS. As a result, a blurry or hallucinated image can yield an arbitrarily strong pose score or SSIM despite lacking true structural coherence. We refer the reviewer to our dedicated pose metric vs. PCC analysis in Appendix C.12.1, which examines these limitations in detail, as well as our qualitative comparisons against LagerNVS and GLD, which show that ESC achieves noticeably better visual quality and structurally plausible content completion that standard pose metrics fail to capture.
Q6: Test-Time Source-Depth Conditioning At test time, source-depth conditioning is computed solely from the single source image—we do not use a 60-frame DA3 reconstruction as inference input, nor is any target-view information used. The statement in Appendix A.4 ("Pairs are sampled from within the 60-frame chunks") refers only to how source–target image pairs are sampled from video sequences, not to multi-frame depth conditioning at inference time. We will revise the wording in Appendix A.4 to make this distinction—and the separation between test-time conditioning inputs versus supervision/evaluation—completely clear.
We thank the reviewer for their thoughtful feedback.
Q1 and W1: Novelty relative to MVGD and SceneCompleter We will expand our Related Work section to include a direct comparison with MVGD and SceneCompleter. ESC differs from these works in key aspects:
Q2 and W2 Attribution of components and losses [Q2, W2] To isolate the contribution of depth, our ESC-Shadow experiment (Table 6, Appendix D.2) directly ablates source-depth conditioning and target-depth prediction
By removing only the depth input and output channels, ESC-Shadow serves as a controlled RGB-only baseline that isolates the effect of joint RGB-D modeling from the backbone and training objectives. Comparing ESC-Shadow against ESC-Base across all three benchmarks (DL3DV, RealEstate10K, and Tanks and Temples) shows consistent improvements in both perceptual quality and 3D geometric consistency (PCC). This confirms that the gains are not merely a byproduct of the pretrained backbone or perceptual losses, but are directly driven by grounding the generation in source depth and jointly predicting target depth. Comparing ESC-Shadow against ESC-Base demonstrates that adding depth conditioning and joint RGB-D prediction yields consistent improvements across all benchmarks. Specifically, while the auxiliary perceptual losses primarily drive gains in 2D visual quality (as reflected in perceptual metrics), the ESC-Shadow ablation confirms that incorporating explicit depth conditioning and prediction provides additional, measurable gains in 3D geometric consistency (PCC) as well as overall synthesis quality. We will bring the definition of ESC-Shadow and Table 6 into the main text to make this ablation clear upfront.
Regarding the comment that "ESC-Shadow simultaneously removes depth conditioning and depth prediction while retaining Ray ControlNet, so it does not isolate these components...", we would like to clarify the rationale behind this design choice. Because our base model (Qwen-Edit) is a 2D image-editing model, retaining the Ray ControlNet is strictly necessary to adapt it to novel-view synthesis; without precise camera ray conditioning, the target viewpoint is unspecified and the NVS task itself becomes ill-defined. Keeping Ray ControlNet fixed in ESC-Shadow therefore provides the proper controlled baseline to specifically isolate the impact of adding depth conditioning and joint depth prediction.