acceptodds
Under review as a conference paper at ICLR 2027

Handling Hidden Holes in View-Based 3D Scene Generation

Abstract

3D scene generation from a single image is a challenging task with many applications in areas such as AR/VR, gaming, and embodied AI. One popular family of methods for this task is iterative view-based 2D inpainting, which expands 3D scene coverage by projecting known geometry from existing views into a novel view and inpainting pixels with missing depth. These methods typically assume that all occluded geometry in existing views appears as holes of missing depth in the novel view, and that filling these visible holes completes the scene. However, this assumption is often incorrect: although visible holes necessarily mean missing geometry, the lack of visible holes does not necessarily mean no geometry is missing. It is possible that there are no visible holes among pixels corresponding to potentially missing geometry because some other object also projects to these pixels in the novel view. We refer to these pixels as hidden holes, which can prevent inpainting methods from properly completing occluded objects because they are never flagged for inpainting. To address this, we introduce QuadMask, our geometric occlusion detection system that identifies both visible and hidden holes in novel views, given existing views and camera poses as input. QuadMask outputs can aid downstream RGB pipelines by signaling areas that could be missing visually important geometry.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D vision, occlusion awareness, monocular scene reconstruction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mPe3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a failure mode in iterative view-based 3D scene generation: missing foreground geometry may not appear as an empty pixel after reprojection because an existing background surface fills the pixel. The authors call these regions hidden holes and propose QuadMask to identify potential locations requiring completion. The method constructs occlusion volumes from input-view depth meshes and tests their intersections with target pixel frustums, using GPU vertex enumeration and quadtree culling. Experiments examine mask prediction on BlendedMVS view pairs and integration into Text2Room and WonderJourney, with an additional hidden-hole-aware inpainting model trained on procedural indoor scenes.

Critical strengths
  1. The paper identifies a concrete limitation of using missing projected depth as the sole criterion for inpainting. The distinction between an empty pixel and missing foreground geometry obscured by a projected background is clearly explained and relevant to iterative scene completion. Treating mask construction as a separate problem is useful, since a stronger image generator cannot recover missing content if the region is never selected for editing.

  2. QuadMask offers an explicit geometric formulation rather than relying entirely on image-space mask expansion. The use of pixel frustums, occlusion volumes, and hierarchical intersection testing is technically well motivated. Integration into two existing pipelines, together with evaluations using both ground-truth and estimated initial depth, is a useful step toward understanding the method’s practical behavior.

Critical weaknesses
  1. The downstream improvement is not cleanly separated from inpainting-model adaptation. Finetuned QM changes both the mask and the inpainting model relative to the baseline. Naive QM, which provides a more direct test of replacing the mask, often degrades LPIPS, PSNR, and geometry. This shows that the proposed mask requires an appropriate completion model, but does not establish how much of the final gain comes from QuadMask itself. A baseline fine-tuned on the same data and training budget while retaining visible-hole-only masks is needed to clarify this contribution.

  2. The generation evaluation is narrow relative to the claimed scope. The inpainter is trained on 64 procedural pose pairs, and the 166 evaluation pairs come from the same randomized room distribution centered around bookshelves. The BlendedMVS experiments broaden the mask-detection evaluation, but do not establish downstream generation quality on more varied scenes. The described two-pose evaluation also provides limited evidence about repeated expansion, where masking errors and generated geometry can accumulate. The current results support a proof of concept more clearly than a generally applicable improvement to iterative scene generation.

  3. Potentially occluded space is not equivalent to missing geometry requiring an edit. The method identifies space in which unobserved geometry could exist, rather than determining that such geometry is actually present. The paper acknowledges this distinction in its description of a conservative superset, but Section 3.3 makes stronger statements about geometric certainty and preserving well-observed regions. The multi-view OR rule in Eq. 2 further broadens the mask: a region hidden in one source view may be constrained by another. Consequently, the mask alone does not ensure that newly added geometry is compatible with all observations. This tradeoff needs a clearer treatment and an evaluation of unnecessary edits to already supported content.

  4. The advantage over closely related geometric masking approaches remains unclear. The related-work discussion acknowledges occupancy-based occlusion reasoning in RealmDreamer and shadow-volume cues in Salimi et al. Differences in the downstream generation framework do not by themselves establish the benefit of QuadMask’s geometric detector. Comparisons on the same mask-prediction task against projected occlusion volumes, backface masks, and simpler depth-boundary mask expansion would help establish the value of the proposed intersection procedure. Accuracy should be considered alongside runtime and memory, since efficiency is a substantial part of the method’s motivation.

  5. The quantitative gains are mixed and need a more careful interpretation. With ground-truth initial depth, Finetuned QM improves both pipelines, but the LPIPS changes are small: 0.455 to 0.453 for Text2Room and 0.483 to 0.481 for WonderJourney. With estimated depth, Text2Room’s mean Chamfer distance worsens from 0.274 to 0.404 despite an improvement in the median. This suggests that typical-case gains may coexist with substantial failures. Per-scene results, failure rates, and uncertainty across random seeds would make the practical benefit easier to assess. Figure 4 explicitly uses the best of eight trials, so these examples should be interpreted as achievable outcomes rather than typical performance.

Questions for authors
  1. A visible-hole-only baseline fine-tuned on the same procedural data and training budget would help separate model adaptation from mask quality. Comparing alternative masks with the same adapted inpainter, together with an oracle hidden-hole mask, would further clarify the contribution and remaining limitations of QuadMask.

  2. The mask evaluation would benefit from a direct accuracy–cost comparison with simpler geometric alternatives. Hidden-hole precision and recall, unnecessary masking of valid content, runtime, and peak memory under both ground-truth and predicted depth would provide the most relevant evidence. The depth tolerance used to construct ground-truth hidden-hole labels also needs specification.

3 comments
AuthorsOpenReview ↗

Reply to mPe3

mPe3: The inpainting performance of finetuned QM may be due to domain adaptation via the process of finetuning, rather than the use of QM’s hidden hole masks.

We provide the following experiment with SD-inpaint finetuned on our dataset with visible only masks. We see that FID solidly prefers finetuned QM. While some image metrics tie or barely favor the finetuned baseline on the given depth setting, this is due to the relative size of hidden holes. True hidden holes that need new geometry are relatively small, and filling them also slightly changes the color of potential hidden hole pixels, which penalizes PSNR. Chamfer distance has a similar pattern. Despite this, hidden hole-aware inpainting decisively improves FID, and usually matches or outperforms other image metrics. Qualitative results, as shown in the main paper, also demonstrate that ignoring hidden holes noticeably damages iterative inpainting to the human eye–in Figure 5, the baseline fills visible holes as true to the domain as Finetuned QM, but it definitionally cannot complete missing bookcase geometry.

In order to further differentiate the effects of filling the hidden holes from slight changes in other pixels from Finetuned QM inpainting, we test the following: for each method, only keep new content in hidden hole pixels if the new predicted depth is [] closer than the existing rendered depth. With this, on the given depth setting, the image metrics outperform the finetuned baseline. With the more realistic setting of no given depth, this is the case even without depth filtering.

Table 1: Text2Room pipeline results with GT initial view depth.

\text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 125.4 & 0.453 & 17.0 & \mathbf{0.042} & \mathbf{3.26 \times 10^{-3}} \\\\ \text{Naive QM} & 114.9 & 0.460 & 15.9 & 0.065 & 7.78 \times 10^{-3} \\\\ \text{Finetuned QM} & 112.5 & 0.453 & 16.9 & 0.054 & 4.37 \times 10^{-3} \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{108.6} & \mathbf{0.450} & \mathbf{17.2} & 0.053 & 4.29 \times 10^{-3} \end{array}$$ **Table 2: WonderJourney pipeline results with GT initial view depth.** $$\begin{array}{lccccc} \text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 136.5 & 0.480 & 17.0 & \mathbf{0.045} & \mathbf{3.29 \times 10^{-3}} \\\\ \text{Naive QM} & 143.6 & 0.520 & 15.1 & 0.078 & 1.15 \times 10^{-2} \\\\ \text{Finetuned QM} & 117.6 & 0.481 & 17.1 & 0.051 & 4.40 \times 10^{-3} \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{116.6} & \mathbf{0.476} & 17.1 & 0.048 & 4.11 \times 10^{-3} \end{array}$$ **Table 3: Text2Room pipeline results without given depth.** $$\begin{array}{lccccc} \text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 150.6 & 0.501 & 16.3 & 0.359 & \mathbf{1.45 \times 10^{-2}} \\\\ \text{Naive QM} & \mathbf{128.6} & 0.501 & 15.6 & \mathbf{0.298} & 2.07 \times 10^{-2} \\\\ \text{Finetuned QM} & 131.2 & \mathbf{0.498} & 16.3 & 0.404 & 1.59 \times 10^{-2} \end{array}$$ **Table 4: WonderJourney pipeline results without given depth.** $$\begin{array}{lccccc} \text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 175.4 & 0.522 & 16.4 & 0.561 & \mathbf{1.54 \times 10^{-2}} \\\\ \text{Naive QM} & 157.6 & 0.539 & 15.0 & 0.636 & 3.25 \times 10^{-2} \\\\ \text{Finetuned QM} & \mathbf{148.0} & \mathbf{0.518} & \mathbf{16.6} & \mathbf{0.468} & 1.95 \times 10^{-2} \end{array}$$ mPe3: QM’s masked region output is a conservative superset of all potentially missing geometry, and can lead to hallucinated geometry that conflicts with existing scene content. We agree that QM gives a superset, but this a conservative approach that remains possible for an iterative pipeline. In order to attempt geometry completion in hidden holes, a generative model must be used. But a generative model may alter existing pixels unnecessarily, specifically, it may alter pixels in regions that should stay static. Having the geometry-based QM superset allows restriction of possible updates to only pixels that have the potential to be hidden holes, instead of the whole image. While this can still alter potential hidden hole pixels that do not end up with new geometry, the alternatives in an iterative inpainting pipeline would be either to allow arbitrary generative control over the image, or to disregard hidden holes and accept visible geometry errors as in Figure 5.
AuthorsOpenReview ↗

Reply to mPe3 (con't)

mPe3: The advantage of QM’s masks over those of previous missing-geometry or occlusion-detection methods is unclear.

We test QuadMask against RealmDreamer’s masking code and our implementation of Salimi et al.’s masking procedure on our synthetic room set. We find that QuadMask has higher precision and recall. While it is true that QM is more time and memory consuming than Salimi et al.’s method, the aim of QuadMask is to provide as exact a mask as possible with volumetric intersections, not purely to be faster. Even so, the time and memory usage of QM is not out of place in an iterative pipeline; inpainting and depth completion take comparable time and sometimes more memory.

Table 5: Accuracy vs. cost comparison of alternative masking methods on 720x720 pixel synthetic room cases. GT hidden holes are defined as having 99% or less depth than the rendered depth.

\text{Masking Method} & \text{Precision} & \text{Recall} & \text{Time (s)} & \text{Peak GPU (GB)} \\\\ \hline \text{QuadMask} & 0.660 & 0.900 & 12.2 & 4.15 \\\\ \text{Salimi et al.} & 0.607 & 0.848 & 1.4 & 1.40 \\\\ \text{RealmDreamer} & 0.361 & 0.421 & 21.3 & 0.34 \end{array}$$ mPe3: Evidence of how well downstream generation quality generalizes to other scenes would be useful, but is missing. We will add visual results from real images and videos to the main paper. While we cannot add images here, for a scene from DL3DV, the novel view scores against the GT novel view are in Table 5. We ran 8 seeds and chose the best for each method by LPIPS. We note that PSNR is higher for baseline because the pretrained visible hole inpaint more closely resembles the GT room in color, but clearly suffers on the other metrics due to artifacts driven by inability to handle hidden holes. Specifically, the scene shows a grandiose office with an ornate table, chair, and background furniture. A hidden hole is created where the table is left incomplete in the novel view render. Every inpainting draw results in finetuned baseline incurring artifacts from the incomplete table, whereas Finetuned QM makes reasonable attempts to complete the table in every draw, but completes the background differently. We can add quantitative results over more DL3DV scenes to the main paper. We also note that Figure 4 in the main paper shows inpainting results on other rooms and objects (while synthetic, the rooms noticeably differ from the bookcase distribution). **Table 6: Results on a real scene from DL3DV.** $$\begin{array}{lccc} \text{Method} & \text{LPIPS} & \text{PSNR} & \text{SSIM} \\\\ \hline \text{Baseline} & 0.623 & \mathbf{12.1} & 0.299 \\\\ \text{Finetuned Baseline} & 0.632 & 11.3 & 0.215 \\\\ \text{Naive QM} & 0.623 & 11.8 & 0.305 \\\\ \text{Finetuned QM} & \mathbf{0.596} & 11.7 & \mathbf{0.376} \end{array}$$ mPe3: Evidence of how well repeated expansion works beyond two-pose evaluation would be useful, but is missing. We provide the following results on 15 rooms from the synthetic room evaluation set. These have three novel views, with poses drawn in the same manner as the two view cases. We find that in these cases, Finetuned QM still attempts geometry completion, while finetuned baseline cannot. We will put qualitative figures in the main paper for this setting as well. **Table 7: Text2Room pipeline results on multiview cases.** $$\begin{array}{lccccc} \text{Method} & \text{PSNR} & \text{SSIM} & \text{LPIPS} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 17.3 & 0.432 & \mathbf{0.520} & \mathbf{4.65 \times 10^{-2}} & 1.96 \times 10^{-2} \\\\ \text{Naive QM} & 14.8 & 0.381 & 0.560 & 8.01 \times 10^{-2} & \mathbf{1.70 \times 10^{-2}} \\\\ \text{Finetuned QM} & 16.9 & 0.432 & 0.535 & 5.12 \times 10^{-2} & 2.31 \times 10^{-2} \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{17.4} & \mathbf{0.442} & 0.528 & 6.10 \times 10^{-2} & 3.09 \times 10^{-2} \end{array}$$ **Table 8: WonderJourney pipeline results on multiview cases.** $$\begin{array}{lccccc} \text{Method} & \text{PSNR} & \text{SSIM} & \text{LPIPS} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 16.8 & 0.358 & \mathbf{0.543} & \mathbf{4.73 \times 10^{-2}} & 2.23 \times 10^{-2} \\\\ \text{Naive QM} & 14.4 & 0.346 & 0.579 & 7.33 \times 10^{-2} & \mathbf{2.02 \times 10^{-2}} \\\\ \text{Finetuned QM} & 16.8 & 0.367 & 0.558 & 5.50 \times 10^{-2} & 3.49 \times 10^{-2} \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{17.0} & \mathbf{0.377} & 0.549 & 5.82 \times 10^{-2} & 3.51 \times 10^{-2} \end{array}$$
AuthorsOpenReview ↗

Reply to mPe3 (con't pt. 3)

mPe3: Quantitative gains are mixed and need more careful interpretation.

Because of the relatively small size of hidden holes, attempting geometry completion is not always clearly rewarded by quantitative metrics. A typical case is where an attempt to complete the bookcase results in a reasonable inpaint, but the resulting color is different than the GT, and this results in an image metric penalty compared to the baseline. The value of attempting hidden hole completion is clearer with qualitative outputs–these show that ignoring potential hidden holes and retaining ground truth render pixels, while working for certain metrics, leave noticeable geometry gaps.

Regarding the best-of-eight setup in Figure 4: the images chosen were primarily done so for the aesthetic quality of visible hole completion. In general, finetuned QM attempts hidden hole geometry completion, and the baseline is definitionally unable to. We can add all eight draws to the Supplement for reference.

We furthermore provide aggregate results on two scenes from our evaluation set, on just the finetuned baseline and finetuned QM variants. There are cases where QM Finetune scores less than the finetuned baseline–in these, the potential hidden hole area may be larger, resulting in more variable finetuned QM inpainting. However, we again note that while baseline may do better on metrics in these cases, it necessarily leaves incomplete geometry untouched, which is not a plausible scene completion.

Table 9: Text2Room pipeline given depth results on room 67 over 150 draws.

\text{Method} & \text{PSNR} & \text{SSIM} & \text{LPIPS} & \text{Mean CD} \\\\ \hline \text{Finetuned Baseline} & 13.8 \pm 0.30 & 0.517 \pm 0.006 & 0.548 \pm 0.003 & \mathbf{0.112} \pm 0.002 \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{15.5} \pm 0.44 & \mathbf{0.612} \pm 0.008 & \mathbf{0.530} \pm 0.008 & 0.170 \pm 0.04 \end{array}$$ **Table 10: WonderJourney pipeline given depth results on room 67 over 150 draws.** $$\begin{array}{lcccc} \text{Method} & \text{PSNR} & \text{SSIM} & \text{LPIPS} & \text{Mean CD} \\\\ \hline \text{Finetuned Baseline} & 13.86 \pm 0.5 & 0.427 \pm 0.029 & 0.569 \pm 0.006 & \mathbf{0.112} \pm 0.004 \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{15.01} \pm 0.8 & \mathbf{0.510} \pm 0.035 & \mathbf{0.543} \pm 0.011 & 0.144 \pm 0.022 \end{array}$$ **Table 11: Text2Room pipeline given depth results on room 83 over 150 draws.** $$\begin{array}{lcccc} \text{Method} & \text{PSNR} & \text{SSIM} & \text{LPIPS} & \text{Mean CD} \\\\ \hline \text{Finetuned Baseline} & \mathbf{19.7} \pm 0.4 & 0.568 \pm 0.005 & \mathbf{0.444} \pm 0.004 & \mathbf{0.3 \times 10^{-3}} \pm 0.1 \times 10^{-3} \\\\ \text{Depth Filtered Finetuned QM} & 18.9 \pm 1.2 & \mathbf{0.583} \pm 0.023 & 0.448 \pm 0.021 & 3.7 \times 10^{-3} \pm 3.2 \times 10^{-3} \end{array}$$ **Table 12: WonderJourney pipeline given depth results on room 83 over 150 draws.** $$\begin{array}{lcccc} \text{Method} & \text{PSNR} & \text{SSIM} & \text{LPIPS} & \text{Mean CD} \\\\ \hline \text{Finetuned Baseline} & \mathbf{19.7} \pm 0.5 & 0.479 \pm 0.006 & \mathbf{0.489} \pm 0.002 & \mathbf{0.4 \times 10^{-3}} \pm 0.2 \times 10^{-3} \\\\ \text{Depth Filtered Finetuned QM} & 18.8 \pm 1.3 & \mathbf{0.501} \pm 0.012 & 0.495 \pm 0.017 & 2.2 \times 10^{-3} \pm 2.8 \times 10^{-3} \end{array}$$
Reviewer_8WLqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes QuadMesh, an invisible missing geometry detection method, for iterative multi-view-based scene generation. This paper tries to resolve the missing mask region because of invisible or occluding missing geometry in multi-view-based scene generation.

QuadMesh specifically first predicts the geometry and depth, then uses the depth information to detect occlusion. This occlusion will be further used to determine the missing geometry, thus identifying missing pixels for the next view inpainting.

Critical strengths
  1. This paper proposes a simple yet effective method to detect invisible missing geometry, which can be further converted into an inpainting mask for next views in scene generation.
  2. The problem setting is practical and valid.
  3. The improvement presented in the tables seems solid over the paper’s selected baseline.
Critical weaknesses
  1. Lack of comparison with masks from previous missing-geometry or occlusion-detection methods (eg., RealmDreamer, Geometry-Aware Diffusion Models for Multiview Scene Inpainting)
  2. Missing control experiment with a fine-tuned inpainting model with different mask strategies—Vanilla view-based mask, mask produced from other methods. (Naive QM shows mixed results against the baseline in Tables 1 and 2, which shows fine-tuning might solely account for improvement.)
Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Reply to 8WLq

8WLq: The inpainting performance of finetuned QM may be due to the process of finetuning, rather than the use of QM’s hidden hole masks.

We provide the following experiment with SD-inpaint finetuned on our dataset with visible only masks; we call this setting “Finetuned Baseline.” We see that FID solidly prefers finetuned QM. While some image metrics tie or barely favor the finetuned baseline on the given depth setting, this is due to the relative size of hidden holes. True hidden holes that need new geometry are relatively small, and filling them also slightly changes the color of potential hidden hole pixels, which penalizes PSNR. Chamfer distance has a similar pattern. Despite this, hidden hole-aware inpainting decisively improves FID, and usually matches or outperforms other image metrics. Qualitative results, as shown in the main paper, also demonstrate that ignoring hidden holes noticeably damages iterative inpainting to the human eye–in Figure 5, the baseline fills visible holes as true to the domain as Finetuned QM, but it definitionally cannot complete missing bookcase geometry.

In order to further differentiate the effects of filling the hidden holes from slight changes in other pixels from Finetuned QM inpainting, we test the following: for each method, only keep new content in hidden hole pixels if the new predicted depth is less than 95% of the existing rendered depth. With this, on the given depth setting, the image metrics outperform the finetuned baseline. With the more realistic setting of no given depth, this is the case even without depth filtering.

Table 1: Text2Room pipeline results with GT initial view depth.

\text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 125.4 & 0.453 & 17.0 & \mathbf{0.042} & \mathbf{3.26 \times 10^{-3}} \\\\ \text{Naive QM} & 114.9 & 0.460 & 15.9 & 0.065 & 7.78 \times 10^{-3} \\\\ \text{Finetuned QM} & 112.5 & 0.453 & 16.9 & 0.054 & 4.37 \times 10^{-3} \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{108.6} & \mathbf{0.450} & \mathbf{17.2} & 0.053 & 4.29 \times 10^{-3} \end{array}$$ **Table 2: WonderJourney pipeline results with GT initial view depth.** $$\begin{array}{lccccc} \text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 136.5 & 0.480 & 17.0 & \mathbf{0.045} & \mathbf{3.29 \times 10^{-3}} \\\\ \text{Naive QM} & 143.6 & 0.520 & 15.1 & 0.078 & 1.15 \times 10^{-2} \\\\ \text{Finetuned QM} & 117.6 & 0.481 & 17.1 & 0.051 & 4.40 \times 10^{-3} \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{116.6} & \mathbf{0.476} & 17.1 & 0.048 & 4.11 \times 10^{-3} \end{array}$$ **Table 3: Text2Room pipeline results without given depth.** $$\begin{array}{lccccc} \text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 150.6 & 0.501 & 16.3 & 0.359 & \mathbf{1.45 \times 10^{-2}} \\\\ \text{Naive QM} & \mathbf{128.6} & 0.501 & 15.6 & \mathbf{0.298} & 2.07 \times 10^{-2} \\\\ \text{Finetuned QM} & 131.2 & \mathbf{0.498} & 16.3 & 0.404 & 1.59 \times 10^{-2} \end{array}$$ **Table 4: WonderJourney pipeline results without given depth.** $$\begin{array}{lccccc} \text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 175.4 & 0.522 & 16.4 & 0.561 & \mathbf{1.54 \times 10^{-2}} \\\\ \text{Naive QM} & 157.6 & 0.539 & 15.0 & 0.636 & 3.25 \times 10^{-2} \\\\ \text{Finetuned QM} & \mathbf{148.0} & \mathbf{0.518} & \mathbf{16.6} & \mathbf{0.468} & 1.95 \times 10^{-2} \end{array}$$
AuthorsOpenReview ↗

Reply to 8WLq (con't)

8WLq: Comparison of QM’s masks with those of previous missing-geometry or occlusion-detection methods would be helpful, but is missing.

We test QuadMask against RealmDreamer’s masking code and our implementation of Salimi et al.’s masking procedure on our synthetic room set. We find that QuadMask has higher precision and recall. While it is true that QM is more time and memory consuming than Salimi et al.’s method, the aim of QuadMask is to provide as exact a mask as possible with volumetric intersections, not purely to be faster. Even so, the time and memory usage of QM is not out of place in an iterative pipeline; inpainting and depth completion take comparable time and sometimes more memory.

Table 5: Accuracy vs. cost comparison of alternative masking methods on 720x720 pixel synthetic room cases. GT hidden holes are defined as having 99% or less depth than the rendered depth.

\text{Masking Method} & \text{Precision} & \text{Recall} & \text{Time (s)} & \text{Peak GPU (GB)} \\\\ \hline \text{QuadMask} & 0.660 & 0.900 & 12.2 & 4.15 \\\\ \text{Salimi et al.} & 0.607 & 0.848 & 1.4 & 1.40 \\\\ \text{RealmDreamer} & 0.361 & 0.421 & 21.3 & 0.34 \end{array}$$

Thanks to the authors, I think this rebuttal resolves most of my concerns. I will raise my ratings, but as indicated by my confidence score, I am still unsure about the level of contribution of this paper. My positive rating will only reflect the fact that my concerns, which are low-confidence ones, are mostly addressed.

Reviewer_vPD9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies "hidden holes" as an important issue in prior works on iterative view-based 3D scene generation: These pipelines inpaint only visible holes, i.e. novel-view pixels with no reprojected depth. When the novel view looks at space that was occluded in the input view, but a background surface (e.g. the wall behind a chair) reprojects onto those pixels, there is no hole to fill and the occluded foreground (the back of the chair) is never generated. To this end, the paper proposes QuadMask, a geometric occlusion detection method that computes both visible and hidden holes. QuadMask converts depth maps into into a watertight per-view mesh that is cut at depth discontinuities. Each triangle in the mesh is extruded from the input camera into a prism representing occluded 3D volumes. The method then computes the minimum-depth intersection between the frustums and prisms. The frustum–prism pairs are pruned with a coarse-to-fine frustum quadtree. Finally, the visible holes, backface self-occlusions, and multi-view hidden hole conditions are combined to form the final mask. Evaluation on BlendedMVS+ view pairs demonstrates a 99.7% recall in flagging unobserved geometry. When combined with a fine-tuned hidden-hole-aware Stable Diffusion inpainter on synthetic Infinigen bookshelf room scenes, QuadMask improves 3D Chamfer Distance and visual completion scores over baseline pipelines.

Critical strengths
  • This paper identified and clearly articulated an important failure mode in prior iterative view-based 3D scene generation approaches. By distinguishing between visible holes and hidden holes, the authors provide a an explanation for why current 2D inpainting pipelines systematically fail to generate complete 3D structures. Formulating this problem explicitly and giving a principled, model-agnostic geometric criterion to address it is the paper's main contribution.
  • The proposed method is technically sound and interesting. The problem formulation is well-motivated, and come with clear theoretical analysis. The quadtree acceleration is an effective spatial data structure for reducing the complexity.
  • The fixed-cost 165-triplet enumeration with quadtree culling is a clean GPU realization.
Critical weaknesses
  • It is not clear how robust the proposed method is to the errors in upstream monocular depth estimation. Depth errors may affect the mesh construction and extrusion step, potentially corrupting downstream results. No sensitivity analysis on depth noise robustness is provided.
  • QuadMask relies on a multi-view logical "OR" operation, flagging any pixel that could potentially be a hidden hole. This produces a conservative superset of all potential missing geometry, causing a drop in precision. As the authors note, providing such large masked regions can reduce the necessary context important for the downstream diffusion models, causing them to hallucinate geometry that conflicts with existing scene content.
  • The downstream benefit may not be completely attributed to QuadMask. Pairing QuadMask with an off-the-shelf inpainter is worse than the baseline on nearly every metrics. The mask alone therefore does not help. The gains come from "Finetuned QM". That inpainter is trained on the data sampled from the same procedural Infinigen bookshelf generator as the test pairs, while the baseline uses an un-tuned inpainter. The important experiment with the same fine-tuning with visible-only masks is important to separate the mask's contribution from mere domain adaptation.
  • There is limited empirical evidence to assess the generality of the proposed method. Reported results are limited to synthetic rooms with a single object class (bookshelves). There is no study on real scenes. Also, the methods are only tested with a custom SD-based inpainter. More recent diffusion-based models especially video-diffusion-based generators are not tested.
Questions for authors
  • How does the baseline that fine-tunes SD-Inpaint on the same 64 Infinigen pairs with visible-only masks perform in the same evaluation? This can help isolate the mask's contribution from domain adaptation.
  • How sensitive are the masks and downstream results with respect to the errors in the input depth maps?
  • How does the method perform on scenes from real videos?
2 comments
AuthorsOpenReview ↗

Reply to vPD9

vPD9: The inpainting performance of finetuned QM may be due to domain adaptation via the process of finetuning, rather than the use of QM’s hidden hole masks.

We provide the following experiment with SD-inpaint finetuned on our dataset with visible only masks; we call this setting “Finetuned Baseline.” We see that FID solidly prefers finetuned QM. While some image metrics tie or barely favor the finetuned baseline on the given depth setting, this is due to the relative size of hidden holes. True hidden holes that need new geometry are relatively small, and filling them also slightly changes the color of potential hidden hole pixels, which penalizes PSNR. Chamfer distance has a similar pattern. Despite this, hidden hole-aware inpainting decisively improves FID, and usually matches or outperforms other image metrics. Qualitative results, as shown in the main paper, also demonstrate that ignoring hidden holes noticeably damages iterative inpainting to the human eye–in Figure 5, the baseline fills visible holes as true to the domain as Finetuned QM, but it definitionally cannot complete missing bookcase geometry.

In order to further differentiate the effects of filling the hidden holes from slight changes in other pixels from Finetuned QM inpainting, we test the following: only keep new content in hidden hole pixels if the new predicted depth is less than 95% of the existing rendered depth. With this, on the given depth setting, the image metrics outperform the finetuned baseline. With the more realistic setting of no given depth, this is the case even without depth filtering.

Table 1: Text2Room pipeline results with GT initial view depth.

\text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 125.4 & 0.453 & 17.0 & \mathbf{0.042} & \mathbf{3.26 \times 10^{-3}} \\\\ \text{Naive QM} & 114.9 & 0.460 & 15.9 & 0.065 & 7.78 \times 10^{-3} \\\\ \text{Finetuned QM} & 112.5 & 0.453 & 16.9 & 0.054 & 4.37 \times 10^{-3} \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{108.6} & \mathbf{0.450} & \mathbf{17.2} & 0.053 & 4.29 \times 10^{-3} \end{array}$$ **Table 2: WonderJourney pipeline results with GT initial view depth.** $$\begin{array}{lccccc} \text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 136.5 & 0.480 & 17.0 & \mathbf{0.045} & \mathbf{3.29 \times 10^{-3}} \\\\ \text{Naive QM} & 143.6 & 0.520 & 15.1 & 0.078 & 1.15 \times 10^{-2} \\\\ \text{Finetuned QM} & 117.6 & 0.481 & 17.1 & 0.051 & 4.40 \times 10^{-3} \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{116.6} & \mathbf{0.476} & 17.1 & 0.048 & 4.11 \times 10^{-3} \end{array}$$ **Table 3: Text2Room pipeline results without given depth.** $$\begin{array}{lccccc} \text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 150.6 & 0.501 & 16.3 & 0.359 & \mathbf{1.45 \times 10^{-2}} \\\\ \text{Naive QM} & \mathbf{128.6} & 0.501 & 15.6 & \mathbf{0.298} & 2.07 \times 10^{-2} \\\\ \text{Finetuned QM} & 131.2 & \mathbf{0.498} & 16.3 & 0.404 & 1.59 \times 10^{-2} \end{array}$$ **Table 4: WonderJourney pipeline results without given depth.** $$\begin{array}{lccccc} \text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 175.4 & 0.522 & 16.4 & 0.561 & \mathbf{1.54 \times 10^{-2}} \\\\ \text{Naive QM} & 157.6 & 0.539 & 15.0 & 0.636 & 3.25 \times 10^{-2} \\\\ \text{Finetuned QM} & \mathbf{148.0} & \mathbf{0.518} & \mathbf{16.6} & \mathbf{0.468} & 1.95 \times 10^{-2} \end{array}$$ vPD9: QM’s masked region output is a conservative superset of all potentially missing geometry, and can lead to hallucinated geometry that conflicts with existing scene content. We agree that QM gives a superset, but this a conservative approach that remains possible for an iterative pipeline. In order to attempt geometry completion in hidden holes, a generative model must be used. But a generative model may alter existing pixels unnecessarily, specifically, it may alter pixels in regions that should stay static. Having the geometry-based QM superset allows restriction of possible updates to only pixels that have the potential to be hidden holes, instead of the whole image. While this can still alter potential hidden hole pixels that do not end up with new geometry, the alternatives in an iterative inpainting pipeline would be either to allow arbitrary generative control over the image, or to disregard hidden holes and accept visible geometry errors as in Figure 5.
AuthorsOpenReview ↗

Reply to vPD9 (con't)

vPD9: Analysis of QM’s sensitivity to upstream monocular depth estimation error is useful for showing robustness, but missing.

We provide quantitative results in main paper Table 2 and Tables 3 and 4 here that show that a conventional upstream depth predictor maintains the benefit of using QuadMask. In other words, the usual amount of upstream depth prediction error still allows effective hidden hole inpainting.

vPD9: Evidence of how the proposed method performs on real scenes would be useful, but is missing.

We will add visual results from real images and videos to the main paper. While we cannot add images here, for a scene from DL3DV, the novel view scores against the GT novel view are in Table 5. We ran 8 seeds and chose the best for each method by LPIPS. We note that PSNR is higher for baseline because the pretrained visible hole inpaint more closely resembles the GT room in color, but clearly suffers on the other metrics due to artifacts driven by inability to handle hidden holes. Specifically, the scene shows a grandiose office with an ornate table, chair, and background furniture. A hidden hole is created where the table is left incomplete in the novel view render. Every inpainting draw results in finetuned baseline incurring artifacts from the incomplete table, whereas Finetuned QM makes reasonable attempts to complete the table in every draw, but completes the background differently. We can add quantitative results over more DL3DV scenes to the main paper. We also note that Figure 4 in the main paper shows inpainting results on other rooms and objects (while synthetic, the rooms noticeably differ from the bookcase distribution).

Table 5: Results on a real scene from DL3DV.

\text{Method} & \text{LPIPS} & \text{PSNR} & \text{SSIM} \\\\ \hline \text{Baseline} & 0.623 & \mathbf{12.1} & 0.299 \\\\ \text{Finetuned Baseline} & 0.632 & 11.3 & 0.215 \\\\ \text{Naive QM} & 0.623 & 11.8 & 0.305 \\\\ \text{Finetuned QM} & \mathbf{0.596} & 11.7 & \mathbf{0.376} \end{array}$$ vPD9: Testing QM with more recent diffusion-based models such as video-diffusion-based generators would be helpful, but is missing. We can add finetuning attempts on more modern image diffusion models. While this is not directly possible yet, we can also attempt integration with video diffusion models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.