- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies a failure mode in iterative view-based 3D scene generation: missing foreground geometry may not appear as an empty pixel after reprojection because an existing background surface fills the pixel. The authors call these regions hidden holes and propose QuadMask to identify potential locations requiring completion. The method constructs occlusion volumes from input-view depth meshes and tests their intersections with target pixel frustums, using GPU vertex enumeration and quadtree culling. Experiments examine mask prediction on BlendedMVS view pairs and integration into Text2Room and WonderJourney, with an additional hidden-hole-aware inpainting model trained on procedural indoor scenes.
Critical strengths
-
The paper identifies a concrete limitation of using missing projected depth as the sole criterion for inpainting. The distinction between an empty pixel and missing foreground geometry obscured by a projected background is clearly explained and relevant to iterative scene completion. Treating mask construction as a separate problem is useful, since a stronger image generator cannot recover missing content if the region is never selected for editing.
-
QuadMask offers an explicit geometric formulation rather than relying entirely on image-space mask expansion. The use of pixel frustums, occlusion volumes, and hierarchical intersection testing is technically well motivated. Integration into two existing pipelines, together with evaluations using both ground-truth and estimated initial depth, is a useful step toward understanding the method’s practical behavior.
Critical weaknesses
-
The downstream improvement is not cleanly separated from inpainting-model adaptation. Finetuned QM changes both the mask and the inpainting model relative to the baseline. Naive QM, which provides a more direct test of replacing the mask, often degrades LPIPS, PSNR, and geometry. This shows that the proposed mask requires an appropriate completion model, but does not establish how much of the final gain comes from QuadMask itself. A baseline fine-tuned on the same data and training budget while retaining visible-hole-only masks is needed to clarify this contribution.
-
The generation evaluation is narrow relative to the claimed scope. The inpainter is trained on 64 procedural pose pairs, and the 166 evaluation pairs come from the same randomized room distribution centered around bookshelves. The BlendedMVS experiments broaden the mask-detection evaluation, but do not establish downstream generation quality on more varied scenes. The described two-pose evaluation also provides limited evidence about repeated expansion, where masking errors and generated geometry can accumulate. The current results support a proof of concept more clearly than a generally applicable improvement to iterative scene generation.
-
Potentially occluded space is not equivalent to missing geometry requiring an edit. The method identifies space in which unobserved geometry could exist, rather than determining that such geometry is actually present. The paper acknowledges this distinction in its description of a conservative superset, but Section 3.3 makes stronger statements about geometric certainty and preserving well-observed regions. The multi-view OR rule in Eq. 2 further broadens the mask: a region hidden in one source view may be constrained by another. Consequently, the mask alone does not ensure that newly added geometry is compatible with all observations. This tradeoff needs a clearer treatment and an evaluation of unnecessary edits to already supported content.
-
The advantage over closely related geometric masking approaches remains unclear. The related-work discussion acknowledges occupancy-based occlusion reasoning in RealmDreamer and shadow-volume cues in Salimi et al. Differences in the downstream generation framework do not by themselves establish the benefit of QuadMask’s geometric detector. Comparisons on the same mask-prediction task against projected occlusion volumes, backface masks, and simpler depth-boundary mask expansion would help establish the value of the proposed intersection procedure. Accuracy should be considered alongside runtime and memory, since efficiency is a substantial part of the method’s motivation.
-
The quantitative gains are mixed and need a more careful interpretation. With ground-truth initial depth, Finetuned QM improves both pipelines, but the LPIPS changes are small: 0.455 to 0.453 for Text2Room and 0.483 to 0.481 for WonderJourney. With estimated depth, Text2Room’s mean Chamfer distance worsens from 0.274 to 0.404 despite an improvement in the median. This suggests that typical-case gains may coexist with substantial failures. Per-scene results, failure rates, and uncertainty across random seeds would make the practical benefit easier to assess. Figure 4 explicitly uses the best of eight trials, so these examples should be interpreted as achievable outcomes rather than typical performance.
Questions for authors
-
A visible-hole-only baseline fine-tuned on the same procedural data and training budget would help separate model adaptation from mask quality. Comparing alternative masks with the same adapted inpainter, together with an oracle hidden-hole mask, would further clarify the contribution and remaining limitations of QuadMask.
-
The mask evaluation would benefit from a direct accuracy–cost comparison with simpler geometric alternatives. Hidden-hole precision and recall, unnecessary masking of valid content, runtime, and peak memory under both ground-truth and predicted depth would provide the most relevant evidence. The depth tolerance used to construct ground-truth hidden-hole labels also needs specification.
3 comments
Reply to mPe3 (con't)
mPe3: The advantage of QM’s masks over those of previous missing-geometry or occlusion-detection methods is unclear.
We test QuadMask against RealmDreamer’s masking code and our implementation of Salimi et al.’s masking procedure on our synthetic room set. We find that QuadMask has higher precision and recall. While it is true that QM is more time and memory consuming than Salimi et al.’s method, the aim of QuadMask is to provide as exact a mask as possible with volumetric intersections, not purely to be faster. Even so, the time and memory usage of QM is not out of place in an iterative pipeline; inpainting and depth completion take comparable time and sometimes more memory.
Table 5: Accuracy vs. cost comparison of alternative masking methods on 720x720 pixel synthetic room cases. GT hidden holes are defined as having 99% or less depth than the rendered depth.
\text{Masking Method} & \text{Precision} & \text{Recall} & \text{Time (s)} & \text{Peak GPU (GB)} \\\\ \hline \text{QuadMask} & 0.660 & 0.900 & 12.2 & 4.15 \\\\ \text{Salimi et al.} & 0.607 & 0.848 & 1.4 & 1.40 \\\\ \text{RealmDreamer} & 0.361 & 0.421 & 21.3 & 0.34 \end{array}$$ mPe3: Evidence of how well downstream generation quality generalizes to other scenes would be useful, but is missing. We will add visual results from real images and videos to the main paper. While we cannot add images here, for a scene from DL3DV, the novel view scores against the GT novel view are in Table 5. We ran 8 seeds and chose the best for each method by LPIPS. We note that PSNR is higher for baseline because the pretrained visible hole inpaint more closely resembles the GT room in color, but clearly suffers on the other metrics due to artifacts driven by inability to handle hidden holes. Specifically, the scene shows a grandiose office with an ornate table, chair, and background furniture. A hidden hole is created where the table is left incomplete in the novel view render. Every inpainting draw results in finetuned baseline incurring artifacts from the incomplete table, whereas Finetuned QM makes reasonable attempts to complete the table in every draw, but completes the background differently. We can add quantitative results over more DL3DV scenes to the main paper. We also note that Figure 4 in the main paper shows inpainting results on other rooms and objects (while synthetic, the rooms noticeably differ from the bookcase distribution). **Table 6: Results on a real scene from DL3DV.** $$\begin{array}{lccc} \text{Method} & \text{LPIPS} & \text{PSNR} & \text{SSIM} \\\\ \hline \text{Baseline} & 0.623 & \mathbf{12.1} & 0.299 \\\\ \text{Finetuned Baseline} & 0.632 & 11.3 & 0.215 \\\\ \text{Naive QM} & 0.623 & 11.8 & 0.305 \\\\ \text{Finetuned QM} & \mathbf{0.596} & 11.7 & \mathbf{0.376} \end{array}$$ mPe3: Evidence of how well repeated expansion works beyond two-pose evaluation would be useful, but is missing. We provide the following results on 15 rooms from the synthetic room evaluation set. These have three novel views, with poses drawn in the same manner as the two view cases. We find that in these cases, Finetuned QM still attempts geometry completion, while finetuned baseline cannot. We will put qualitative figures in the main paper for this setting as well. **Table 7: Text2Room pipeline results on multiview cases.** $$\begin{array}{lccccc} \text{Method} & \text{PSNR} & \text{SSIM} & \text{LPIPS} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 17.3 & 0.432 & \mathbf{0.520} & \mathbf{4.65 \times 10^{-2}} & 1.96 \times 10^{-2} \\\\ \text{Naive QM} & 14.8 & 0.381 & 0.560 & 8.01 \times 10^{-2} & \mathbf{1.70 \times 10^{-2}} \\\\ \text{Finetuned QM} & 16.9 & 0.432 & 0.535 & 5.12 \times 10^{-2} & 2.31 \times 10^{-2} \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{17.4} & \mathbf{0.442} & 0.528 & 6.10 \times 10^{-2} & 3.09 \times 10^{-2} \end{array}$$ **Table 8: WonderJourney pipeline results on multiview cases.** $$\begin{array}{lccccc} \text{Method} & \text{PSNR} & \text{SSIM} & \text{LPIPS} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 16.8 & 0.358 & \mathbf{0.543} & \mathbf{4.73 \times 10^{-2}} & 2.23 \times 10^{-2} \\\\ \text{Naive QM} & 14.4 & 0.346 & 0.579 & 7.33 \times 10^{-2} & \mathbf{2.02 \times 10^{-2}} \\\\ \text{Finetuned QM} & 16.8 & 0.367 & 0.558 & 5.50 \times 10^{-2} & 3.49 \times 10^{-2} \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{17.0} & \mathbf{0.377} & 0.549 & 5.82 \times 10^{-2} & 3.51 \times 10^{-2} \end{array}$$Reply to mPe3 (con't pt. 3)
mPe3: Quantitative gains are mixed and need more careful interpretation.
Because of the relatively small size of hidden holes, attempting geometry completion is not always clearly rewarded by quantitative metrics. A typical case is where an attempt to complete the bookcase results in a reasonable inpaint, but the resulting color is different than the GT, and this results in an image metric penalty compared to the baseline. The value of attempting hidden hole completion is clearer with qualitative outputs–these show that ignoring potential hidden holes and retaining ground truth render pixels, while working for certain metrics, leave noticeable geometry gaps.
Regarding the best-of-eight setup in Figure 4: the images chosen were primarily done so for the aesthetic quality of visible hole completion. In general, finetuned QM attempts hidden hole geometry completion, and the baseline is definitionally unable to. We can add all eight draws to the Supplement for reference.
We furthermore provide aggregate results on two scenes from our evaluation set, on just the finetuned baseline and finetuned QM variants. There are cases where QM Finetune scores less than the finetuned baseline–in these, the potential hidden hole area may be larger, resulting in more variable finetuned QM inpainting. However, we again note that while baseline may do better on metrics in these cases, it necessarily leaves incomplete geometry untouched, which is not a plausible scene completion.
Table 9: Text2Room pipeline given depth results on room 67 over 150 draws.
\text{Method} & \text{PSNR} & \text{SSIM} & \text{LPIPS} & \text{Mean CD} \\\\ \hline \text{Finetuned Baseline} & 13.8 \pm 0.30 & 0.517 \pm 0.006 & 0.548 \pm 0.003 & \mathbf{0.112} \pm 0.002 \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{15.5} \pm 0.44 & \mathbf{0.612} \pm 0.008 & \mathbf{0.530} \pm 0.008 & 0.170 \pm 0.04 \end{array}$$ **Table 10: WonderJourney pipeline given depth results on room 67 over 150 draws.** $$\begin{array}{lcccc} \text{Method} & \text{PSNR} & \text{SSIM} & \text{LPIPS} & \text{Mean CD} \\\\ \hline \text{Finetuned Baseline} & 13.86 \pm 0.5 & 0.427 \pm 0.029 & 0.569 \pm 0.006 & \mathbf{0.112} \pm 0.004 \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{15.01} \pm 0.8 & \mathbf{0.510} \pm 0.035 & \mathbf{0.543} \pm 0.011 & 0.144 \pm 0.022 \end{array}$$ **Table 11: Text2Room pipeline given depth results on room 83 over 150 draws.** $$\begin{array}{lcccc} \text{Method} & \text{PSNR} & \text{SSIM} & \text{LPIPS} & \text{Mean CD} \\\\ \hline \text{Finetuned Baseline} & \mathbf{19.7} \pm 0.4 & 0.568 \pm 0.005 & \mathbf{0.444} \pm 0.004 & \mathbf{0.3 \times 10^{-3}} \pm 0.1 \times 10^{-3} \\\\ \text{Depth Filtered Finetuned QM} & 18.9 \pm 1.2 & \mathbf{0.583} \pm 0.023 & 0.448 \pm 0.021 & 3.7 \times 10^{-3} \pm 3.2 \times 10^{-3} \end{array}$$ **Table 12: WonderJourney pipeline given depth results on room 83 over 150 draws.** $$\begin{array}{lcccc} \text{Method} & \text{PSNR} & \text{SSIM} & \text{LPIPS} & \text{Mean CD} \\\\ \hline \text{Finetuned Baseline} & \mathbf{19.7} \pm 0.5 & 0.479 \pm 0.006 & \mathbf{0.489} \pm 0.002 & \mathbf{0.4 \times 10^{-3}} \pm 0.2 \times 10^{-3} \\\\ \text{Depth Filtered Finetuned QM} & 18.8 \pm 1.3 & \mathbf{0.501} \pm 0.012 & 0.495 \pm 0.017 & 2.2 \times 10^{-3} \pm 2.8 \times 10^{-3} \end{array}$$
Reply to mPe3
mPe3: The inpainting performance of finetuned QM may be due to domain adaptation via the process of finetuning, rather than the use of QM’s hidden hole masks.
We provide the following experiment with SD-inpaint finetuned on our dataset with visible only masks. We see that FID solidly prefers finetuned QM. While some image metrics tie or barely favor the finetuned baseline on the given depth setting, this is due to the relative size of hidden holes. True hidden holes that need new geometry are relatively small, and filling them also slightly changes the color of potential hidden hole pixels, which penalizes PSNR. Chamfer distance has a similar pattern. Despite this, hidden hole-aware inpainting decisively improves FID, and usually matches or outperforms other image metrics. Qualitative results, as shown in the main paper, also demonstrate that ignoring hidden holes noticeably damages iterative inpainting to the human eye–in Figure 5, the baseline fills visible holes as true to the domain as Finetuned QM, but it definitionally cannot complete missing bookcase geometry.
In order to further differentiate the effects of filling the hidden holes from slight changes in other pixels from Finetuned QM inpainting, we test the following: for each method, only keep new content in hidden hole pixels if the new predicted depth is [] closer than the existing rendered depth. With this, on the given depth setting, the image metrics outperform the finetuned baseline. With the more realistic setting of no given depth, this is the case even without depth filtering.
Table 1: Text2Room pipeline results with GT initial view depth.
\text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 125.4 & 0.453 & 17.0 & \mathbf{0.042} & \mathbf{3.26 \times 10^{-3}} \\\\ \text{Naive QM} & 114.9 & 0.460 & 15.9 & 0.065 & 7.78 \times 10^{-3} \\\\ \text{Finetuned QM} & 112.5 & 0.453 & 16.9 & 0.054 & 4.37 \times 10^{-3} \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{108.6} & \mathbf{0.450} & \mathbf{17.2} & 0.053 & 4.29 \times 10^{-3} \end{array}$$ **Table 2: WonderJourney pipeline results with GT initial view depth.** $$\begin{array}{lccccc} \text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 136.5 & 0.480 & 17.0 & \mathbf{0.045} & \mathbf{3.29 \times 10^{-3}} \\\\ \text{Naive QM} & 143.6 & 0.520 & 15.1 & 0.078 & 1.15 \times 10^{-2} \\\\ \text{Finetuned QM} & 117.6 & 0.481 & 17.1 & 0.051 & 4.40 \times 10^{-3} \\\\ \text{Depth Filtered Finetuned QM} & \mathbf{116.6} & \mathbf{0.476} & 17.1 & 0.048 & 4.11 \times 10^{-3} \end{array}$$ **Table 3: Text2Room pipeline results without given depth.** $$\begin{array}{lccccc} \text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 150.6 & 0.501 & 16.3 & 0.359 & \mathbf{1.45 \times 10^{-2}} \\\\ \text{Naive QM} & \mathbf{128.6} & 0.501 & 15.6 & \mathbf{0.298} & 2.07 \times 10^{-2} \\\\ \text{Finetuned QM} & 131.2 & \mathbf{0.498} & 16.3 & 0.404 & 1.59 \times 10^{-2} \end{array}$$ **Table 4: WonderJourney pipeline results without given depth.** $$\begin{array}{lccccc} \text{Method} & \text{FID} & \text{LPIPS} & \text{PSNR} & \text{Mean CD} & \text{Median CD} \\\\ \hline \text{Finetuned Baseline} & 175.4 & 0.522 & 16.4 & 0.561 & \mathbf{1.54 \times 10^{-2}} \\\\ \text{Naive QM} & 157.6 & 0.539 & 15.0 & 0.636 & 3.25 \times 10^{-2} \\\\ \text{Finetuned QM} & \mathbf{148.0} & \mathbf{0.518} & \mathbf{16.6} & \mathbf{0.468} & 1.95 \times 10^{-2} \end{array}$$ mPe3: QM’s masked region output is a conservative superset of all potentially missing geometry, and can lead to hallucinated geometry that conflicts with existing scene content. We agree that QM gives a superset, but this a conservative approach that remains possible for an iterative pipeline. In order to attempt geometry completion in hidden holes, a generative model must be used. But a generative model may alter existing pixels unnecessarily, specifically, it may alter pixels in regions that should stay static. Having the geometry-based QM superset allows restriction of possible updates to only pixels that have the potential to be hidden holes, instead of the whole image. While this can still alter potential hidden hole pixels that do not end up with new geometry, the alternatives in an iterative inpainting pipeline would be either to allow arbitrary generative control over the image, or to disregard hidden holes and accept visible geometry errors as in Figure 5.