Handling Hidden Holes in View-Based 3D Scene Generation

3D scene generation from a single image is a challenging task with many applications in areas such as AR/VR, gaming, and embodied AI. One popular family of methods for this task is iterative view-based 2D inpainting, which expands 3D scene coverage by projecting known geometry from existing views into a novel view and inpainting pixels with missing depth. These methods typically assume that all occluded geometry in existing views appears as holes of missing depth in the novel view, and that filling these visible holes completes the scene. However, this assumption is often incorrect: although visible holes necessarily mean missing geometry, the lack of visible holes does not necessarily mean no geometry is missing. It is possible that there are no visible holes among pixels corresponding to potentially missing geometry because some other object also projects to these pixels in the novel view. We refer to these pixels as hidden holes, which can prevent inpainting methods from properly completing occluded objects because they are never flagged for inpainting. To address this, we introduce QuadMask, our geometric occlusion detection system that identifies both visible and hidden holes in novel views, given existing views and camera poses as input. QuadMask outputs can aid downstream RGB pipelines by signaling areas that could be missing visually important geometry.