D3SS: Depth Recovery Covering Diverse 3D Scene Scales

Existing depth perception methods overemphasize 2D features of depth maps, introducing diverse 3D distortions caused by the scale problem, creating the demand for depth recovery in 3D scenes. Conventional depth recovery approaches only simulate raw depth at fixed scales in limited scenes, causing a significant simulation-to-reality gap that hinders the generalization of depth recovery models. To address the problem, we propose a diffusion-based raw depth generation pipeline for simulating diverse scene-scale distortions in both global and local patterns. To enhance the generalization capability in diverse RGB conditions, we first integrate the frozen depth foundation model (DFM), which is trained on large-scale RGB-D data, into the proposed framework. We further propose loss functions based on different views of 3D scenes to refine the 3D structure. On the one hand, our loss functions enforce the model to learn 3D features; on the other hand, our losses mitigate overfitting of the 3D distortions caused by DFM. Extensive studies on the recovery of different conditions in multiple datasets demonstrate the effectiveness and progressiveness of the proposed method.