- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work proposes a new paradigm to learn self-supervised visual-spatial representation, meaning an encoder for RGB-D frames. The task is well-motivated, however the motivation to use only self-supervised objectives instead of supervising with features from large foundation models that exceed in geometric and semantic tasks (e.g. VGGT, and DINO) remains unclear. The authors, propose to extend classic ViT with 3D RoPE for encoding for inter-patch, and depth embedding for intra-patch information. Further, they propose to use the DINOv2 objectives together with ranking contrastive SmoothAP loss for multi-view geometric correspondences. All design choices are extensively ablated, and compared with other approaches such as DFormerv2, and MultiMAE, and the distilled model DUNE. Overall, they find that their paradigm improves over DINOv2 in semantic segmentation while remaining competitive in geometric correspondence tasks.
Critical strengths
Tackling important task of building RGB-D encoders.
Extensive ablation shows that a) 3D RoPE is useful, b) that the depth embedding is useful when using dropout, c) that contrastive smoothAP loss is useful, d) MVImgNet2.0 is first not useful on top of ImgNet-1K, however it enables contrastive SmoothAP loss and together it is a trade-off of worsening semantic segmentation vs improving geometric correspondence.
Critical weaknesses
Does not evaluate on Semantic 3D Correspondences for single RGB-D frames, as possibly on HouseCorr3D (ECCV 2026) for at least visible (modal) features. This task would bring geometric and semantic evaluation together in one benchmark, evaluating the features with a simple nearest neighbor setup.
Does not compare with LitePT (CVPR 2026) using DINOv2 features together with PointROPE embedding. This also demonstrates a SOTA RGB-D encoder.
The motivation to use only self-supervised objectives instead of supervising with features from large foundation models that exceed in geometric and semantic tasks (e.g. VGGT, and DINO) remains unclear. Here the authors are basically distilling also from a bigger model with MapAnything. As there does not exist such a diverse real-world dataset with depth. We can only distill bigger models, hence one question remains if it would not be smarter to distill from MapAnything/VGGT and DINO features rather than using the depth output of MapAnything only.
This paper is hard to follow as many tables with different datasets, there could be more consistency. E.g. Table 1 and Table 2 are not evaluated on the same benchmarks.
There is a missing fair comparison with DINOv2-B trained on ImgNet1K and MV-ImgNet2.0 in Table 3, and Table 4. (ViT-B / RMT-L models at comparable data size)
Also a lot of results are just throwed at the read, even if they might not be so important. As the main take differentiate between geometric and semantic task, I would rather have one benchmark for each, which is the most challenging. E.g. ADE20k for semantic segmenation as it yields the most categories, and ScanNet due to more large-scale environments.. Furthermore, the big ablation table could be reduced to the relevant chosen features, and the more detailed information can be kept in the supplementary.
Questions for authors
Is your 3D RoPE implementation different to the PointRoPE of Lite-PT?
In Table 3, why is there not DINOcular-L (Ours) without MV for fair dataset size?