acceptodds
Under review as a conference paper at ICLR 2027

DINOcular: Self-Supervised Visuospatial Representations

Abstract

We introduce a self-supervised framework for learning joint visuospatial representations from RGB-D observations. While modern vision foundation models are trained almost exclusively on RGB images, many embodied systems have access to explicit depth sensing, which provides geometric information that monocular inputs cannot recover. Our method integrates depth-derived geometric priors with a visual backbone through inter-patch and intra-patch fusion, enabling the model to encode both appearance and spatial structure efficiently. The resulting representation shows promising improvements on 3D awareness while preserving semantic transfer: it outperforms prior methods of comparable scale on multiple 3D geometry benchmarks, and remains competitive when probed for standard RGB-D semantic segmentation tasks.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, spatial learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fP68OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This work proposes a new paradigm to learn self-supervised visual-spatial representation, meaning an encoder for RGB-D frames. The task is well-motivated, however the motivation to use only self-supervised objectives instead of supervising with features from large foundation models that exceed in geometric and semantic tasks (e.g. VGGT, and DINO) remains unclear. The authors, propose to extend classic ViT with 3D RoPE for encoding for inter-patch, and depth embedding for intra-patch information. Further, they propose to use the DINOv2 objectives together with ranking contrastive SmoothAP loss for multi-view geometric correspondences. All design choices are extensively ablated, and compared with other approaches such as DFormerv2, and MultiMAE, and the distilled model DUNE. Overall, they find that their paradigm improves over DINOv2 in semantic segmentation while remaining competitive in geometric correspondence tasks.

Critical strengths

Tackling important task of building RGB-D encoders.

Extensive ablation shows that a) 3D RoPE is useful, b) that the depth embedding is useful when using dropout, c) that contrastive smoothAP loss is useful, d) MVImgNet2.0 is first not useful on top of ImgNet-1K, however it enables contrastive SmoothAP loss and together it is a trade-off of worsening semantic segmentation vs improving geometric correspondence.

Critical weaknesses

Does not evaluate on Semantic 3D Correspondences for single RGB-D frames, as possibly on HouseCorr3D (ECCV 2026) for at least visible (modal) features. This task would bring geometric and semantic evaluation together in one benchmark, evaluating the features with a simple nearest neighbor setup.

Does not compare with LitePT (CVPR 2026) using DINOv2 features together with PointROPE embedding. This also demonstrates a SOTA RGB-D encoder.

The motivation to use only self-supervised objectives instead of supervising with features from large foundation models that exceed in geometric and semantic tasks (e.g. VGGT, and DINO) remains unclear. Here the authors are basically distilling also from a bigger model with MapAnything. As there does not exist such a diverse real-world dataset with depth. We can only distill bigger models, hence one question remains if it would not be smarter to distill from MapAnything/VGGT and DINO features rather than using the depth output of MapAnything only.

This paper is hard to follow as many tables with different datasets, there could be more consistency. E.g. Table 1 and Table 2 are not evaluated on the same benchmarks.

There is a missing fair comparison with DINOv2-B trained on ImgNet1K and MV-ImgNet2.0 in Table 3, and Table 4. (ViT-B / RMT-L models at comparable data size)

Also a lot of results are just throwed at the read, even if they might not be so important. As the main take differentiate between geometric and semantic task, I would rather have one benchmark for each, which is the most challenging. E.g. ADE20k for semantic segmenation as it yields the most categories, and ScanNet due to more large-scale environments.. Furthermore, the big ablation table could be reduced to the relevant chosen features, and the more detailed information can be kept in the supplementary.

Questions for authors

Is your 3D RoPE implementation different to the PointRoPE of Lite-PT?

In Table 3, why is there not DINOcular-L (Ours) without MV for fair dataset size?

Reviewer_A6FHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a joint semantic and geometric pretraining approach based on RGBD images. The core idea is to adapt depth information as a positional embedding in the form of 3D RoPE. Additionally, the method integrates conventional DINO/iBOT semantic pre-training and cross-view pre-training into a unified framework. Experiments show the improvement over DINO pretraining trained on ImageNet.

Critical strengths
  • The paper focuses on a new pre-training task where the input is RGBD
  • The paper is well-written and easy to understand. The diagrams are nice and I really like them.
  • The paper tried different loss combinations and dropout tricks, which might be useful for a broader range of tasks.
Critical weaknesses

1 Lack of critical comparisons

  • In the comparison of the geometric task in Table 4, the paper mainly compares against 2D semantic pretraining methods like DINO, yet omits the prevailing 3D approaches such as CroCo, DUSt3R, MASt3R, or VGGT (although they are image-based methods, it's a privilege for the proposed method), which may lead to an incomplete comparison.

  • The multi-view loss appears to affect DINO-based semantic pre-training, as revealed in Table 2. I assume that incorporating the geometric capability will generally degrade the semantic capability due to the competition between objectives. However, in Table 3, the proposed method with geoemtric loss and DINO loss shows better results than DINO alone, which is inconsistent with the observations in Table 2 and the assumption. I assume one reason is that the results of DINO-v2 in Table 3 are trained only on ImageNet rather than on both ImageNet and MVImgNet, as in the proposed method. In this case, DINOv2 should be trained on both datasets for a fair comparison.

Questions for authors
  1. As discussed above, please provide explanations if there are misunderstandings, or provide additional results for a complete comparison.

  2. Rationale for the design choice.

  • Pipeline

From what I understand, the Teacher networks provide privileged information for the student (with insufficient information) to reconstruct semantic representation learning. In this case, the information given to the students must be incomplete. For the geometric pre-training, however, the goal is to give sufficient but not pixel-aligned information across two views for the (Student) network to learn correspondence through reconstruction.

When combining the two, the complete information (local crops and view-complementary cues despite being masked out) from the student branch will bypass the information gap designed for semantic representation learning. I suspect it is one reason that causes the semantic performance drop when incorporating the current geometric target. Are there any designs to avoid this information leakage, or something that I omitted in the paper?

  • Network

The method uses a Swin Transformer architecture, following previous approaches. However, ViT seems to be a better choice that works for broader application scenarios. How does the model behave if we switch to ViT?

I am open to further discussion.

Reviewer_a29SOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DINOcular, a self-supervised framework for learning joint visuospatial representations from RGB-D observations. The framework incorporates depth-derived geometric information through inter-patch modeling with 3D rotary positional encoding and intra-patch fusion with lightweight depth embeddings. Additionally, it combines DINO/iBOT self-distillation with multi-view contrastive learning to enhance semantic representations and geometric consistency.

Critical strengths

1.The paper is well motivated and addresses the limited geometric awareness of existing visual foundation models. By integrating 3D RoPE, intra-patch depth embeddings, and multi-view contrastive learning, the proposed framework effectively combines semantic and geometric information within a unified architecture.

2.The paper provides comprehensive evaluations on semantic segmentation, 3D correspondence estimation, and object pose estimation, supported by ablation studies on geometric encoding and learning objectives. The competitive results on benchmarks such as ScanNet demonstrate the effectiveness of the proposed method.

Critical weaknesses

1.The methodological novelty is somewhat limited, as the proposed framework largely combines existing techniques, including 3D positional encoding (Schenck et al., 2025), depth-guided representation learning (Yin et al., 2025), and multi-view contrastive learning (You et al., 2025). The paper does not sufficiently distinguish its technical contributions from these prior works.

2.The contribution of multi-view learning to general-purpose representation learning remains unclear. As shown in Table 3, introducing the multi-view objective reduces mIoU from 36.02% to 33.88% on ADE20K, 41.47% to 40.82% on NYUDepthv2, and 56.31% to 55.35% on Cityscapes, raising concerns about its effectiveness in preserving semantic representations.

3.The effectiveness of intra-patch depth embedding remains unclear. As shown in Table 1, adding depth embeddings only marginally improves segmentation mIoU from 39.92% to 40.16%, while reducing correspondence recall from 22.74% to 17.90% on NAVI and from 12.00% to 10.50% on ScanNet. These results raise concerns about its effectiveness in enhancing geometric representations.

4.The mathematical formulation of 3D RoPE in Section 3.1 is insufficiently specified. In particular, the construction of the rotation matrices for different spatial axes and how the encoding avoids ambiguity between spatial coordinates are not clearly explained.

5.The cosine similarity loss in Section 3.2 is defined using a dot product, but the normalization of the feature vectors is not explicitly specified.

Questions for authors

please refer to critical weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.