- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Semantics from Structure (SfS), a framework for unsupervised representation learning from multi-view images using explicit 3D geometric constraints. Its core method, DiFF, jointly optimizes a randomly initialized 2D encoder and a 3D Gaussian feature field through bidirectional distillation, without relying on pretrained models or data augmentation. The method combines feature alignment, SIGReg, and 3D spatial smoothness regularization to learn view-consistent representations. Experiments on several multi-view datasets demonstrate promising segmentation performance and cross-view consistency under limited-data settings.
Critical strengths
- The paper explores a novel and intrinsic approach for unsupervised representation learning using explicit 3D geometric constraints. It shares some intuition with JEPA, using a shared 3D feature field to encourage consistency across 2D observations.
- The method learns spatially coherent features from a small set of multi-view images, showing promising segmentation results and improved multi-view consistency.
Critical weaknesses
-
My main concern is the significance of the proposed setting. It is unclear why the method focuses on small-scale, from-scratch learning without pretrained VFMs. I believe the greater significance of this approach lies in its potential to use 3D geometric supervision to improve large-scale visual representation learning. The current experiments feel more like a proof of concept, and the broader impact remains unclear. Moreover, the relatively low performance on 3D-OVS raises further questions about the practical value of this approach compared to methods leveraging pretrained VFMs.
This is my main reason for the current rating. I am open to raising my score if the rebuttal or discussion with other reviewers provides a convincing justification for this setting.
-
The bidirectional distillation mechanism is not sufficiently validated. Jointly optimizing the 2D encoder and 3D feature field is central to the method, but the ablations do not clearly show why this design is necessary or how much it contributes. Comparisons with other alternatives would be helpful.
Questions for authors
See weaknesses
2 comments
Response to Reviewer gCMc (Part 2)
We are writing to follow up on our response to Weaknesses 1 and 2 with new experimental results on the Wheat3DGS dataset that directly validate our bidirectional distillation mechanism against unidirectional 3D feature-field distillation baselines, as well as direct backbone probing and the use of DiFF for adaptation of pretrained DINO and ImageNet encoders. The full results tables (Tables 1 and 2) have been posted in our follow-up response to Reviewer 5Vqh and incorporated into the revised manuscript (Table 1, Section 5.2, and Appendix A.5).
These experiments directly address Weaknesses 1 and 2 as follows:
-
Significance in Both the From-Scratch and Pretrained-Adaptation Settings (Weakness 1): Under direct backbone probing, from-scratch DiFF† improves IoU by 8.3 percentage points over the projection head (reaching 54.6% IoU), surpassing the supervised Wheat3DGS baseline by 4.6 percentage points using only the unlabeled images from a single scene. In addition, applying DiFF to adapt pretrained backbones improves 1-shot 2D segmentation over frozen DINO-R50 and ImageNet-R50 by 3.0 and 2.0 percentage points in IoU, respectively (improving on 6 of 7 plots in both cases and reaching 60.2% IoU overall).
-
Validation of Bidirectional Distillation (Weakness 2): To isolate the contribution of jointly optimizing the 2D encoder and the 3D feature field, we constructed unidirectional feature field distillation baselines from frozen DINO (DINO-GS) and frozen ImageNet (ImageNet-GS) ResNet50 encoders. Our results show that unidirectional distillation from frozen pretrained encoders exhibits substantial degradation when transferring from 2D to 3D because the 3D field passively averages view-inconsistent 2D features through a linear projection layer. Jointly optimizing both components via DiFF reduces this drop. In 1-shot 3D segmentation, DiFF_DINO improves over DINO-GS by 6.1 percentage points in IoU (+4.2 in 0-shot 3D) and DiFF_IN improves over ImageNet-GS by 9.2 percentage points in IoU (+2.4 in 0-shot 3D), while even our from-scratch DiFF† surpasses both frozen feature-field baselines. Furthermore, when directly probing the pre-projection backbone features in 1-shot 2D segmentation, our from-scratch DiFF† outperforms the 2D-only from-scratch LeJEPA† baseline across all seven plots by 7.9 percentage points in IoU, confirming the benefit of the 3D feature field even without the projection-head bottleneck.
Response to Reviewer gCMc (Part 1)
We thank the reviewer for their thoughtful review and for recognizing our novel use of explicit 3D geometric constraints for unsupervised representation learning, as well as the spatial coherence, multi-view consistency, and promising segmentation performance of our learned features.
Response to Critical Weaknesses:
1. We are very excited to see that the reviewer identified the potential of our method for large-scale visual representation learning. We also believe that our method has great potential for large-scale pretraining, especially for applications such as replacing frame-wise vision transformers in feed-forward 3D reconstruction models like VGGT. To directly address the main concerns of the reviewer, we will explain why we targeted single-scene experiments and why the proposed setting is still significant even at a small scale. Our work on a small scale is specifically proposed to target niche settings that are out-of-distribution, where VFMs tend to fail. We chose the Wheat3DGS benchmark to demonstrate this application. We compare directly with FruitNeRF, which fully utilizes VFMs for 3D segmentation; specifically, FruitNeRF employs GroundingDINO and SAM for detecting and segmenting wheat heads in 2D, and these predictions are then distilled into the radiance field for 3D segmentation. As shown in Table 1, the fully unsupervised clustering using our learned representations performs competitively with FruitNeRF despite using far less data, a smaller encoder, less compute, and a simpler clustering method for segmentation. With only a single example, a linear probe can separate the learned representations and outperforms FruitNeRF. Furthermore, our approach achieves similar performance to Wheat3DGS, which utilizes a fully supervised YOLO model along with SAM for segmentation and a more complex 3D segmentation strategy. We believe that our results on the Wheat3DGS benchmark demonstrate the power of our learned representations for niche settings where VFMs tend to fail.
Furthermore, although the method is trained per scene, our experiments demonstrate that the trained encoder maintains strong generalizability to new scenes and even different domains, retaining 94.3% of the per-scene accuracy on new scenes of the same domain and 89.7% of the accuracy on new domains (cross-dataset), and can also be used to fine-tune pretrained backbones (such as our ImageNet-pretrained experiment in Table 1). Establishing this formulation in the single-scene setting provides the necessary foundation before scaling to multi-scene training (for which we outline three concrete strategies in our response to Reviewer 5Vqh, Weakness 3).
To briefly address the performance on 3D-OVS, we want to reiterate that our method is not intended to replace VFMs in common settings that remain in-distribution for VFMs. We wanted to include the results on this dataset to demonstrate that our method can be applied to indoor scenes. We hypothesize that the lower performance on this dataset is due to the human-defined semantic classes. For example, the dataset contains objects such as a Gundam action figure, which consists of multiple semantic sub-parts. When applying our method, the linear probe must learn from a single example which sub-parts of the object make up the whole object, which may not be linearly separable.
2. Jointly optimizing both components is necessary because the 3D feature field enforces explicit cross-view geometric consistency, while the shared weights of the 2D encoder allow spatially separated 3D Gaussians with similar visual appearance to be mapped to the same semantic representation (which local 3D smoothness alone cannot achieve). We experiment with LeJEPA to provide a direct comparison to our bidirectional distillation mechanism. LeJEPA, similar to our method, employs an MSE loss in the latent space with SIGReg to prevent representation collapse, but aligns 2D image views directly without a 3D feature field. As shown in Table 1, when constrained to the same data for training, our method outperforms LeJEPA by a large margin, demonstrating the superiority of our proposed bidirectional distillation.