acceptodds
Under review as a conference paper at ICLR 2027

Semantics from Structure: Unsupervised Representation Learning from Multi-View Images via Distillation with Feature Fields

Abstract

Unsupervised representation learning typically relies on heavy data augmentation and massive datasets. Reducing these dependencies is necessary for applications in domains where large-scale data collection is impractical. To this end, we introduce Semantics from Structure, a novel paradigm for unsupervised representation learning from multi-view observations that leverages 3D geometry to learn robust visual representations from limited data. Central to our methodology is Distillation with Feature Fields, which performs bidirectional distillation between a radiance field and a randomly initialized image encoder. We demonstrate that our framework successfully learns multi-view consistent semantic representations from a small collection of images without any augmentation or momentum encoders. The learned representations achieve strong downstream performance on 2D and 3D multi-view segmentation across a variety of scenes with no additional fine-tuning. Our approach demonstrates that learning from structural priors yields robust representations with drastically reduced data requirements.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
self-supervised learning, unsupervised representation learning, 3D Gaussian splatting, feature fields, multi-view consistency, 3D semantic segmentation, few-shot segmentation, plant phenotyping

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gCMcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Semantics from Structure (SfS), a framework for unsupervised representation learning from multi-view images using explicit 3D geometric constraints. Its core method, DiFF, jointly optimizes a randomly initialized 2D encoder and a 3D Gaussian feature field through bidirectional distillation, without relying on pretrained models or data augmentation. The method combines feature alignment, SIGReg, and 3D spatial smoothness regularization to learn view-consistent representations. Experiments on several multi-view datasets demonstrate promising segmentation performance and cross-view consistency under limited-data settings.

Critical strengths
  1. The paper explores a novel and intrinsic approach for unsupervised representation learning using explicit 3D geometric constraints. It shares some intuition with JEPA, using a shared 3D feature field to encourage consistency across 2D observations.
  2. The method learns spatially coherent features from a small set of multi-view images, showing promising segmentation results and improved multi-view consistency.
Critical weaknesses
  1. My main concern is the significance of the proposed setting. It is unclear why the method focuses on small-scale, from-scratch learning without pretrained VFMs. I believe the greater significance of this approach lies in its potential to use 3D geometric supervision to improve large-scale visual representation learning. The current experiments feel more like a proof of concept, and the broader impact remains unclear. Moreover, the relatively low performance on 3D-OVS raises further questions about the practical value of this approach compared to methods leveraging pretrained VFMs.

    This is my main reason for the current rating. I am open to raising my score if the rebuttal or discussion with other reviewers provides a convincing justification for this setting.

  2. The bidirectional distillation mechanism is not sufficiently validated. Jointly optimizing the 2D encoder and 3D feature field is central to the method, but the ablations do not clearly show why this design is necessary or how much it contributes. Comparisons with other alternatives would be helpful.

Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer gCMc (Part 1)

We thank the reviewer for their thoughtful review and for recognizing our novel use of explicit 3D geometric constraints for unsupervised representation learning, as well as the spatial coherence, multi-view consistency, and promising segmentation performance of our learned features.

Response to Critical Weaknesses:

1. We are very excited to see that the reviewer identified the potential of our method for large-scale visual representation learning. We also believe that our method has great potential for large-scale pretraining, especially for applications such as replacing frame-wise vision transformers in feed-forward 3D reconstruction models like VGGT. To directly address the main concerns of the reviewer, we will explain why we targeted single-scene experiments and why the proposed setting is still significant even at a small scale. Our work on a small scale is specifically proposed to target niche settings that are out-of-distribution, where VFMs tend to fail. We chose the Wheat3DGS benchmark to demonstrate this application. We compare directly with FruitNeRF, which fully utilizes VFMs for 3D segmentation; specifically, FruitNeRF employs GroundingDINO and SAM for detecting and segmenting wheat heads in 2D, and these predictions are then distilled into the radiance field for 3D segmentation. As shown in Table 1, the fully unsupervised clustering using our learned representations performs competitively with FruitNeRF despite using far less data, a smaller encoder, less compute, and a simpler clustering method for segmentation. With only a single example, a linear probe can separate the learned representations and outperforms FruitNeRF. Furthermore, our approach achieves similar performance to Wheat3DGS, which utilizes a fully supervised YOLO model along with SAM for segmentation and a more complex 3D segmentation strategy. We believe that our results on the Wheat3DGS benchmark demonstrate the power of our learned representations for niche settings where VFMs tend to fail.

Furthermore, although the method is trained per scene, our experiments demonstrate that the trained encoder maintains strong generalizability to new scenes and even different domains, retaining 94.3% of the per-scene accuracy on new scenes of the same domain and 89.7% of the accuracy on new domains (cross-dataset), and can also be used to fine-tune pretrained backbones (such as our ImageNet-pretrained experiment in Table 1). Establishing this formulation in the single-scene setting provides the necessary foundation before scaling to multi-scene training (for which we outline three concrete strategies in our response to Reviewer 5Vqh, Weakness 3).

To briefly address the performance on 3D-OVS, we want to reiterate that our method is not intended to replace VFMs in common settings that remain in-distribution for VFMs. We wanted to include the results on this dataset to demonstrate that our method can be applied to indoor scenes. We hypothesize that the lower performance on this dataset is due to the human-defined semantic classes. For example, the dataset contains objects such as a Gundam action figure, which consists of multiple semantic sub-parts. When applying our method, the linear probe must learn from a single example which sub-parts of the object make up the whole object, which may not be linearly separable.

2. Jointly optimizing both components is necessary because the 3D feature field enforces explicit cross-view geometric consistency, while the shared weights of the 2D encoder allow spatially separated 3D Gaussians with similar visual appearance to be mapped to the same semantic representation (which local 3D smoothness alone cannot achieve). We experiment with LeJEPA to provide a direct comparison to our bidirectional distillation mechanism. LeJEPA, similar to our method, employs an MSE loss in the latent space with SIGReg to prevent representation collapse, but aligns 2D image views directly without a 3D feature field. As shown in Table 1, when constrained to the same data for training, our method outperforms LeJEPA by a large margin, demonstrating the superiority of our proposed bidirectional distillation.

AuthorsOpenReview ↗

Response to Reviewer gCMc (Part 2)

We are writing to follow up on our response to Weaknesses 1 and 2 with new experimental results on the Wheat3DGS dataset that directly validate our bidirectional distillation mechanism against unidirectional 3D feature-field distillation baselines, as well as direct backbone probing and the use of DiFF for adaptation of pretrained DINO and ImageNet encoders. The full results tables (Tables 1 and 2) have been posted in our follow-up response to Reviewer 5Vqh and incorporated into the revised manuscript (Table 1, Section 5.2, and Appendix A.5).

These experiments directly address Weaknesses 1 and 2 as follows:

  1. Significance in Both the From-Scratch and Pretrained-Adaptation Settings (Weakness 1): Under direct backbone probing, from-scratch DiFF† improves IoU by 8.3 percentage points over the projection head (reaching 54.6% IoU), surpassing the supervised Wheat3DGS baseline by 4.6 percentage points using only the unlabeled images from a single scene. In addition, applying DiFF to adapt pretrained backbones improves 1-shot 2D segmentation over frozen DINO-R50 and ImageNet-R50 by 3.0 and 2.0 percentage points in IoU, respectively (improving on 6 of 7 plots in both cases and reaching 60.2% IoU overall).

  2. Validation of Bidirectional Distillation (Weakness 2): To isolate the contribution of jointly optimizing the 2D encoder and the 3D feature field, we constructed unidirectional feature field distillation baselines from frozen DINO (DINO-GS) and frozen ImageNet (ImageNet-GS) ResNet50 encoders. Our results show that unidirectional distillation from frozen pretrained encoders exhibits substantial degradation when transferring from 2D to 3D because the 3D field passively averages view-inconsistent 2D features through a linear projection layer. Jointly optimizing both components via DiFF reduces this drop. In 1-shot 3D segmentation, DiFF_DINO improves over DINO-GS by 6.1 percentage points in IoU (+4.2 in 0-shot 3D) and DiFF_IN improves over ImageNet-GS by 9.2 percentage points in IoU (+2.4 in 0-shot 3D), while even our from-scratch DiFF† surpasses both frozen feature-field baselines. Furthermore, when directly probing the pre-projection backbone features in 1-shot 2D segmentation, our from-scratch DiFF† outperforms the 2D-only from-scratch LeJEPA† baseline across all seven plots by 7.9 percentage points in IoU, confirming the benefit of the 3D feature field even without the projection-head bottleneck.

Reviewer_5VqhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper present a multi-view representation learning framework using 3D GS to establish view correpondance. To be more specific. give multi-view images, first a 3D Gaussian splatting representation of the env is constructed. Each GS is then associated with a feature vector. Then a image encoder and the GS vectors are trained together to make sure the feature predicted by the encoder align with the feature rendered from GS feature vector. The results show that the encoder managed to learn to produce meaningful features of input images.

Critical strengths
  1. The presented method make a meaningful methodological contribution. While it shares DINO's learning principle, the use of GS as geometry constraint is novel.

  2. The experiments show that the learned features are indeed meaningful and semantically rich.

  3. The trained encoder is not scene specific and can server as a generalizable semantic info inference tool.

Critical weaknesses
  1. While the method is sound and novel itself, I have concerns regarding the practical advantages. DINO can be applied also in this case to images then merge those feature into semantic GS. However, I didn't see such comparsion. There are also many different methods that work on distill feature from large model onto GS for various tasks.

  2. Better across view consistency does not establish better semantic understanding.

  3. I have have concerns about the scalability of the method. Training such model requires multi-view images, which is more difficult to get. Scaling things up to make it more general seems non-trivial.

Questions for authors
  1. The paper currently positions itself as a novel method in parallel to image based representation learning. Do the authors have any evidence on using the proposed method to augment existing image-based framework?

  2. Did the author compare the learned 3D semantic GS with semantic GS constructed from models like DINO?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5Vqh (Part 1)

We thank the reviewer for their constructive feedback and for highlighting our novel use of 3DGS as a geometric constraint, the semantic richness of our learned representations, and the generalizability of our trained encoder across scenes.

Response to Critical Weaknesses:

1. The concern is valid and completely understandable. We do not propose our method as a replacement for DINO-distilled features in scenarios that are in-distribution for DINO; rather, our method is advantageous over DINO in scenes that fall out-of-distribution for typical vision foundation models. We compare our method in Table 1 to FruitNeRF, which distills predictions from GroundingDINO + SAM, as well as Wheat3DGS, which utilizes a supervised YOLO model in combination with SAM for distillation. We did not compare with other methods that distill DINO representations into GS because we believe that this would not be a fair comparison, as DINO is pretrained on a massive corpus of images. However, your question (1) about augmenting existing frameworks is very intriguing. We hypothesize, based on results from prior works, such as Fit3D, and our results from fine-tuning an ImageNet-pretrained encoder using our DiFF framework, that our method could also be used for adapting models like DINO to new domains. To verify this hypothesis, we are currently running experiments to adapt DINO with our framework and will compare the results directly with a semantic GS constructed from DINO. We will update this open discussion with the results of these experiments.

2. We agree that multi-view consistency does not imply better semantics. Rather, we include the experimental results validating the multi-view consistency of our learned representations to demonstrate the potential advantages of learning semantics with geometric constraints.

3. While scaling our method to multi-scene pretraining requires dedicated engineering efforts, we believe that these efforts are feasible and worth investigating for future work. Although multi-view images are more difficult to collect than single images, any monocular video of a static scene provides multi-view supervision, and there is a massive effort in the 3D vision community to collect such datasets (e.g., DL3DV-10K, CO3D, ScanNet) for pretraining large 3D foundation models like VGGT. We designed our method to use COLMAP poses to be able to leverage these same datasets for potential scaling. We believe that there are three straightforward approaches to apply our method at scale. The first is to train individual scenes separately, as done in this work, and then distill all learned 3D representations into a single encoder. This approach is promising as it only requires 2D rendered feature maps during distillation, which avoids having to store multiple 3D feature fields in VRAM at once. Second, we could train the same encoder on various scenes sequentially. Finally, we could train multiple scenes in parallel with a single encoder. We believe that any of these approaches would result in a strong foundation model; however, experiments should be performed to determine which approach is optimal.

Response to Questions:

1. In Table 1, we have applied our DiFF framework to adapt an ImageNet-pretrained encoder to the Wheat3DGS dataset. This is the closest experiment we have performed to augment an existing image-based framework. As discussed in our response to critical weakness 1, we believe this is an interesting application for our method and are currently running experiments to evaluate the possibility of augmenting DINO.

2. We did not compare directly with a semantic GS constructed from DINO; however, we will compare a DiFF-augmented DINO with a semantic GS constructed from a frozen DINO.

AuthorsOpenReview ↗

Response to Reviewer 5Vqh (Part 2)

We are writing to follow up on our previous response to Weakness 1 and Questions 1 and 2 with the results of our new experiments comparing a semantic GS distilled from DINO (DINO-GS) against representations learned and adapted with DiFF on the Wheat3DGS dataset. For completeness, we also include results for a semantic GS distilled from a frozen ImageNet-pretrained ResNet50 (ImageNet-GS), as well as DiFF adaptation of both DINO and ImageNet backbones.

To construct a 16D semantic feature field from a frozen ResNet50 without modifying the encoder, we apply a learnable linear projection layer on the rendered 3D Gaussian features to match the frozen encoder embeddings during 3D optimization. Since the frozen encoders do not have a projection head, evaluating their 2D representations in the 1-shot 2D setting requires probing the backbone features directly. To compare on equal footing, we also probed the pre-projection features of our DiFF-trained encoders and the from-scratch LeJEPA baseline prior to the projection head.

Table 1: Multi-view segmentation on the Wheat3DGS dataset (mean across seven plots; † denotes models trained from scratch without external data, proj. denotes the 16D projection head, and enc. denotes direct probing of the 512D Layer 2 encoder features).

MethodSettingIoU (%)Prec. (%)Rec. (%)F1 (%)
FruitNeRFDefault34.095.035.050.0
Wheat3DGSDefault50.081.057.067.0
DINO-GS (Frozen)0-Shot 3D33.356.244.949.3
ImageNet-GS (Frozen)0-Shot 3D36.061.847.351.9
DiFF†0-Shot 3D30.857.147.046.9
DiFF_DINO0-Shot 3D37.571.043.453.6
DiFF_IN0-Shot 3D38.469.245.254.1
ImageNet-GS (Frozen)1-Shot 3D37.778.441.554.0
DINO-GS (Frozen)1-Shot 3D39.673.145.655.9
DiFF†1-Shot 3D40.257.858.256.8
DiFF_DINO1-Shot 3D45.766.259.262.2
DiFF_IN1-Shot 3D46.968.259.563.2
LeJEPA†1-Shot 2D (proj.)17.629.833.929.3
DiFF†1-Shot 2D (proj.)46.358.568.562.7
DiFF_DINO1-Shot 2D (proj.)52.465.171.768.0
DiFF_IN1-Shot 2D (proj.)53.565.972.268.6
LeJEPA†1-Shot 2D (enc.)46.761.964.863.2
DiFF†1-Shot 2D (enc.)54.668.073.470.4
DINO-R50 (Frozen)1-Shot 2D (enc.)56.771.072.671.7
ImageNet-R50 (Frozen)1-Shot 2D (enc.)58.271.075.373.0
DiFF_DINO1-Shot 2D (enc.)59.772.077.174.3
DiFF_IN1-Shot 2D (enc.)60.272.877.074.6

Table 2: Feature-level ablation for unidirectional DINO-GS distillation on the Wheat3DGS dataset (mean across seven plots).

MethodFeature LevelSettingIoU (%)Prec. (%)Rec. (%)F1 (%)
DINO-GS (Frozen)Layer 4 (2048D)0-Shot 3D26.549.436.041.3
DINO-GS (Frozen)Layer 2 (512D)0-Shot 3D33.356.244.949.3
DINO-GS (Frozen)Layer 4 (2048D)1-Shot 3D22.570.524.936.1
DINO-GS (Frozen)Layer 2 (512D)1-Shot 3D39.673.145.655.9
DINO (Frozen)Layer 4 (2048D)1-Shot 2D34.458.845.550.8
DINO (Frozen)Layer 2 (512D)1-Shot 2D56.771.072.671.7

These results clarify two points regarding your questions:

  1. Augmenting Existing Frameworks with DiFF (Question 1): Adapting either pretrained backbone with DiFF consistently improves both 3D and 2D segmentation over frozen baselines. In 1-shot 3D, DiFF_DINO and DiFF_IN improve over DINO-GS and ImageNet-GS by +6.1 and +9.2 percentage points in IoU, respectively (+4.2 and +2.4 points in 0-shot 3D). In 1-shot 2D (enc.), DiFF_DINO improves over frozen DINO-R50 by 3.0 percentage points in IoU and DiFF_IN improves over frozen ImageNet-R50 by 2.0 points, improving upon their frozen counterparts on 6 of the 7 plots (see Table 9 in Appendix A.5 for the per-plot breakdown) and achieving the highest overall accuracy at 60.2% IoU.

  2. Comparison with a Semantic GS Constructed from DINO (Question 2 and Weakness 1): As shown in Table 2, Layer 2 (512D, stride 8) features outperform coarser Layer 4 (2048D) features across every setting for DINO-GS because fine-grained wheat heads require higher spatial resolution. Even with Layer 2 features, however, unidirectional distillation exhibits substantial degradation when transferring from 2D to 3D because the 3D field passively averages view-inconsistent 2D features through a linear projection layer. Joint optimization via DiFF overcomes this drop: in 1-shot 3D segmentation, DiFF_DINO improves in IoU over DINO-GS by 6.1 percentage points, and even our from-scratch DiFF† surpasses both frozen DINO-GS and frozen ImageNet-GS.

We have incorporated these results and tables into Table 1, Section 5.2, and Appendix A.5 of the revised manuscript. Thank you again for suggesting these experiments; we believe the newly included results greatly strengthen our paper.

Reviewer_FykmOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SfS (Semantics from Structure), a self-supervised learning approach from multi-view images. SfS leverages 3D consistency between views to simultaneously train a 2D image encoder and a 3D feature field. To prevent feature collapse, SfS additionally uses a SIGReg Regularizer and a KNN smoothness term. Training on one scene allows zero-shot segmentation and one-shot classification.

Critical strengths

(1) The proposed method, SfS, is clean and paper writing is easy to understand.

(2) SfS achieves decent performance on zero-shot segmentation and 1-shot classfication.

Critical weaknesses

(1) This paper is not well motivated. The authors claim that the proposed approach is useful when the training data is scarce for certain domains. However, the results from Tab. 1 show that this issue can be solved by leveraging general-purpose ImageNet-pretrained encoders, which can already significantly boost performance. Therefore, it does not make sense to train the feature field and the 2D encoder from scratch.

(2) The proposed method is trained per scene and can barely generalize.

(3) The domain of the evaluation benchmark is narrow.

(4) There lacks a discussion about a body of work that conducts multi-view self-supervised learning via rendering. For example, RayZer and E-RayZer perform self-supervised camera pose prediction and reconstruction with photometric loss.

(5) The reviewer is curious about where is the term Joint-Embedding Architectures (JEAs) from in Section 2? Can authors list any reference to this term?

Questions for authors

Questions have been included in "Critical Weaknesses".

3 comments
AuthorsOpenReview ↗

Response to Reviewer Fykm (Part 1)

We thank the reviewer for their time and feedback, and we appreciate their positive comments on the clarity of our paper and our performance on zero-shot segmentation and 1-shot classification.

Response to Critical Weaknesses:

1. Our method is designed for use in domains that are out-of-distribution for typical vision foundation models. We demonstrate the advantage of our DiFF framework in the agricultural setting on the Wheat3DGS benchmark dataset, where our method outperforms FruitNeRF and performs competitively with Wheat3DGS. FruitNeRF directly applies foundation models (GroundingDINO and SAM) to create a semantic feature field, whereas Wheat3DGS utilizes a supervised YOLO model with SAM. Despite utilizing far less data and smaller encoders, our method remains competitive and even outperforms previous methods on recall.

We also want to clarify the experiment that uses the ImageNet-pretrained encoder. For this experiment, we did not freeze the encoder. Instead, we fine-tuned the pretrained encoder utilizing our DiFF framework and our full loss function (latent MSE, SIGReg, and KNN smoothness).

2. Although the method is trained per scene, our experiments demonstrate that the trained encoder maintains strong generalizability to new scenes and even different domains. Specifically, we retain 94.3% of the per-scene accuracy when applying the encoder to new scenes of the same domain and 89.7% of the accuracy on new domains (cross-dataset).

3. Our experiments span both complex outdoor agricultural scenes (Wheat3DGS) and indoor multi-object scenes (3D-OVS), evaluating both specialized out-of-distribution segmentation and cross-dataset generalization across distinct visual domains.

4. Our work targets unsupervised semantic feature field learning, whereas RayZer and E-RayZer target self-supervised 3D reconstruction. Our method results in an explicit 3D semantic representation, whereas the others do not. Furthermore, our 2D encoder is a single-image encoder that exhibits multi-view consistency as opposed to RayZer and E-RayZer, which utilize a multi-view encoder. We believe that methods such as RayZer are complementary to our method, as they can be used in place of COLMAP to generate the initial 3D representation for 3DGS training and bidirectional distillation.

5. The term Joint-Embedding Architecture (JEA) was first introduced by Yann LeCun in his position paper, “A path towards autonomous machine intelligence” [1]. This paper presents a diagram of general JEA frameworks in Figure 10. Additionally, the I-JEPA paper, “Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture” [2], also includes a similar diagram in Figure 2a. The term was introduced to give a general description of architectures that jointly optimize two encoders against each other’s representations.

References

[1] Yann LeCun. A path towards autonomous machine intelligence version 0.9.2, 2022-06-27. Open Review, 62(1):1–62, 2022.
[2] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15619–15629, 2023.

AuthorsOpenReview ↗

Response to Reviewer Fykm (Part 2)

We are writing to follow up on our response to Weakness 1 with new experimental results on the Wheat3DGS dataset comparing our bidirectional framework against unidirectional distillation from frozen pretrained encoders (ImageNet-GS and DINO-GS), as well as direct backbone probing and the use of DiFF for adaptation of DINO and ImageNet encoders. The full results tables (Tables 1 and 2) have been posted in our follow-up response to Reviewer 5Vqh and added to the revised manuscript (Table 1, Section 5.2, and Appendix A.5).

These results reinforce our response to Weakness 1 in two ways:

  1. From-Scratch Performance Against Pretrained and Supervised Baselines (Weakness 1): Directly probing the pre-projection backbone features raises the from-scratch DiFF† 1-shot 2D performance by 8.3 percentage points in IoU over the 16D projection head (reaching 54.6% IoU), outperforming the from-scratch 2D-only LeJEPA† baseline across all seven plots by 7.9 percentage points in IoU and surpassing the supervised Wheat3DGS baseline by 4.6 percentage points without any external pretraining data. Moreover, in 1-shot 3D segmentation, our from-scratch DiFF† surpasses both frozen ImageNet-GS and frozen DINO-GS.

  2. Adapting Pretrained Encoders Over Frozen Distillation (Weakness 1): DiFF also improves pretrained encoders over frozen distillation baselines. Adapting ImageNet and DINO encoders with DiFF improves IoU on 1-shot 3D segmentation over frozen ImageNet-GS and DINO-GS by 9.2 and 6.1 percentage points, respectively (+2.4 and +4.2 percentage points in 0-shot 3D), and improves IoU on 1-shot 2D segmentation over frozen ImageNet-R50 and DINO-R50 by 2.0 and 3.0 percentage points, reaching 60.2% IoU overall.

Dear authors,

Thanks for the response. The comparison between frozen VFM and fine-tuned VFM partially solved my concerns. However, similar to Reviewer gCMc, I am still concerned about the scope of the paper, and not sure if the proposed method generalizes well across other domains, and in which domains we care have this specific data scarcity challenging, which cannot be readily solved by foundation models. Therefore, I decoded to raise my score to 2.

Best, Reviewer

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewers for their thoughtful feedback and constructive suggestions. We are encouraged that the reviewers recognized the novelty of using 3D Gaussian splatting as an explicit geometric constraint for unsupervised representation learning, its strong cross-view consistency, and its potential for scaling to large-scale visual representation learning.

We have uploaded a revised version of our manuscript incorporating the clarifications and new experimental results discussed during the rebuttal period. Below is a summary of the changes made to the manuscript:

  1. Clarified 2D Linear Probing Protocol (Section 4): Under "1-Shot 2D Feature Matching", we added a sentence to clarify that 2D features can be probed either from the projection head or directly from the pre-projection backbone features, consistent with standard self-supervised linear evaluation protocols and enabling direct comparison against frozen pretrained encoders that lack a 16D projection head.

  2. Added Unidirectional Distillation Baselines, DINO Adaptation, and Direct Encoder Probing (Section 5.2 and Table 1): In response to Reviewer 5Vqh (Weakness 1, Questions 1 and 2), Reviewer gCMc (Weaknesses 1 and 2), and Reviewer Fykm (Weakness 1), we expanded Table 1 into a standalone table and updated Section 5.2 with the new experimental results on the Wheat3DGS dataset.

  3. Reorganized Cross-Scene and Cross-Dataset Generalization into the Main Text (Section 5.3, Tables 2 and 3): In response to Reviewer Fykm (Weakness 2), we moved the cross-dataset transfer experiment (3DCotton to Wheat3DGS, Table 3) and its accompanying discussion from the appendix into Section 5.3 alongside the cross-scene transfer results (Table 2) so both generalization evaluations appear together in the main paper.

  4. Added Feature-Level Ablation and Per-Plot Probing Breakdown (Appendix A.5, Tables 8 and 9): We added a new appendix section A.5 detailing the learnable linear projection layer used to match rendered 16D Gaussian features to frozen encoder embeddings in DINO-GS and ImageNet-GS. This section includes Table 8, which compares Layer 2 (512D, stride 8) against Layer 4 (2048D) features for DINO-GS across all evaluation methods, and Table 9, which provides the per-plot 1-shot 2D comparison under direct backbone probing across all six models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.