- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces VINO, a self-supervised representation learning framework for 3D scenes based on teacher–student self-distillation. The authors systematically revisit the current 3D SSL recipe (e.g., Sonata) and propose four key design improvements: (1) adopting the isotropic Volume Transformer (Volt) to replace hierarchical backbones; (2) using asymmetric augmentations (clean global teacher vs. corrupted local student) with an independently learned prototype target head; (3) eliminating geometric shortcuts in masked modeling by removing masked tokens from the encoder and reconstructing them via a cross-attention decoder; and (4) performing view-consistent 2D-to-3D distillation by clustering in the frozen 2D VFM feature space. Evaluated on six indoor and outdoor benchmarks, VINO sets new state-of-the-art results on frozen linear probing and fine-tuning, while demonstrating strong out-of-domain transfer on zero-shot registration and single-view RGB-D probing.
Critical strengths
S1. The systematic redesign of 3D self-distillation is well-motivated and effective. Rather than proposing a single isolated heuristic, the paper carefully identifies key limitations in existing 3D self-distillation frameworks (e.g., Sonata) and addresses each with a principled solution. The step-by-step ablations cleanly isolate the orthogonal contribution of each design choice on ScanNet200. This cumulative improvement convincingly demonstrates that these design choices are complementary and form a cohesive, well-engineered framework.
S2. Strong out-of-domain transfer validates the general-purpose representation quality. Beyond standard in-domain semantic segmentation, VINO's frozen features transfer remarkably well to unseen tasks and domains: achieving competitive performance on cross-modal probing. These results substantiate that the learned representations capture intrinsic geometric correspondence and view-invariant semantics rather than merely overfitting to segmentation-specific priors.
Critical weaknesses
W1. The architectural contribution of the backbone is partially confounded with the algorithmic redesign. In Table 5, VINO (Volt-B, 85.7M) is compared against competing methods predominantly based on the hierarchical PTv3-B backbone . While Table 1 shows that replacing PTv3 with Volt under the Sonata baseline already yields a +1.3 mIoU gain, the full benchmark comparisons do not isolate how much of the final mIoU improvement over Sonata stems from the stronger Volt backbone itself versus the proposed self-distillation objectives. To cleanly disentangle these factors, the paper should ideally provide either: (a) competing methods (e.g., Sonata, Concerto) ported to the Volt backbone, or (b) VINO implemented on a PTv3 backbone. Without these controlled comparisons, it remains difficult to assess whether VINO's improvements over prior SOTA are driven primarily by the algorithmic innovations or the shift in backbone architecture.
W2. Outdoor performance gains are notably modest and lack in-depth analysis. While VINO achieves substantial improvements on indoor benchmarks, its gains on outdoor LiDAR datasets are significantly less pronounced. Under 3D-only pretraining (Table 5), VINO's 65.3 mIoU on Waymo falls 1.7 mIoU short of DOS (67.0), and its 68.5 mIoU on SemanticKITTI is only marginally higher than DOS (68.3). Since separate models are trained for indoor and outdoor settings, this performance disparity is striking. The paper lacks a dedicated discussion analyzing whether this plateau is due to the inherent sparsity of LiDAR point clouds, the absence of RGB features outdoors, or lower camera-LiDAR overlap in driving datasets. Providing this analysis is crucial for understanding the boundaries and failure modes of the framework.
Questions for authors
Q1. Disentangling Backbone vs. Algorithmic Gains: In Table 5, VINO (Volt-B) achieves impressive state-of-the-art results over prior methods (e.g., Concerto, Utonia, DOS), yet all competing approaches rely on the hierarchical PTv3 architecture. Given that Table 1 already shows a +1.3 mIoU gain on Sonata purely from switching the backbone (28.3 → 29.6), it is unclear how much of the final benchmark advantage comes from Volt's intrinsic inductive bias versus the proposed distillation objectives.
- Could the authors provide a controlled comparison to disentangle these factors, such as: (a) porting a baseline method (e.g., Sonata or Concerto) to the Volt backbone, or (b) training VINO with a PTv3 backbone?
- If running a full pretraining suite is computationally prohibitive during the rebuttal, a controlled ablation on a single representative benchmark (e.g., ScanNet200 linear probing) under the same compute budget would be sufficient to clarify this attribution.
Q2. Diagnosis of Modest Outdoor Performance and Scaling Limits:
- Since indoor and outdoor models are pretrained independently, what explains this noticeable performance disparity?
2 comments
Thank you for the detailed rebuttal. I will maintain my current positive rating.
Rebuttal Response to 9WZ1
We thank the reviewer for the detailed review and constructive feedback.
W1 / Q1: Comparison under the same backbone. The paper already contains two controlled comparisons on Volt-B. First, Tabs. 1 and 11 report Sonata trained with both PTv3 and Volt-B. Swapping the backbone alone improves Sonata by +1.3 mIoU on ScanNet200 and +1.1 on ScanNet++. On the same Volt-B backbone, our 3D-only objectives reach 35.9 and 42.4, which is +6.3 and +3.3 mIoU over Sonata (Volt). This shows that most of the improvement comes from our redesigned objectives rather than from the backbone. Second, Concerto combines Sonata’s 3D self-distillation with a 2D–3D distillation loss. Their 2D–3D distillation relies on two main design choices: (i) aligning with 2D features from the start of training, and (ii) cosine alignment to raw DINOv2 features. In our 2D–3D ablations (Tabs. 4 and 11), we use these two choices as the baseline, applied to Volt-B on top of our 3D objectives. We then revisit both: we distill in a second stage after 3D-only pretraining, and we use multi-view-consistent prototype assignments as targets instead of raw DINOv2 features. Each change improves performance, and together they outperform the baseline by +2.6 mIoU on ScanNet200 and +3.8 on ScanNet++. This shows the benefit of our approach over Concerto’s key design choices under fair conditions with the Volt-B backbone.
For a complete same-backbone comparison, we additionally trained Concerto on Volt-B with the same pretraining data and the same GPU-hour budget as VINO.
Under the same backbone, pretraining data, and GPU-hour budget, VINO outperforms our Concerto reproduction by +2.7 mIoU on ScanNet200, +0.9 on ScanNet, and +5.0 on ScanNet++. VINO also exceeds Concerto's published PTv3-B results on all three benchmarks (+2.4, +0.6, and +3.0 mIoU, respectively).
Our Concerto (Volt-B) reproduction closely matches the published Concerto (PTv3-B) results on ScanNet200 and ScanNet (-0.3 mIoU each), but performs 2.0 mIoU lower on ScanNet++. This gap may arise from differences in the backbone and pretraining data. First, Concerto includes synthetic Structured3D scenes, which are excluded from VINO's training data. Second, Concerto was designed around PTv3's multi-resolution hierarchy, applying its 3D self-distillation and 2D–3D alignment losses at different feature levels. Concerto reports that applying both losses at the same level reduces performance (Tab. 6f). Since Volt lacks this hierarchy, both losses operate on the same output representations in our adaptation. Thus, Volt may be less suitable for Concerto's original training formulation, although we cannot isolate the contributions of these differences.
W2 / Q2: Outdoor performance. Please see the general response for updated outdoor results. With the updated settings, VINO achieves the best 3D-only linear-probing results on all three outdoor benchmarks, including Waymo. Under fine-tuning, VINO is best on nuScenes and SemanticKITTI but remains 1.0 mIoU below DOS on Waymo. We do not have a conclusive explanation for this gap. We added new results to the paper.