- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a unified frozen-feature probing framework to compare VLMs and VGMs across semantic tagging, instance grouping, and 3D geometry prediction. The experiments show a consistent result that VLMs are stronger in semantic and instance-level understanding, while VGMs are stronger in geometry. A simple feature-level fusion further combines these complementary strengths and improves downstream embodied tracking performance.
Critical strengths
The main strength of this paper is that it puts VLMs and VGMs into a unified frozen-feature probing framework and evaluates them on semantic tagging, instance grouping, and 3D geometry prediction. The conclusion is stable across models, where VLMs are stronger in semantics and instance-level understanding, while VGMs are stronger in geometry.
Critical weaknesses
-
The main conclusion itself is less suprising to us. VGMs need to model cross-frame and spatiotemporal features, so it is not surprising that geometric information is more encoded in the models. VLMs are explicitly trained with language and semantic supervision, so stronger semantic representations are expected.
-
The fusion strategy is straightforward, where two frozen features are normalized independently, concatenated, and then fed into the probe backbone. The improvement shows that the two feature sources are complementary and can coexist in the downstream probe. However, this might not be viewed as a major technical contribution.
-
The input features of VLMs and VGMs are not fully aligned. VGM takes the 76-frame input, while VLM only takes 20 frames, and the spatial feature grids are also different. It might be a bit hard to attribute the performance gap to the pretraining paradigm difference. Part of the gap may come from having access to different features or spatial/temporal resolutions. This weakens the major claim of the paper.
Questions for authors
Please see Critical Weaknesses.