acceptodds
Withdrawn from ICLR 2027

Which Pretraining Paradigm Better Serves Spatial Intelligence? An Empirical Comparison of Vision-Language and Video Generation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial intelligence requires visual representations that capture both semantic objects and geometric structure in the physical world. To support this, two major pre-training schemes are now widely used as foundation backbones: Vision-Language Models (VLMs), which use language supervision to align visual observations with semantic concepts, and Video Generation Models (VGMs), which learn from temporally evolving visual worlds. However, it still remains unclear which pre-training scheme provides a better representation substrate for spatial intelligence. In this paper, we present the first systematic frozen-feature probing study of VLMs and VGMs across three representative axes of spatial intelligence: semantic tagging, instance grouping, and 3D geometry prediction. Using the lightweight probe, our framework enables a controlled comparison of what information is already encoded in frozen representations from two model families. Experimental results reveal a clear complementarity: VLMs are stronger at semantic tagging and instance grouping, while VGMs provide more accessible signals for dense geometry and camera motion. Moreover, a naive fusion of the two already yields a representation that excels at both geometry and semantics, suggesting a promising direction for building stronger spatial-intelligence backbones by effectively integrating features from both model families.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Video Generation, Video Foundation Models, Frozen Feature Probing, Spatial Intelligence, 3D Scene Understanding, Multimodal Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GCREOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a unified frozen-feature probing framework to compare VLMs and VGMs across semantic tagging, instance grouping, and 3D geometry prediction. The experiments show a consistent result that VLMs are stronger in semantic and instance-level understanding, while VGMs are stronger in geometry. A simple feature-level fusion further combines these complementary strengths and improves downstream embodied tracking performance.

Critical strengths

The main strength of this paper is that it puts VLMs and VGMs into a unified frozen-feature probing framework and evaluates them on semantic tagging, instance grouping, and 3D geometry prediction. The conclusion is stable across models, where VLMs are stronger in semantics and instance-level understanding, while VGMs are stronger in geometry.

Critical weaknesses
  1. The main conclusion itself is less suprising to us. VGMs need to model cross-frame and spatiotemporal features, so it is not surprising that geometric information is more encoded in the models. VLMs are explicitly trained with language and semantic supervision, so stronger semantic representations are expected.

  2. The fusion strategy is straightforward, where two frozen features are normalized independently, concatenated, and then fed into the probe backbone. The improvement shows that the two feature sources are complementary and can coexist in the downstream probe. However, this might not be viewed as a major technical contribution.

  3. The input features of VLMs and VGMs are not fully aligned. VGM takes the 76-frame input, while VLM only takes 20 frames, and the spatial feature grids are also different. It might be a bit hard to attribute the performance gap to the pretraining paradigm difference. Part of the gap may come from having access to different features or spatial/temporal resolutions. This weakens the major claim of the paper.

Questions for authors

Please see Critical Weaknesses.

Reviewer_NGBaOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a systematic empirical comparison of Vision-Language Models (VLMs) and Video Generation Models (VGMs), investigating the differences in spatial information encoded in their pretrained representations. Through a unified frozen-feature probing framework, the authors find that VLMs favor object semantics and instance-level understanding, whereas VGMs provide stronger geometric representations, particularly for depth and camera pose estimation. Furthermore, feature fusion experiments demonstrate the complementary strengths of these two representations, with improvements in both spatial probing tasks and downstream closed-loop embodied tracking.

Critical strengths
    1. Well-motivated and straightforward experimental design
      Given the different pretraining objectives of VLMs and VGMs, it is reasonable to expect their learned representations to emphasize different types of information. The paper adopts a unified frozen-feature probing framework to directly investigate these differences, revealing complementary strengths in semantic understanding and geometric representation. This provides a clear and meaningful empirical insight into spatial representations learned through different pretraining paradigms.
    1. Practical validation through downstream embodied tasks
      A particular strength of the paper is its feature fusion experiment on closed-loop embodied tracking. Without such downstream validation, the findings would primarily establish differences in representation characteristics, with limited evidence of their relevance to spatial intelligence. By demonstrating that combining VLM and VGM features improves embodied task performance, the paper provides meaningful evidence of their practical complementarity and potential value for spatial intelligence applications.
Critical weaknesses
    1. Insufficient control of confounding factors in the pretraining paradigm comparison
      Although the unified probing framework provides a reasonable basis for comparison, the evaluated VLMs and VGMs differ in model architectures, training data, temporal inputs, and feature extraction strategies. Therefore, it remains unclear to what extent the observed differences in semantic and geometric capabilities can be attributed specifically to their pretraining paradigms rather than these confounding factors. This weakens the causal interpretation of the paper's central findings.
    1. Limited ablations to establish the source of feature fusion improvements
      While the downstream embodied experiments demonstrate promising improvements, it remains unclear to what extent these gains arise from the complementary semantic and geometric information encoded by VLMs and VGMs, rather than simply combining additional visual features. Additional comparisons, such as same-paradigm fusion or dimension-matched baselines, would help better understand the source of these improvements and further strengthen the paper's findings.
Questions for authors
    1. Pretraining Paradigm Comparison
      Given the differences in model architectures, training data, temporal inputs, and feature extraction strategies between VLMs and VGMs, could the authors clarify how these factors might influence the observed differences in semantic and geometric capabilities? What evidence supports attributing these differences primarily to the pretraining paradigms?
    1. Understanding Feature Fusion Improvements
      The downstream embodied experiments demonstrate promising improvements through VLM–VGM feature fusion. Could the authors further clarify whether these gains primarily arise from the complementary semantic and geometric information, or from the additional features available to the downstream model? Comparisons with same-paradigm fusion (e.g., VLM+VLM or VGM+VGM) or dimension-matched baselines would help strengthen this interpretation.
Reviewer_7UPMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper compares frozen features from 18 vision-language and video generation models on semantic tagging, instance grouping, and 3D geometry prediction. VLMs perform better on semantic and instance tasks, while VGMs perform better on geometry prediction using VGGT-generated references. Concatenating features from the two families retains their respective strengths and improves person-following success from 33.75% to 47.5%.

Critical strengths
  1. The paper asks an interesting question: how do VLM and VGM differ in the spatial information they encode? Comparing these two models across semantics, instances, and geometry offers a useful perspective on their strengths and potential complementarity.

  2. The study tests 18 model checkpoints across several families and model sizes. This coverage helps show whether the observed differences extend across models within each family.

Critical weaknesses
  1. The fusion part is underdeveloped. The authors normalize and concatenate the features, but give limited detail on how the features are aligned or why this fusion strategy was chosen. A clearer description and a comparison with other fusion methods would make this part more convincing.

  2. The benefits of fusion are uneven. It mostly matches the stronger individual model on geometry and slightly reduces instance-grouping performance. The downstream evaluation is limited to a single person-following benchmark. It remains unclear whether the benefit extends to other tasks.

Questions for authors
  1. How are the 76-frame context, the 81-frame WAN extraction window, and the 20 query positions aligned?

2.Have author checked whether the geometric ranking holds when using ground-truth annotations independent of VGGT?

  1. Have you considered other ways to combine the models’ complementary strengths, such as using one model’s features as a supervision signal for the other?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.