- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the geometric organization of low-level visual representations in vision-language models (VLMs), with a particular focus on task relationships, representation sharing, and cross-model universality. The authors propose GeoSim, a unified four-level analytical framework that examines global representation similarity, local geometric structure, sparse feature decomposition, and topological consistency. The study covers 24 low-level vision tasks across five perceptual categories and seven VLMs spanning different architectural paradigms, including autoregressive models and diffusion transformers. It provides a comprehensive empirical characterization of low-level visual representations in VLMs, highlighting both the existence and limitations of shared geometric structures and offering insights that may inform future research on task transferability and unified low-level vision models.
Critical strengths
- The paper introduces GeoSim, a systematic framework that integrates global similarity, local geometry, sparse feature decomposition, and topological analysis to investigate low-level visual representations in VLMs. These complementary perspectives enable a more nuanced characterization of representational similarity than relying on any single metric.
- The paper conducts a systematic evaluation across 24 low-level vision tasks, five task families, and seven VLMs spanning different architectural paradigms, and the results reveal several non-obvious findings such as different VLMs exhibit substantial global representational similarity despite relatively weak local geometric alignment, while cross-task relationships are more strongly associated with shared visual processing mechanisms than with predefined task categories.
Critical weaknesses
-
The proposed framework primarily analyzes instruction-induced representation differences between task-specific and neutral prompts, using last-token hidden states for autoregressive models and spatially aggregated condition-image features for the diffusion-based model. However, it remains unclear whether the resulting geometric structures reflect genuine low-level visual processing mechanisms or are largely driven by the linguistic semantics of task instructions. Although neutral-prompt subtraction removes some shared components, it does not necessarily disentangle language-induced task representations from image-dependent visual computations. Additional controls, such as image-independent baselines, prompt paraphrasing, and alternative visual-token representations, would help clarify this distinction. Moreover, vision-only generalist models such as Painter[1] and LVM[2] could provide complementary evidence for whether similar task structures emerge without linguistic supervision or conditioning.
-
While the paper provides a comprehensive characterization of geometric similarities among low-level vision tasks, the practical relationship between these similarities and actual task transferability remains unexplored. The authors motivate GeoSim partly through its potential implications for knowledge transfer, parameter sharing, and unified image restoration models. However, the experiments primarily characterize representational geometry and do not establish whether geometrically similar tasks benefit more from joint training or knowledge transfer. A controlled experiment evaluating the relationship between GeoSim similarity and downstream transfer performance, even on a representative subset of task pairs, would substantially strengthen the practical significance of the findings.
[1] Images Speak in Images: A Generalist Painter for In-Context Visual Learning, CVPR 2023
[2] Sequential Modeling Enables Scalable Learning for Large Vision Models, CVPR 2024
Questions for authors
- The proposed GeoSim framework comprises four analytical levels: Global Similarity Structure (Level 1), Local Geometric Structure (Level 2), Sparse Feature Decomposition (Level 3), and Topological Verification (Level 4). While these components provide complementary perspectives on representation geometry, the rationale for organizing them into a hierarchy of "levels" is not entirely clear. Could the authors clarify whether this ordering reflects a conceptual progression, methodological dependency, or increasing analytical depth? If the four components are largely complementary rather than hierarchically dependent, would describing them as four analytical perspectives or dimensions more accurately reflect the framework's structure?