acceptodds
Under review as a conference paper at ICLR 2027

Geometric Similarity in VLM Low-Level Vision Representations

Abstract

Vision-language models (VLMs) have emerged as powerful candidates for universal vision backbones, with representative architectures including autoregressive (AR) models and diffusion transformers (DiTs). Yet, adapting them efficiently for all-in-one low-level image restoration remains a challenge. Crucially, the field lacks an understanding of how VLMs organize hidden-layer representations and whether these structurally distinct paradigms share a common geometric organization for pixel-level perception. Such shared organization is a prerequisite for building highly transferable, unified restoration VLMs and adapters. In this paper, we systematically investigate representational similarity across 24 low-level tasks spanning 5 categories. We propose GeoSim, a unified four-level framework that analyzes task-conditioned representations from global similarity, local geometry, sparse feature decomposition, and topological verification perspectives. Our formulation applies to the analysis of hidden states in AR models and feature maps in DiTs across same- and cross-task/model settings. Our results reveal the organizing principles of low-level visual representations while exposing their limits in cross-task and cross-model agreement. Ultimately, GeoSim provides an interpretability lens for probing latent transferability in low-level vision and diagnosing model limitations in task- or model-specific scenarios.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Geometric Similarity, Vision-Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qKhmOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the geometric organization of low-level visual representations in vision-language models (VLMs), with a particular focus on task relationships, representation sharing, and cross-model universality. The authors propose GeoSim, a unified four-level analytical framework that examines global representation similarity, local geometric structure, sparse feature decomposition, and topological consistency. The study covers 24 low-level vision tasks across five perceptual categories and seven VLMs spanning different architectural paradigms, including autoregressive models and diffusion transformers. It provides a comprehensive empirical characterization of low-level visual representations in VLMs, highlighting both the existence and limitations of shared geometric structures and offering insights that may inform future research on task transferability and unified low-level vision models.

Critical strengths
  1. The paper introduces GeoSim, a systematic framework that integrates global similarity, local geometry, sparse feature decomposition, and topological analysis to investigate low-level visual representations in VLMs. These complementary perspectives enable a more nuanced characterization of representational similarity than relying on any single metric.
  2. The paper conducts a systematic evaluation across 24 low-level vision tasks, five task families, and seven VLMs spanning different architectural paradigms, and the results reveal several non-obvious findings such as different VLMs exhibit substantial global representational similarity despite relatively weak local geometric alignment, while cross-task relationships are more strongly associated with shared visual processing mechanisms than with predefined task categories.
Critical weaknesses
  1. The proposed framework primarily analyzes instruction-induced representation differences between task-specific and neutral prompts, using last-token hidden states for autoregressive models and spatially aggregated condition-image features for the diffusion-based model. However, it remains unclear whether the resulting geometric structures reflect genuine low-level visual processing mechanisms or are largely driven by the linguistic semantics of task instructions. Although neutral-prompt subtraction removes some shared components, it does not necessarily disentangle language-induced task representations from image-dependent visual computations. Additional controls, such as image-independent baselines, prompt paraphrasing, and alternative visual-token representations, would help clarify this distinction. Moreover, vision-only generalist models such as Painter[1] and LVM[2] could provide complementary evidence for whether similar task structures emerge without linguistic supervision or conditioning.

  2. While the paper provides a comprehensive characterization of geometric similarities among low-level vision tasks, the practical relationship between these similarities and actual task transferability remains unexplored. The authors motivate GeoSim partly through its potential implications for knowledge transfer, parameter sharing, and unified image restoration models. However, the experiments primarily characterize representational geometry and do not establish whether geometrically similar tasks benefit more from joint training or knowledge transfer. A controlled experiment evaluating the relationship between GeoSim similarity and downstream transfer performance, even on a representative subset of task pairs, would substantially strengthen the practical significance of the findings.

[1] Images Speak in Images: A Generalist Painter for In-Context Visual Learning, CVPR 2023

[2] Sequential Modeling Enables Scalable Learning for Large Vision Models, CVPR 2024

Questions for authors
  1. The proposed GeoSim framework comprises four analytical levels: Global Similarity Structure (Level 1), Local Geometric Structure (Level 2), Sparse Feature Decomposition (Level 3), and Topological Verification (Level 4). While these components provide complementary perspectives on representation geometry, the rationale for organizing them into a hierarchy of "levels" is not entirely clear. Could the authors clarify whether this ordering reflects a conceptual progression, methodological dependency, or increasing analytical depth? If the four components are largely complementary rather than hierarchically dependent, would describing them as four analytical perspectives or dimensions more accurately reflect the framework's structure?
Reviewer_FMBcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on the geometric characteristics of the internal representation of VLMs in processing all-in-one low-level image restoration tasks, and proposes an analysis framework, GeoSim. This framework includes four-levels of analysis, namely global similarity, local geometry, sparse feature decomposition, and topological verification perspectives. The paper systematically explores the representation similarity across multiple models across 24 low-level tasks spanning 5 categories.

Critical strengths
  1. The research question is relevant and comparatively underexplored. Examining low-level task representations across multiple models could complement existing studies of semantic alignment and provide useful evidence for understanding unified visual systems.
  2. The paper conducted thorough experiments on multiple models and tasks, demonstrating in detail the geometric characteristics of VLM hidden representations in low-level image restoration tasks.
Critical weaknesses
  1. The metrics used in the four-level analysis framework designed in the paper are all existing indicators, and the paper only divided them into four groups and conducted experiments. Lack of analysis on the relationship between the analysis results of the four levels and the ability to solve low-level image restoration tasks in VLM.
  2. The experimental results and analysis on multiple models and datasets do not effectively answer the research questions raised in the Introduction, and form conclusions that have reference value for the development of the field.
  3. The seven models include both understanding-oriented VLMs (InternVL3.5, Qwen3-VL) and generation-oriented models (Qwen-Image-Edit, Emu3.5-Image, Janus-Pro, Anole). It is not clear whether the same layer, token position, and pooling strategy are meaningful across such different architectures.
  4. The paper cites RSA, SVCCA, CKA, the Platonic representation hypothesis, and SAE interpretability work, but does not clearly state what is conceptually new in GeoSim beyond combining existing metrics.
Questions for authors

What is the most important actionable insight?

Reviewer_aBL8OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces GeoSim, a four-level framework for analyzing representation geometry across 24 low-level vision tasks and 7 models, covering global similarity, local geometry, sparse features, and topology. The results reveal substantial global geometric consistency across models but limited agreement in local neighborhoods and sparse representations.

Critical strengths

1. The paper investigates representation geometry across low-level vision tasks and model architectures, addressing a meaningful research question.

2. The experiments cover 24 tasks and 7 models, combining multiple geometric and sparse feature analyses to provide broad empirical evidence.

3. The findings highlight the gap between global representation similarity and local geometric or sparse feature agreement, offering insights into the limitations of cross-model representation sharing.

Critical weaknesses

1. Insufficient control of task and dataset effects. Each low-level vision task is associated with a specific dataset, confounding task differences with image content, data distributions, and textual instructions. The experiments lack controlled comparisons using matched images and task prompts, leaving task effects indistinguishable from dataset effects.

2. Unclear definitions and interpretations of key metrics. Eq (1) essentially computes the inner product of mean normalized task representations, which does not fully characterize representation geometry. The instruction-induced displacement in Fig 2 lacks an explicit mathematical definition. Additionally, Eq (2) includes self-similarity terms, contradicting the description of distinct image pairs.

3. Lack of direct validation linking geometric similarity to shared task mechanisms. dCor measures statistical dependence between representations without distinguishing shared image information from task-specific computations. SAE-MMD compares activation-rate distributions without establishing feature correspondence across models. The absence of task transfer and feature intervention experiments leaves the claims about shared visual processing mechanisms unsupported by direct evidence.

4. Insufficient statistical analysis of task-family relationships. Fig 5 reports 36 task relations, including 16 cross-family edges, without adequately accounting for differences in the number of tasks across families. Furthermore, Figs 5 and 6 use different metrics to analyze task-family relationships, but the paper does not sufficiently explain how these findings jointly support the proposed mechanism-centric task organization.

Questions for authors
  1. How do the authors disentangle task effects from image content, dataset distributions, and textual prompts? Can controlled experiments using matched images be provided?

  2. How is the instruction-induced displacement in Fig 2 defined? How does Eq 1 capture overall geometry?

  3. Can the authors validate the connection between geometric similarity and shared visual processing mechanisms through task transfer or SAE feature intervention experiments?

  4. How does Fig 5 account for task-family size differences? How do the authors reconcile the findings in Figs 5 and 6 to support mechanism-centric task organization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.