acceptodds
Withdrawn from ICLR 2027

Equal State Fit, Different Finite Responses in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cross-model steering uses maps between hidden states to reuse activation interventions in another model. We ask whether fitting paired states determines the effect of a mapped intervention on held-out target inputs. In an eight-dimensional residual subspace, four paired examples constrain at most six dimensions, leaving maps with the same calibration fit and target update norm that can produce different finite responses. We measure this difference by the response diameter, the range of target responses across these maps. Across three target vision-language models, 123 of 249 controlled groups have scalar diameters above a conservative numerical threshold. The separation recurs on image-disjoint Visual Question Answering v2 directions, including 19/24 Qwen-to-IDEFICS2 and 16/24 LLaVA-to-IDEFICS2 groups, and on a fixed Multimodal Visual Patterns subset spanning eight of nine visual patterns. All evaluable natural responses remain stable when the intervention step is halved, with further evidence from fixed angle grids and a second residual site. These results distinguish the role of state calibration from target-response evaluation: calibration fixes the map on the observed span, while updates outside that span require direct comparison at matched target norms.

Primary area:
interpretability and explainable AI
Keywords:
Vision-Language Models, Cross-model Steering, Mechanistic Interpretability, Representation Alignment, Activation Intervention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qAHFOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper explores cross-model steering in VLMs. The authors explore whether state fit determines what the mapped intervention does to the target model on held-out inputs. In the paper, they use three models from different backbones, Qwen2.5VL, IDEFICS2, LLaVA 1.5 with the controllable experiments with VQAv2 and MMVP. They show that in an 8-dimensional residual subspace, four calibration pairs constrain the map on at most six dimensions. The authors introduce a metric, response diameter, is the spread of the target's finite responses across these maps. They argue that mapping accuracy fixes the map only on the calibrated span, so mapped interventions must be evaluated directly on held-out target responses at matched update norms.

Critical strengths
  1. Cross-model steering work usually validates a map by its low mapping error and then tests one selected map on the downstream tasks. The paper frames the issue clearly: what a low mapping error identifies is different from what held-out target responses establish.
  2. An interpretable theory formulation. Proposition 1 gives a simple rank upper bound, which the measured designs attain. Proposition 2 constructs completions with identical mapping error, operator norm and update norm. Corollary 1 gives the response diameter a decision-theoretic meaning.
  3. Controllable experiments with three VLM families, and two datasets.
Critical weaknesses
  • Limited conceptual novelty. Equal mapping error with different held-out behavior is related to underspecification [0].
  • No exploration on high dimensions. All the experiments is on 8-dimensional subspace. In practical, the experiments usually use enough data to do the mapping. For example over 100k in [1], and over 5k image-text pairs [3]. There, this ambiguity largely disappears. Although the authors acknowledge the restriction to linear maps in an 8 dimensional subspace. The paper does not test the realistic cases where its concern would bite: full-dimensional maps fitted on fewer paired samples than dimensions, such as least-squares maps on roughly 100–1k samples in ~4k-dimensional spaces;
  • The experiments setup is not clearly define. The paper doesn’t give the reason for intervening at 55% depth.
  • The results don’t have a clear explaination. Qwen2.5-VL-7B resolves only 3/83 controlled groups and 0/24 on both natural edges, but the paper does not explain why. The intervention may simply be too small to register in Qwen. A model-relative update norm or a sweep over update norms is needed to rule this out.
  • There is no explain why choose 8-dimensional compression. The subspace keeps only 47–64% of the energy of the paired-image differences. The paper does not report how much of each donor direction survives the projection. Since r = 8 also determines the size of the unconstrained complement, an ablation over r is needed.

[0] D’Amour, Alexander, et al. "Underspecification presents challenges for credibility in modern machine learning." 2022 [1] Chen, Alan, et al. "Transferring linear features across language models with model stitching." 2025 [2] Li, Jiaang, et al. "Do vision and language models share concepts? a vector space alignment study." 2024.

Questions for authors

See weaknesses.

Reviewer_RECFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether matching the internal representations of two vision-language models (VLMs) is sufficient to transfer activation interventions from one model to another. The authors show that different mappings between models can fit the same calibration data equally well but still produce different responses when applied to new inputs. They provide a mathematical explanation for this problem and introduce a measure called response diameter to quantify the differences. The experiments use Qwen, LLaVA, and IDEFICS2 models, with evaluations on controlled examples and natural-image datasets, including VQAv2 and MMVP. The results show that equally fitting mappings can produce different target-model response scores, particularly for IDEFICS2. The paper concludes that matching internal representations alone is not sufficient to determine how a transferred intervention will affect another model.

Critical strengths

The paper addresses an interesting question about the limitations of transferring activation interventions between different models. One strength is its combination of mathematical analysis and experimental validation. The authors clearly demonstrate that multiple mappings can fit the same calibration data but produce different responses, even when the intervention strength is controlled. I also appreciate the careful experimental design, including comparisons across several model families, natural-image datasets, and additional tests with different intervention settings. The paper highlights an important distinction between matching internal representations and achieving similar model behavior. This is a useful consideration for future research on cross-model activation steering.

Critical weaknesses

My main concern is whether the findings are sufficiently connected to practical cross-model activation steering. Although previous research has shown that some steering directions can be transferred between models, the reliability of this approach is still not well established, particularly for VLMs. Furthermore, learning a mapping between two models from a small number of examples does not necessarily mean that the mapping is meaningful or generalizable. The paper studies differences between equally fitting mappings, but it does not first establish that the source interventions produce meaningful behavioral changes or that the learned mappings can reliably transfer them. In fact, the authors report that the source interventions do not show measurable effects under their experimental settings. The mathematical result is also somewhat expected because the experiments deliberately use an underconstrained mapping with only four calibration pairs. While the observed differences in target response scores are interesting, it remains unclear whether they explain actual failures in transferring useful steering behaviors. I believe the paper needs stronger experiments using validated steering directions and meaningful downstream behavioral outcomes to establish the significance of its contribution.

Questions for authors

The paper assumes that mappings between the hidden representations of different VLMs can be learned from paired calibration examples. However, it is unclear whether these mappings provide meaningful correspondence beyond the calibration data. The experiments deliberately use four calibration pairs in an eight-dimensional space, leaving some dimensions unconstrained. How representative is this setting of practical cross-model steering, where more calibration data may be available? Have the authors evaluated how well the mappings generalize to unseen examples, and whether the identified ambiguity remains important when the mappings achieve stronger held-out alignment? Is there a meaningful behavior to transfer in the first place? Is the mapping between the models sufficiently meaningful and generalizable to make the analysis relevant?

Reviewer_dfsvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates under what conditions interventions derived from one VLM can be reliably transferred to another (i.e., cross-model steering). The authors show that cross-model interventions that are effective under the training distribution can produce different effects on the downstream (i.e., target) model's responses to new heldout images. They evaluate this phenomenon across models from the Qwen2.5-VL, LLaVA, and IDEFICS2 families, and show that cross-model steering doesn't always succeed on heldout test images.

Critical strengths

None

Critical weaknesses

Lack of clarity: The abstract, introduction, and discussion sections of the paper read as an overly compressed enumeration of empirical results and seemingly arbitrary modeling decisions (e.g., the use of 8 dimensional intervention subspaces), which makes it difficult to evaluate the core contribution without already being familiar with the analysis framework proposed by the authors. I found it challenging to grasp broader significance of the papers results as presented in these key sections of the paper -- I suspect that the broader readership of this conference will also struggle here. To name just a few places where the paper would benefit from improved clarity and expanded exposition:

  • Line 23: "These results distinguish the role of state calibration from target-response evaluation: calibration fixes the map on the observed span, while updates outside that span require direct comparison at matched target norms."
  • Line 86: "The rank result states what paired states determine, matched completions expose the remaining freedom, and response diameter tests whether that freedom matters to target behavior. Controlled and natural evaluations cover three model families, five directed edges, two intervention steps, and a second residual site." These examples illustrates a recurring problem throughout the paper, where the definition of quantities and concepts, and explanations of why these quantities are being measured is omitted. A little expanded exposition in the introduction and abstract that doesn't pre-suppose the readers familiarity with the framework being used in the paper would go a long way in improving the papers clarity.

Results section: The above mentioned issues with the framing of the paper results in the subsections of the methods and results as reading overly terse and requiring additional effort by the reader to interpret what these results mean. The results subsections in particular contain limited interpretation of how their analyses connect to the broader contribution of the work. More clearly establishing the contribution early in the paper (i.e., in abstract/introduction) will help the reader to interpret the subsequent methods and results sections more easily.

  • Take subsection 5.2 of the results as an example: "The calibrated span behaves exactly as the theory predicts." should be expanded to actually provide an interpretation of how and why these results align with the theory's predictions.

Limited justification of key modeling decisions: The paper does not justify key modeling decisions that leave the reader wondering why specific hyperparameters were chosen (e.g., the use of 8 examples for calibration). Moreover, characterizing how the choice of these hyperparameters influences the measured outcomes would help generalize the papers findings beyond the more narrow configurations explored in the papers current analyses.

Questions for authors

A few suggestions for the authors that I think would improve the papers accessibility, and that may improve my assessment of the papers presentation:

  • The abstract and introduction of the paper should provide stronger conceptual anchors that orient the reader to the problem the work addresses without assuming familiarity with the methods and terminology adopted in the paper. For example, even by the third sentence of abstract the authors assume familiarity with the papers terminology (e.g., "calibration fit") that is unfair to assume of the reader at this stage: "In an eight-dimensional residual subspace, four paired examples constrain at most six dimensions, leaving maps with the same calibration fit and target update norm that can produce different finite responses."
  • I suspect one reason the abstract and introduction feel overly compressed is that some of the following sections (e.g., "WHAT PAIRED CALIBRATION IDENTIFIES", "EVALUATION DESIGN", and "RESULTS") expand on details that are not directly relevant to the papers core contribution. I would encourage the authors to move non-essential subsections to the Appendix, and to use this additional space to more clearly motivate the question the paper addresses and the methods used in the subsequent analyses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.