- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper explores cross-model steering in VLMs. The authors explore whether state fit determines what the mapped intervention does to the target model on held-out inputs. In the paper, they use three models from different backbones, Qwen2.5VL, IDEFICS2, LLaVA 1.5 with the controllable experiments with VQAv2 and MMVP. They show that in an 8-dimensional residual subspace, four calibration pairs constrain the map on at most six dimensions. The authors introduce a metric, response diameter, is the spread of the target's finite responses across these maps. They argue that mapping accuracy fixes the map only on the calibrated span, so mapped interventions must be evaluated directly on held-out target responses at matched update norms.
Critical strengths
- Cross-model steering work usually validates a map by its low mapping error and then tests one selected map on the downstream tasks. The paper frames the issue clearly: what a low mapping error identifies is different from what held-out target responses establish.
- An interpretable theory formulation. Proposition 1 gives a simple rank upper bound, which the measured designs attain. Proposition 2 constructs completions with identical mapping error, operator norm and update norm. Corollary 1 gives the response diameter a decision-theoretic meaning.
- Controllable experiments with three VLM families, and two datasets.
Critical weaknesses
- Limited conceptual novelty. Equal mapping error with different held-out behavior is related to underspecification [0].
- No exploration on high dimensions. All the experiments is on 8-dimensional subspace. In practical, the experiments usually use enough data to do the mapping. For example over 100k in [1], and over 5k image-text pairs [3]. There, this ambiguity largely disappears. Although the authors acknowledge the restriction to linear maps in an 8 dimensional subspace. The paper does not test the realistic cases where its concern would bite: full-dimensional maps fitted on fewer paired samples than dimensions, such as least-squares maps on roughly 100–1k samples in ~4k-dimensional spaces;
- The experiments setup is not clearly define. The paper doesn’t give the reason for intervening at 55% depth.
- The results don’t have a clear explaination. Qwen2.5-VL-7B resolves only 3/83 controlled groups and 0/24 on both natural edges, but the paper does not explain why. The intervention may simply be too small to register in Qwen. A model-relative update norm or a sweep over update norms is needed to rule this out.
- There is no explain why choose 8-dimensional compression. The subspace keeps only 47–64% of the energy of the paired-image differences. The paper does not report how much of each donor direction survives the projection. Since r = 8 also determines the size of the unconstrained complement, an ablation over r is needed.
[0] D’Amour, Alexander, et al. "Underspecification presents challenges for credibility in modern machine learning." 2022 [1] Chen, Alan, et al. "Transferring linear features across language models with model stitching." 2025 [2] Li, Jiaang, et al. "Do vision and language models share concepts? a vector space alignment study." 2024.
Questions for authors
See weaknesses.