acceptodds
Withdrawn from ICLR 2027

MAST: Medoid-Anchored Subspace Steering for Visual Grounding in Multimodal LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing activation steering methods for multimodal large language models (MLLMs) use the global activation mean as the steering vector, which tends to cause cross-task interference and unstable performance. We trace this failure to the multi-centered geometry of task activations. The global mean lands in the empty region between clusters and is faithful to no single task. Moreover, the steering signal in each task-pure local cluster is low-rank, and different query tasks benefit from emphasizing different families of local-basis directions under the shared routing protocol. To address this, we propose MAST (Medoid-Anchored Subspace Steering), a training-free, input-adaptive activation steering framework. MAST groups samples by task, performs multi-center clustering within each group, and selects one representative medoid per cluster. At inference time, each query is routed to multiple representative medoids. For each matched cluster, a low-rank local steering direction is extracted and aggregated across clusters to produce the final steering vector. Building on MAST, MAST-PR (Probe-Reweighted) uses a lightweight task-aware probe to reweight task-specialized local-direction families, yielding a further average improvement of +8.09 points over MAST.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, activation steering, visual grounding, medoid-based routing, low-rank subspace, hallucination mitigation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DPxZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes MAST, an input-adaptive activation-steering method for multimodal LLMs. It clusters support samples within each task and, at inference time, routes each query to its most relevant medoids. It then combines the low-rank steering directions from the matched clusters. Experiments on CV-Bench and POPE show gains over the compared methods.

Critical strengths
  1. I appreciate the motivation. The paper recognizes that activation-steering methods can overlook differences among visual tasks. Section 3 and Appendix A examine these differences from several angles, including the clustering structure of support activations, conflicts among cluster directions, and the concentration of within-cluster directions in a low-rank subspace. These analyses give the method a solid empirical motivation.
  2. The method is simple, effective, and clearly presented. The claim that steering signals within task-pure clusters concentrate in an extremely low-rank subspace is supported by the SVD analysis. The paper also includes extensive component ablations and an analysis across intervention layers, which help clarify where the gains come from and how sensitive they are to layer choice.
Critical weaknesses
  1. The paper gives limited analysis of how MAST improves visual grounding. Given the title, I expected a more direct analysis of whether the model uses visual evidence more effectively after steering. The current evaluation mainly reports benchmark accuracy, along with a few qualitative examples where MAST answers correctly. These results show that performance improves on the tested tasks, but do not explain what changes in the model’s processing of the image.
  2. The validation of the method is limited in scope. The experiments cover CV-Bench and POPE, but I would like to see results on additional vision-centric benchmarks to assess whether the gains extend beyond these settings. In particular, evaluation on a benchmark such as BLINK [1], which tests a broader range of visual perception abilities, could help clarify how well MAST generalizes across visual tasks.
Questions for authors
  1. Could the authors analyze whether MAST improves visual grounding beyond the benchmark accuracy gains, with evaluations that help distinguish better use of image evidence from changes in answer or option preferences?
  2. I suggest the authors evaluate MAST on additional vision-centric benchmarks such as BLINK [1]. Results on a broader set of visual tasks would help establish how well the gains generalize beyond CV-Bench and POPE.
  3. More than this, the evaluation in this paper uses closed-ended answer formats, such as multiple-choice questions or yes/no questions. Could the authors clarify whether MAST can be applied to open-ended visual question answering? I am interested in that under this setting, how to construct and validate incorrect support completions, especially when multiple answers or phrasings are valid.

[1] Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N. A., Ma, W.-C., & Krishna, R. (2024). BLINK: Multimodal Large Language Models Can See but Not Perceive*. In Proceedings of the European Conference on Computer Vision (ECCV).

Reviewer_bic9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MAST, an activation steering method for multimodal large language models motivated by the observation that activation differences can exhibit multi-center structure. Instead of using a single global steering direction, MAST clusters support examples, routes each query to relevant local regions using medoids, and constructs local low-rank steering directions. The paper further introduces MAST-PR, which uses a task probe to reweight task-specific steering directions. Experiments on LLaVA-v1.5 and Qwen2-VL-7B across CV-Bench and POPE show substantial improvements over global steering baselines, together with ablations and analyses of the proposed components.

Critical strengths
  1. The empirical improvements are substantial. MAST achieves strong gains across several visual reasoning tasks and also shows improvements on hallucination evaluation, making the empirical findings practically interesting.
  2. The paper provides useful analysis beyond aggregate performance. The ablations and the analysis of the MAST-PR failure case help connect the observed performance to the behavior of the steering mechanism.
Critical weaknesses
  1. Several design choices are insufficiently justified by the central multi-center motivation. Clustering and local routing follow naturally from the observation of multi-center structure, but it is less clear why local low-rank/PCA subspaces and probe-based reweighting are the appropriate subsequent choices. This makes the overall method feel less unified than the motivating geometric story suggests.
  2. The experiments do not fully establish the necessity of the additional methodological complexity. In particular, stronger controlled comparisons against simple cluster-specific steering would help determine how much of the improvement comes from localization itself versus the proposed local-subspace construction and subsequent components.
Questions for authors
  1. Why is the local low-rank subspace construction necessary beyond simple cluster-specific steering? A direct comparison against cluster-specific mean/direction baselines would help clarify the contribution of this component.
  2. Can the authors provide a more unified justification for the individual components of MAST and MAST-PR? In particular, it would be useful to explain how local subspace construction and probe-based reweighting follow from the initial multi-center hypothesis rather than being independent design choices.
  3. How robust are the main conclusions across model families? The current results are encouraging, but broader evidence would strengthen the claim that the observed multi-center structure is a general property of MLLM activation steering.
Reviewer_fku1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper mainly improves activation steering for Multimodal Large Language Models (MLLMs) to address perception problems. Specifically, authors claim that the mean of contrastive activations across data samples is unsuitable for multi-task. They propose MAST, a method that use task-specific contrastive vectors, built by clustered vectors. Experiments on LLaVA-1.5 and Qwen2-VL shows the significance of the design.

Critical strengths

Writing is clear. The writing is overall clear to me but can be improved.

Critical weaknesses

Not Well-Motivated. The "not-well motivated" focus on two aspects. One is the base model authors use in their experiments, LLaVA-1.5 and Qwen2-VL. Improvements over these models (even in large margins) cannot guarantee the same improvements on frontier models (Qwen3.8, at least Qwen3-VL?). For example, under CVBench (Distance), Qwen3-VL can achieve much better result than LLaVA-1.5 (near 90 >> 51.67 for MAST). Another is CVBench and POPE, the results from this paper are not state-of-the-arts (especially on POPE). Overall, I do not think this paper make a very clear contribution, given the base model they use, and the problems are already well solved.
Weak Positioning to Related Works. Authors should do a more careful survey on "steering over MLLMs and LLMs", and make sure they do a detailed discussion about their difference to other methods, or direct comparisons on results, rather than ignoring them.
Incremental Novelty. Task-specific (or a more general name, divide-and-conquer) is a very naive idea in all fields, it should be used as a comparing methods, but not a design. I do not see the unique contribution of this idea to meet the requirement of this conference.

Questions for authors

Please see weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.