- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes MAST, an input-adaptive activation-steering method for multimodal LLMs. It clusters support samples within each task and, at inference time, routes each query to its most relevant medoids. It then combines the low-rank steering directions from the matched clusters. Experiments on CV-Bench and POPE show gains over the compared methods.
Critical strengths
- I appreciate the motivation. The paper recognizes that activation-steering methods can overlook differences among visual tasks. Section 3 and Appendix A examine these differences from several angles, including the clustering structure of support activations, conflicts among cluster directions, and the concentration of within-cluster directions in a low-rank subspace. These analyses give the method a solid empirical motivation.
- The method is simple, effective, and clearly presented. The claim that steering signals within task-pure clusters concentrate in an extremely low-rank subspace is supported by the SVD analysis. The paper also includes extensive component ablations and an analysis across intervention layers, which help clarify where the gains come from and how sensitive they are to layer choice.
Critical weaknesses
- The paper gives limited analysis of how MAST improves visual grounding. Given the title, I expected a more direct analysis of whether the model uses visual evidence more effectively after steering. The current evaluation mainly reports benchmark accuracy, along with a few qualitative examples where MAST answers correctly. These results show that performance improves on the tested tasks, but do not explain what changes in the model’s processing of the image.
- The validation of the method is limited in scope. The experiments cover CV-Bench and POPE, but I would like to see results on additional vision-centric benchmarks to assess whether the gains extend beyond these settings. In particular, evaluation on a benchmark such as BLINK [1], which tests a broader range of visual perception abilities, could help clarify how well MAST generalizes across visual tasks.
Questions for authors
- Could the authors analyze whether MAST improves visual grounding beyond the benchmark accuracy gains, with evaluations that help distinguish better use of image evidence from changes in answer or option preferences?
- I suggest the authors evaluate MAST on additional vision-centric benchmarks such as BLINK [1]. Results on a broader set of visual tasks would help establish how well the gains generalize beyond CV-Bench and POPE.
- More than this, the evaluation in this paper uses closed-ended answer formats, such as multiple-choice questions or yes/no questions. Could the authors clarify whether MAST can be applied to open-ended visual question answering? I am interested in that under this setting, how to construct and validate incorrect support completions, especially when multiple answers or phrasings are valid.
[1] Fu, X., Hu, Y., Li, B., Feng, Y., Wang, H., Lin, X., Roth, D., Smith, N. A., Ma, W.-C., & Krishna, R. (2024). BLINK: Multimodal Large Language Models Can See but Not Perceive*. In Proceedings of the European Conference on Computer Vision (ECCV).