- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper extends the ‘candidate-specific perturbation reporting’ paradigm developed by Fornasiere et al. (2026) for LLMs to VLMs. The experiment was designed such that a perturbation was applied to one of two images, and the model was asked to identify which of the two had been perturbed. The paper ensures image quality by category and counting. Finally, the paper does not provide sufficient support for the hypothesis it puts forward. Experimental results are reported on Qwen2.5-VL, Qwen3-VL and Idefics3.
Critical strengths
This article is honest, and its claims are restrained.
Critical weaknesses
(1) This paper cannot rule out another important explanation (the degradation of image representation); therefore, it cannot substantiate the claim made in the paper: that visual language models can identify which image’s visual features have been perturbed.
Also, category classification and counting cannot sufficiently guarantee that the model’s visual functions are functioning properly.
(2) The steering analysis offers limited insight and novelty. The observation that linearly solvable probes of equal accuracy exhibit different steering effects is not theoretically surprising and does not introduce any new concepts, as linear solvability does not imply causal validity.("Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?", ACL 2021)
The authors also acknowledge that it fails to demonstrate a native monitoring mechanism, consequently, this set of experiments makes little contribution to the paper’s central research question.
Questions for authors
Let the models report which one has undergone perturbation after controlling for the degree of feature degradation in the two images. Only if the model still significantly outperforms chance levels is there strong reason for substantiating the claim made in this paper.