- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper analyzed how vlm will read the visual tokens for vqa questions and revealed that vlm often looks at regions where we might not expect by ablations like replace embeddings from "donor" images or by introducing "slot change vector" to direct vlm readouts.
Critical strengths
- An interesting surgery paper to reveal where vlm is looking inside when trying to answer vqa questions.
Critical weaknesses
- Tokens after vision encoder have already exchanged information. The core question behind the scene is
- 1\ how text tokens will attend to visual tokens
- 2\ which information is stored in the visual tokens
- The current study can be more thorough
- how's the attention map looks like?
- if we replace letters with symbols e.g. literally put donor's star to base letter, will model read the letter information stored in the donor symbol?
- Limited novelty. What's additional contribution this paper is trying to provide? We switched the embeddings of a patch, then get the information out from that switched embedding.
- the probing is misleading. it's a trained classifier to read that particular patch of embeddings to a target label, which only proves that the embedding is linear separable in its embedding space for the labels, but has nothing to do with which information in that space will be leveraged by the vlm readout.
Questions for authors
The paper looks over AI-polished e.g. "Look inside a vision–language model and you can often find what it saw in the image. It is natural to conclude that the model answers from what we found there. We show that it does not: what we read out is the vision encoder’s description of the image, and the language model answers from somewhere else. SightLine tests use by intervention rather than by reading.."
which makes the reading a bit tedious.