- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper explores scene text representation and processing in large vision-language models. Scene texts are not objects, yet they require visual understanding. On the other hand, it is similar to text in the need to process individual characters and linguistic information. In this paper the authors ask how scene text is represented and attended to within models, and how this process differs from visual objects and plain texts.
The paper relies on two analyses: attention patterns and logit lens. Both of these analyses for objects and texts is not novel [1,2], but comparing it with scene texts does give us some new insights on the similarities an differences between scene text processing vs. objects and direct text.
[1] Why do LLMs attend to the first token? Barbero et al. 2025. [2] VISION TRANSFORMERS NEED REGISTERS. Darcet et al. 2024
Critical strengths
-
Although I find the scope of the paper narrow - the paper presents several questions and provides a convincing exploration of them. The similarities and differences to objects and text processing positions scene text as somewhat of a "middle ground" with distinct characteristics.
-
The experimental setup is robust: the claims in the paper are evaluated on 4 different models and are extended in section 5 to 4 different tasks (5 tasks overall).
Critical weaknesses
Narrow scope and novelty: The scope of new insights from this paper is limited. As i mentioned in the paper summary, the analyses for objects and texts are not novel, and the novelty is in comparing them to scene text, which is a narrow case. You give Typographic jailbreaks as motivation to understand the representation and processing of scene text, but I am missing an additional step - showing how this can be used to mitigate jailbreaks, bypass existing defense methods, or other applications. Exploring the representations and attention patterns alone adds a limited amount of new knowledge, and I would be excited to see examples of how this knowledge can be actionable or useful for some application.
The flow of the paper is difficult to follow:
- Intro already shows results before discussing any setup (Figure 1). I suggest moving this figure to a results section and further discuss it there (what are the points in the plot?, which models and layers?).
- Figures order does not match the order they are referred to in the text - figs 1 and 3 are referenced in the intro, while fig 2 is only referenced in section 3. The next fig. referenced is fig. 7, and so on.
- I would suggest moving Section 3.1 (and perhaps even 3.2) entirely to be part of the intro, including the reference to fig.2, which explains the setup.
- the term ROI is mentioned in the related work section and in Fig. 2 before it is explained.
- The position of Fig. 5 in the text is really disruptive for reading, I suggest moving it to the next page.
Additional Comments:
-
Section 4 begins by stating: “Section 3.2 showed that scene text questions concentrate attention on text regions, whereas object questions do not comparably emphasize the same region”, however section 3.2 is motivation, and does not include any experiments or results. Seems like those are described in 4.1.
-
“Figure 5(a) shows that .. in the image conditions, attention to image tokens is generally higher in later layers than around the early-middle layers.” - while this is true - the higher attention to image tokens for objects is still lower than the attention to the textual tokens in the same layers, and even for scene text it's still similar to direct text (although lower). Both image inputs do not get the same attention rates as text, and this is not discussed.
This paper has good potential, and I think with some (major) additions and revisions could be accepted. I encourage the authors to continue working on improving the writings (such that the reader does not jump around pages and figures when reading, and that motivation is presented before results), and to show actionable contributions based on their findings - I would be really excited about applications for this in typographic jailbreaks. For now, the contributions and new insights are limited and the methodology relies solely on existing approaches (which is not a weakness in itself, but adds to the limited novelty). For these reasons I marked this paper as a clear reject.
Questions for authors
See Weaknesses section