acceptodds
Withdrawn from ICLR 2027

Neither Object nor Text: A Mechanistic Analysis of Scene Text in LVLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models (LVLMs) are increasingly required to recognize scene text embedded in images. Scene text is spatially localized like visual objects, yet conveys linguistic content like textual input. This dual nature motivates us to investigate how scene text is internally represented and processed within LVLMs. Through empirical analyses of LVLM computation, we identify processing differences between scene text, visual objects, and directly supplied text. On identical images, scene text questions induce stronger attention concentration in the corresponding regions than object questions. Beyond this attention pattern, hidden-state analysis shows that scene text information is already detectable in visual representations before generation begins. Attention knockout further shows that scene text recognition depends more strongly than object recognition on access to image tokens in later decoder layers. Scene text also differs from directly supplied text in its layerwise attention patterns. Together, these findings suggest that scene text may function as a distinct representational modality within LVLMs: neither fully object-like nor purely language-like. These findings motivate further investigation into how scene text representations guide answer generation and how this understanding can improve recognition and response behavior.

Primary area:
interpretability and explainable AI
Keywords:
Scene text recognition, Vision-language models, Mechanistic interpretability, Visual representations, Attention knockout

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8xgrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper explores scene text representation and processing in large vision-language models. Scene texts are not objects, yet they require visual understanding. On the other hand, it is similar to text in the need to process individual characters and linguistic information. In this paper the authors ask how scene text is represented and attended to within models, and how this process differs from visual objects and plain texts.

The paper relies on two analyses: attention patterns and logit lens. Both of these analyses for objects and texts is not novel [1,2], but comparing it with scene texts does give us some new insights on the similarities an differences between scene text processing vs. objects and direct text.

[1] Why do LLMs attend to the first token? Barbero et al. 2025. [2] VISION TRANSFORMERS NEED REGISTERS. Darcet et al. 2024

Critical strengths
  • Although I find the scope of the paper narrow - the paper presents several questions and provides a convincing exploration of them. The similarities and differences to objects and text processing positions scene text as somewhat of a "middle ground" with distinct characteristics.

  • The experimental setup is robust: the claims in the paper are evaluated on 4 different models and are extended in section 5 to 4 different tasks (5 tasks overall).

Critical weaknesses

Narrow scope and novelty: The scope of new insights from this paper is limited. As i mentioned in the paper summary, the analyses for objects and texts are not novel, and the novelty is in comparing them to scene text, which is a narrow case. You give Typographic jailbreaks as motivation to understand the representation and processing of scene text, but I am missing an additional step - showing how this can be used to mitigate jailbreaks, bypass existing defense methods, or other applications. Exploring the representations and attention patterns alone adds a limited amount of new knowledge, and I would be excited to see examples of how this knowledge can be actionable or useful for some application.

The flow of the paper is difficult to follow:

  • Intro already shows results before discussing any setup (Figure 1). I suggest moving this figure to a results section and further discuss it there (what are the points in the plot?, which models and layers?).
  • Figures order does not match the order they are referred to in the text - figs 1 and 3 are referenced in the intro, while fig 2 is only referenced in section 3. The next fig. referenced is fig. 7, and so on.
  • I would suggest moving Section 3.1 (and perhaps even 3.2) entirely to be part of the intro, including the reference to fig.2, which explains the setup.
  • the term ROI is mentioned in the related work section and in Fig. 2 before it is explained.
  • The position of Fig. 5 in the text is really disruptive for reading, I suggest moving it to the next page.

Additional Comments:

  • Section 4 begins by stating: “Section 3.2 showed that scene text questions concentrate attention on text regions, whereas object questions do not comparably emphasize the same region”, however section 3.2 is motivation, and does not include any experiments or results. Seems like those are described in 4.1.

  • “Figure 5(a) shows that .. in the image conditions, attention to image tokens is generally higher in later layers than around the early-middle layers.” - while this is true - the higher attention to image tokens for objects is still lower than the attention to the textual tokens in the same layers, and even for scene text it's still similar to direct text (although lower). Both image inputs do not get the same attention rates as text, and this is not discussed.

This paper has good potential, and I think with some (major) additions and revisions could be accepted. I encourage the authors to continue working on improving the writings (such that the reader does not jump around pages and figures when reading, and that motivation is presented before results), and to show actionable contributions based on their findings - I would be really excited about applications for this in typographic jailbreaks. For now, the contributions and new insights are limited and the methodology relies solely on existing approaches (which is not a weakness in itself, but adds to the limited novelty). For these reasons I marked this paper as a clear reject.

Questions for authors

See Weaknesses section

Reviewer_iJ9cOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks how LVLMs process scene text compared with visual objects and with the same words given directly as text. Across four ~8B models (LLaVA-OV-1.5, Qwen2.5-VL, Qwen3-VL, InternVL3.5) on 404 COCO-Text images, it concludes with three main findings:

  • Scene text questions concentrate attention on the text region much more than object questions concentrate it on the object region, especially in later layers.
  • A logit lens can decode target words from image tokens inside the text region and from the last input positions.
  • Blocking image access in middle and late layers hurts scene text recall 32-47 points more than object recall, and blocking the text ROI hurts more than blocking random patches.

Controls include small localized objects, no-text logos, other reading benchmarks (TextVQA, CTW1500, DocVQA), a 32B model, and OCR models. A FigStep case study shows late knockout of the numbering region produces a few refusals. The authors also conclude that scene text acts as a distinct "representational modality".

Critical strengths

The same-image comparison is a good design choice. Asking for text versus objects on identical images, with paired per-image differences and bootstrap intervals, makes the late-layer knockout gap fairly convincing. The ROI-versus-random-patch control (5.8-20.8 vs. 0.3-1.7 points) shows the effect is spatially specific rather than a generic "less image" effect.

The empirical coverage is broad and the authors clearly tried to stress-test things: selection thresholds, image resolution, five prompts, and extra models and benchmarks. As a set of observations about where reading depends on visual access, the paper is useful.

Critical weaknesses
  • The main conclusion doesn't seem to follow from the evidence. "Distinct representational modality" is never defined in a way that could be falsified. What the experiments actually show is that reading an exact character string needs more late visual access than naming a COCO category. That could simply be a granularity or difficulty effect. Object names like "person" or "car" are coarse and often guessable from context early on, while reproducing "LECANTO" needs fine detail until the end. The logo control points the same way: Qwen2.5 loses 34 points on logos, so fine-grained visual identity without any text can show the same late dependence. Without a fine-grained object control (bird species, car models, etc.) matched in difficulty, I can't tell whether this is about text or about precision.
  • Some comparisons are confounded, and some effect sizes are small.
  • The direct-text condition uses a caption context repeated to match the image-token count. Attention-peak differences could partly come from that repetition, and the task is basically copying.
  • Knockout is scored by recall on generated answers. Scene text answers are often longer lists, so part of the loss may be disrupted generation rather than recognition. The authors admit degraded outputs appear under broad removal.
  • The logit-lens match rates on image positions are only a few percent, on a single-token subset of 201 images. "Detectable before the question" is also partly automatic under causal attention.
  • The FigStep part (7 refusals out of 100) feels tangential to the main claim.

Also, several appendix results come from "archived" runs with missing generation configurations, unexplained exclusions, and a judge identified only by an alias. This makes me less sure how reproducible those extensions are.

Questions for authors
  1. Could you add a fine-grained object recognition control (species, car models, landmarks) with difficulty similar to reading? If late-layer dependence shows up there too, the "scene text is special" claim weakens a lot.
  2. Does the extra scene-text loss hold when scoring only the first generated token, or only single-word targets, so that answer length and generation degradation are controlled?
  3. How would you define "representational modality" operationally? What result would have counted against the hypothesis?
  4. Attention is averaged over all heads. Is the text-ROI concentration driven by a few heads (e.g., the OCR heads of Baek et al.) or spread out?

One final minor thing is, Figure 1 compares natural object photos against generated label images. Is the PCA separation just a natural-vs-synthetic image difference?

Reviewer_99TJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks how LVLMs represent, attend to, and use scene text when they generate answers, compared with visual objects and with text given directly as tokens. Using several current LVLMs, it finds that attention concentrates inside text regions, that answer tokens can be read from text-region states before generation, and that blocking attention to the text region hurts scene text recall much more than object recall. Controls cover small objects, logos, other text benchmarks, and a jailbreak case study. The paper concludes that scene text works as a distinct representational modality.

Critical strengths

The paper runs three analyses on the same images: attention, representation, and causal knockout. They point the same way. The tests use four current models, matched trials with 95% intervals, and random-patch controls. The paper also states its own limits (lines 470–471).

Critical weaknesses
  1. The findings are informative, but the paper doesn’t show how to use them. The one use it tries, attention steering, gives only small TextVQA gains (Appendix H). If a finding cannot be used, its value is limited.

  2. The "representational modality" claim is not supported. Knockout shows scene text depends more on late-layer vision than objects do. Stronger dependence does not suggest a new modality. The two conditions are also different tasks: reading characters vs naming a category. The gap may come from the task, not from a modality.

Questions for authors

Please refer to above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.