acceptodds
Under review as a conference paper at ICLR 2027

From Retrieval to Recognition: How Vision–Language Models Become OCR Specialists

Abstract

Does a general vision–language model acquire specialized OCR ability by developing a new reading circuit or by reusing an existing mechanism? We address this question in the setting of full-sequence OCR, rather than local-answer retrieval. Using an evidence-grounded protocol with held-out causal interventions, we identify sparse and stable OCR-head sets in GLM-OCR, MinerU2.5, and PaddleOCR-VL-1.6. We then investigate the mechanistic origin of these OCR heads by comparing them with independently identified textual retrieval/copy heads in general VLMs. Across two general VLMs, visual OCR heads strongly overlap independently identified textual retrieval/copy heads, yielding untuned top-20 intersections of 73.3% and all-head Spearman correlations of 0.677–0.886. The overlap and causal interventions suggest that full-sequence OCR operates as dense sequential multimodal copy-and-paste, repeatedly retrieving visual evidence and routing it to the current output position. Finally, we examine how this shared circuit changes as a general VLM becomes an OCR specialist. Matched base-to-specialized comparisons show that OCR specialization largely preserves head identity, retaining 17–20 of the top 20 heads per task with all-head rank correlations of 0.874–0.942, while redistributing their functional and causal strengths.

Primary area:
interpretability and explainable AI
Keywords:
Optical Character Recognition, interpretability of VLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WjjxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates whether models specialized for OCR reuse attention heads that were already involved in retrieving and copying textual information. It identifies heads associated with the transcription of text, tables, and formulas, and assesses their importance through ablations on held-out data. Its main contribution is demonstrating a significant overlap with attention heads independently identified in the context of textual retrieval by means of cross-model interventions and the substitution of value vectors. It also finds that this specialization retains many of the core attention heads, though their relative importance shifts.

Critical strengths
  1. The overlap between heads found separately through OCR and text retrieval is an important result. The experiment in Section 5.4 helps explain what those heads do. Retaining the recipient’s attention weights fixed and inserting values from a different character shifts the logits toward that character. This shows that the shared heads carry information about the character being read, rather than only attending to its location. The cross-task ablations also show that removing these heads hurts performance on both tasks.

  2. Heads selected on two separate data splits overlap by at least 19 out of 20 across all nine specialist model–task combinations, and their ablations are tested on held-out data. The before-and-after comparison adds another useful finding, as OCR specialization retains 17–20 of the top 20 heads per task, while changing their scores and ablation effects. These results show both a stable selection procedure and how adaptation changes the model’s reliance on existing heads.

  3. I liked how the paper connects its experiments. The overlap between OCR and retrieval heads is a good example. It raises the question of whether the same heads matter for both tasks, which the cross-task ablations then test. This connection between results and new questions runs through the paper and makes it interesting to read.

Critical weaknesses
  1. I am missing one check in Section 5.2: how badly would OCR perform after removing 20 randomly chosen heads from the same layers? The reported NED rises from 0.1106 to 0.6234, but without that check I cannot judge how much worse it is to remove retrieval heads specifically. I also could not find a random-head result for NIAH after the authors ablate OCR heads.

  2. In Section 5.4, the target region and the distant control positions are not matched for attention mass. If the head assigns little weight to the distant positions, replacing their values will have little effect regardless of spatial selectivity. The size of the value changes is not matched either. Both quantities need to be controlled before the near-zero effect at distant positions can support that claim.

  3. As an additional limitation, the argmax criterion ignores the amount of attention directed toward the evidence and how that attention is distributed across the patches. It would be useful to assess the extent to which the findings depend on this specific choice.

Questions for authors
  1. In the comparison presented in Section 6, can you disentangle the effects of the model parameters from those of the prompts? A cross-evaluation, testing different checkpoints against different prompts (where the formats are compatible), would help clarify whether the observed changes are due to model specialization.

  2. Baek et al. (2025) reached a different conclusion about the overlap between OCR and retrieval heads. Does your use of full-sequence transcription explain this? I would like to understand what supports that explanation and whether the choice of models or head-selection method could also account for the result.

3 comments
AuthorsOpenReview ↗

Comment part 1

Dear Reviewer Wjjx,

Thank you for your insightful comments, which gave us much to think about. Our point-by-point responses are below.

Question 1: Section 5.2 lacks a control check.

Answer 1: Thank you for this comment. We have added the random-control experiments and will include them in the revised version of the paper. The results are given in the tables below.

Ablating retrieval heads: OCR versus random controls

OCR taskMetricNo ablationAblate Retrieval Top-20Ablate 20 random heads in the same layers
TextNED ↓0.11060.62340.1727
TableTEDS ↑0.67200.15970.4246
FormulaCDM ↑0.96140.35020.9159

Ablating OCR heads: random controls on the retrieval task (NIAH)

Ablated head setNo ablationAblate OCR Top-20Ablate the 20 corresponding random heads in the same layers
Text OCR heads0.90560.35560.9089
Table OCR heads0.90560.40000.8889
Formula OCR heads0.90560.25000.8544

These experiments show that ablating OCR heads and retrieval heads damages both the retrieval and the OCR tasks far more than ablating random attention heads in the same layers.

Question 2: In Section 5.4, the total attention weight on the target region and on the distant control location is not matched.

Answer 2: This is a good question. In the paper we have tried our best to keep the discrepancy between the target region and the distant control location from being too large, but we cannot fully eliminate the effect this has — after all, there is no way for us to directly control this model-generated variable, the total attention mass. We will explore this in future work.

Question 3: The argmax criterion has additional limitations.

Answer 3: We additionally introduced an attention-mass metric, defined in “In-context Learning and Induction Heads” (2022). For head hh and generated position tt, it is the sum of the attention weights that head hh places at position tt on the evidence patches:

Smass(h,t)=∑p∈Etah,t,pS_{\mathrm{mass}}(h,t)=\sum_{p\in E_t} a_{h,t,p}

where EtE_t is the set of evidence patches for position tt and ah,t,pa_{h,t,p} is the attention weight from position tt to patch pp.

We first ask whether the top-20 OCR heads selected by argmax are also robust on SmassS_{\mathrm{mass}}. Compared with layer-matched random attention heads, we observe behavior consistent with the argmax metric: under every condition, the attention mass of the top-20 heads selected by argmax is significantly higher than that of the layer-matched random group. The argmax criterion is therefore not incidental; other corresponding metrics show good agreement as well:

ModelTaskBefore SFTBefore SFT, randomAfter SFTAfter SFT, random
Qwen2Text16.17%3.40%19.47%5.31%
Qwen2Table16.08%3.73%16.96%4.35%
Qwen2Formula14.55%2.73%15.78%2.58%
Qwen3Text36.81%3.76%42.49%4.55%
Qwen3Table25.97%2.69%29.90%2.94%
Qwen3Formula20.53%2.68%21.71%2.63%
AuthorsOpenReview ↗

Comment part 2

Question 4: Can the respective effects of model parameters and prompts be separated?

Answer 4: We adjusted the prompts of the general models without SFT mainly because they do not adequately understand the short prompts(i.e. Text Recognition:) designed for the specialized models, which leads to disorganized output formats that cannot be analyzed sensibly. To address your concern further, we repeated the analysis using a single consistent long prompt before and after fine-tuning, and the conclusions are essentially the same as those obtained with different prompts. The corresponding experiments are as follows.

Proportion of incorrect tokens under different prompts:

ModelNo fine-tuning + short promptNo fine-tuning + original long promptSFT + short prompt
Qwen2-VL44.70%8.98%2.52%
Qwen3-VL9.81%1.68%3.76%

Intersection of the OCR heads identified when long prompts are used for fine-tuning in both settings:

ModelTaskSame long prompt before and after: intersection / ρ (Spearman)
Qwen2-VLText18/20 · 0.9277
Qwen2-VLTable18/20 · 0.9393
Qwen2-VLFormula18/20 · 0.8816
Qwen3-VLText18/20 · 0.9286
Qwen3-VLTable20/20 · 0.9594
Qwen3-VLFormula18/20 · 0.9072

Question 5: How should the conclusions of the paper be reconciled with the different conclusion of Baek et al. (2025)?

Answer 5: First, the task formulations of our work and of Baek's work differ substantially. Our task requires the model to reproduce all visible text by making full use of the information in the image, whereas Baek's work mainly adopts the visual NIAH form. What their work calls an “OCR head” in fact refers to “identifying a specified piece of text within an image containing mixed text”, i.e. “image-text retrieval”, which is different from our definition.

Second, they use CoT as guidance to make the model produce its answer, which may affect the model's attention patterns.

Third, in our paper we mainly use top-k as the characterization of “overlap”, that is, we use top-k(O)∩top-k(R)\text{top-}k(O) \cap \text{top-}k(R) to measure the overlap between the heads of the two specific tasks. We do this to assess the similarity of the two on the top-ranked, more important heads, and to show similarity in mechanism rather than in number, as far as possible through causal experiments. Baek instead describes the overlap using Jaccard similarity computed over successive score bands, which measures set-level similarity more. They can therefore easily reach the conclusion that “the overlap between the so-called OCR heads and retrieval heads is low” (since the intersection is divided by the size of the union). Our conclusions are not actually in conflict; they are simply two ways of expressing. In fact, if we measure the overlap by the number of attention heads in a manner similar to Baek's work, the OCR heads identified by our method also exhibit the sparsity reported in their work (the number of heads activated by OCR is far smaller than that for retrieval). To pin down the difference between the two further, we also computed Jaccard values as a supplement; on text recognition, our Jaccard value is close to theirs. The results are as follows:

ModelTask# OCR heads# Retrieval headsIntersectionFraction of OCR heads containedJaccard
Qwen2-VL-2BText13311292.31%37.50% (34.7% in Baek et al.)
Qwen2-VL-2BTable16311593.75%46.88%
Qwen2-VL-2BFormula731685.71%18.75%
Reviewer_WjjxOpenReview ↗

Thanks for the responses

Thank you for your responses. Most of my concerns have been addressed, so I will keep my positive rating.

Reviewer_WL2tOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work explores OCR attention heads in sequence transcription and their relation to text retrieval/copy heads. Generated text is aligned with image regions; candidate heads are screened on a discovery set and validated via ablation on a held-out set. Comparisons of OCR heads and retrieval heads are conducted across three OCR models, Qwen2-VL and Qwen3-VL. Attention value patching is adopted to examine visual information transfer. Comparisons between base and OCR-finetuned models show top-ranked attention heads are mostly preserved, though their scores and intervention effects change.

Critical strengths
  1. The paper centers on how full OCR reuses general retrieval capabilities and how specialization alters such capabilities, linking visual transcription, text retrieval, and task adaptation. Unlike explanations limited to text retrieval in local question answering, this perspective aids understanding of the organization of continuous transcription capability and provides concrete, testable hypotheses for discussing the relationship between general multimodal models and OCR-specialized models.
  2. Texts, tables and formulas cover distinct transcription requirements. Analyses of three OCR-specialized models and two generations of Qwen connect observations from specialized models to adaptation in general models. Despite varying depths of validation across models, this setup helps identify replicable findings and specialization-related variations, and offers a unified comparison framework for follow-up research.
Critical weaknesses

Functional head discovery in specialized models, comparisons of OCR/text retrieval heads in Qwen, and source identity intervention in Qwen2-VL serve arguments at different levels. They mutually support one another, yet do not equate to equal validation of the same mechanism across all models. Consistency across two generations of Qwen also cannot fully represent the specialization rules for independent architectures. This limits the generalizability of the paper’s overall mechanistic claims. It is recommended to explicitly indicate which models support each conclusion, and prioritize supplementing key cross-task and source-information interventions on a separate model family, rather than merely adding more ranking results of similar heads.

Questions for authors

Please address the key concerns raised above.

1 comment

Dear Reviewer WL2t,

Thank you for your kind recognition of this work. In the present experiments we focus primarily on the Qwen series, for two reasons. First, these are open-source models widely studied within the AI community, which gives us a large body of reference work to draw on during our research. Second, this series represents a class of VLM paradigm that converts an image into patches and then into visual tokens, a paradigm that is now adopted in many VLMs. In future work we plan to extend the scope of our study to models beyond this paradigm.

Reviewer_fhCYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks which attention heads underlie full-sequence OCR transcription in vision-language models, and proposes an evidence-grounded protocol for identifying them: free-generated tokens are aligned to visual evidence patches, heads whose attention argmax falls inside the evidence are scored and ranked, and candidates are validated on held-out data via ablation against layer-matched random head sets. Across three OCR specialists (GLM-OCR, MinerU2.5, PaddleOCR-VL-1.6), the discovered top-20 heads are cross-fold stable and causally consequential for text, table, and formula recognition. The authors then compare these heads with textual retrieval/copy heads independently identified from NIAH tasks in Qwen2-VL-2B and Qwen3-VL-2B, finding substantial overlap, cross-task causal transfer, and counterfactual value-patching evidence for source-identity transmission. Finally, matched general-to-specialized comparisons suggest that OCR specialization largely preserves head identity while redistributing functional and causal strength.

Critical strengths
  1. A mechanistically meaningful question. The paper asks not merely whether a VLM can perform OCR, but how visual evidence is routed into token-by-token text generation.

  2. Candidate discovery is separated from causal validation. Attention alignment is explicitly used only to nominate candidates, which are then tested by held-out interventions. More rigorous than concluding attention heatmaps alone.

Critical weaknesses
  1. Head discovery uses only correctly generated tokens. Tokens that do not match the reference are excluded before OCR heads are identified. As a result, the analysis may explain where the model attends when it reads correctly, but not what happens when it makes errors or omits text. Please report how many tokens are excluded and whether the head rankings remain stable when partially matched or incorrect tokens are included.

  2. Ablation shows that these heads matter, but does not establish that they implement retrieval and copying. Removing a set of heads can harm OCR because they support general information routing or generation, rather than because they specifically retrieve and copy visual content. Layer-matched random heads are a useful control, but may not rule out these alternatives. Controls matched by visual contribution or activation strength would make the causal interpretation stronger.

  3. Correlated rankings do not necessarily imply a shared mechanism. OCR heads are identified from visual transcription, while retrieval/copy heads are identified from text-based NIAH tasks. The two tasks differ in their inputs and scoring criteria. Their overlap supports a relationship, but the claim that they implement the same computation is stronger. Please test whether the results are robust to changes in needle position, context length, and the copying criterion.

  4. The general-to-specialized comparison confounds fine-tuning with prompt changes. The paper states that the general and specialized models use different deployment prompts. The observed changes could therefore result from fine-tuning, prompt differences, or output-format constraints. A comparison using the same prompt and output format is needed to attribute the changes to OCR specialization.

  5. The value-patching control does not fully match intervention strength. The authors note that target and non-target positions are not matched for attention mass or value-difference magnitude. Since the target region is more relevant by design, the larger logit shift may not by itself establish selective transmission of character identity. Please provide non-target controls matched by attention mass or patch magnitude, and test sensitivity to the choice of top-k, thresholds, and sample filters.

Questions for authors

See the weaknesses.

1 comment

Dear Reviewer fhCY,

Thank you for your insightful comments. They were very helpful and gave us much to think about. Our point-by-point responses are below.

Question 1: In our study, the attention-head findings are based only on correctly generated tokens.

Answer 1: The main reason is that correct tokens make it easy to locate the corresponding evidence patches (also used in Baek et al.), whereas for incorrect tokens we may not only fail to locate a patch conveniently, but may even be unable to find a corresponding evidence patch in the image at all, since the model may output content that does not exist in the image. To explain this further, we first define an incorrect token as any token outside the set of correct tokens, and then measure the proportion of incorrect tokens (Note that a portion of these incorrect tokens are structure-related tokens):

ModelTextTableFormula
GLM-OCR1.91% (115/6,026)4.05% (2,541/62,665)12.54% (1,397/11,136)
MinerU2.51.68% (99/5,903)1.81% (1,144/63,154)13.31% (1,521/11,428)
PaddleOCR-VL-1.62.35% (139/5,917)1.84% (1,105/60,166)27.75% (3,383/12,189)
Qwen2-VL, no fine-tuning8.98% (528/5,877)21.11% (14,205/67,284)16.61% (1,907/11,481)
Qwen2-VL, SFT, short prompt2.52% (150/5,960)8.21% (5,456/66,469)20.64% (2,455/11,895)
Qwen2-VL, SFT, long prompt3.12% (222/5,983)8.19% (5,511/67,326)20.91% (2,498/11,946)
Qwen3-VL, no fine-tuning1.68% (99/5,898)9.62% (6,657/69,235)6.80% (709/10,429)
Qwen3-VL, SFT, short prompt3.76% (225/5,991)5.58% (3,776/67,715)13.88% (1,565/11,273)
Qwen3-VL, SFT, long prompt3.71% (222/5,983)5.63% (3,812/67,718)14.19% (1,605/11,308)

Baek et al., How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Heads, EMNLP'25.

Question 2: The ablation study shows that these heads matter, but it does not prove that they perform retrieval and copying.

Answer 2: Your comment is very insightful. We also considered this issue while completing this work: could it be that these heads are important for many VLM tasks in general? We therefore conducted experiments on the POPE dataset, a task unrelated to copy-and-paste (mentioned in Section 5.3). The results on this dataset show that the OCR heads we identify form a specialized class of heads rather than generic ones.

Question 3: Rank correlation does not necessarily imply a shared mechanism.

Answer 3: Our main evidence for a degree of overlap in their mechanisms comes from causal experiments. The rank correlation mainly shows that the correlation holds not only within the top 20 heads but also over a wider range.

Question 4: The comparison between general and specialized models conflates the effects of fine-tuning and of prompt changes.

Answer 4: We adjusted the prompts of the general models without SFT mainly because they do not adequately understand the short prompts designed for the specialized models (especially Qwen2 VL), which results in disorganized output formats that cannot be analyzed sensibly. To address your concern further, we repeated the analysis using a single consistent long prompt before and after fine-tuning, and the conclusions are essentially the same as those obtained with different prompts. The corresponding experiments are as follows.

Proportion of incorrect tokens under different prompts:

ModelNo fine-tuning + short promptNo fine-tuning + original long promptSFT + short prompt
Qwen2-VL44.70%8.98%2.52%
Qwen3-VL9.81%1.68%3.76%

Intersection of the OCR heads identified when long prompts are used for fine-tuning in both settings:

ModelTaskSame long prompt before and after: intersection / ρ (Spearman)
Qwen2-VLText18/20 · 0.9277
Qwen2-VLTable18/20 · 0.9393
Qwen2-VLFormula18/20 · 0.8816
Qwen3-VLText18/20 · 0.9286
Qwen3-VLTable20/20 · 0.9594
Qwen3-VLFormula18/20 · 0.9072

Question 5: The controls in the value-replacement experiment do not adequately match the strength of the intervention.

Answer 5: We controlled the magnitude of the value change as far as possible. Across multiple experimental settings, we consistently find that after replacing the value, the model has a clear tendency to shift toward the replacement value. For example, in Appendix H, after replacing "3" with "8", logit(8) − logit(3) increases markedly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.