- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper investigates whether models specialized for OCR reuse attention heads that were already involved in retrieving and copying textual information. It identifies heads associated with the transcription of text, tables, and formulas, and assesses their importance through ablations on held-out data. Its main contribution is demonstrating a significant overlap with attention heads independently identified in the context of textual retrieval by means of cross-model interventions and the substitution of value vectors. It also finds that this specialization retains many of the core attention heads, though their relative importance shifts.
Critical strengths
-
The overlap between heads found separately through OCR and text retrieval is an important result. The experiment in Section 5.4 helps explain what those heads do. Retaining the recipient’s attention weights fixed and inserting values from a different character shifts the logits toward that character. This shows that the shared heads carry information about the character being read, rather than only attending to its location. The cross-task ablations also show that removing these heads hurts performance on both tasks.
-
Heads selected on two separate data splits overlap by at least 19 out of 20 across all nine specialist model–task combinations, and their ablations are tested on held-out data. The before-and-after comparison adds another useful finding, as OCR specialization retains 17–20 of the top 20 heads per task, while changing their scores and ablation effects. These results show both a stable selection procedure and how adaptation changes the model’s reliance on existing heads.
-
I liked how the paper connects its experiments. The overlap between OCR and retrieval heads is a good example. It raises the question of whether the same heads matter for both tasks, which the cross-task ablations then test. This connection between results and new questions runs through the paper and makes it interesting to read.
Critical weaknesses
-
I am missing one check in Section 5.2: how badly would OCR perform after removing 20 randomly chosen heads from the same layers? The reported NED rises from 0.1106 to 0.6234, but without that check I cannot judge how much worse it is to remove retrieval heads specifically. I also could not find a random-head result for NIAH after the authors ablate OCR heads.
-
In Section 5.4, the target region and the distant control positions are not matched for attention mass. If the head assigns little weight to the distant positions, replacing their values will have little effect regardless of spatial selectivity. The size of the value changes is not matched either. Both quantities need to be controlled before the near-zero effect at distant positions can support that claim.
-
As an additional limitation, the argmax criterion ignores the amount of attention directed toward the evidence and how that attention is distributed across the patches. It would be useful to assess the extent to which the findings depend on this specific choice.
Questions for authors
-
In the comparison presented in Section 6, can you disentangle the effects of the model parameters from those of the prompts? A cross-evaluation, testing different checkpoints against different prompts (where the formats are compatible), would help clarify whether the observed changes are due to model specialization.
-
Baek et al. (2025) reached a different conclusion about the overlap between OCR and retrieval heads. Does your use of full-sequence transcription explain this? I would like to understand what supports that explanation and whether the choice of models or head-selection method could also account for the result.
3 comments
Comment part 2
Question 4: Can the respective effects of model parameters and prompts be separated?
Answer 4: We adjusted the prompts of the general models without SFT mainly because they do not adequately understand the short prompts(i.e. Text Recognition:) designed for the specialized models, which leads to disorganized output formats that cannot be analyzed sensibly. To address your concern further, we repeated the analysis using a single consistent long prompt before and after fine-tuning, and the conclusions are essentially the same as those obtained with different prompts. The corresponding experiments are as follows.
Proportion of incorrect tokens under different prompts:
| Model | No fine-tuning + short prompt | No fine-tuning + original long prompt | SFT + short prompt |
|---|---|---|---|
| Qwen2-VL | 44.70% | 8.98% | 2.52% |
| Qwen3-VL | 9.81% | 1.68% | 3.76% |
Intersection of the OCR heads identified when long prompts are used for fine-tuning in both settings:
| Model | Task | Same long prompt before and after: intersection / ρ (Spearman) |
|---|---|---|
| Qwen2-VL | Text | 18/20 · 0.9277 |
| Qwen2-VL | Table | 18/20 · 0.9393 |
| Qwen2-VL | Formula | 18/20 · 0.8816 |
| Qwen3-VL | Text | 18/20 · 0.9286 |
| Qwen3-VL | Table | 20/20 · 0.9594 |
| Qwen3-VL | Formula | 18/20 · 0.9072 |
Question 5: How should the conclusions of the paper be reconciled with the different conclusion of Baek et al. (2025)?
Answer 5: First, the task formulations of our work and of Baek's work differ substantially. Our task requires the model to reproduce all visible text by making full use of the information in the image, whereas Baek's work mainly adopts the visual NIAH form. What their work calls an “OCR head” in fact refers to “identifying a specified piece of text within an image containing mixed text”, i.e. “image-text retrieval”, which is different from our definition.
Second, they use CoT as guidance to make the model produce its answer, which may affect the model's attention patterns.
Third, in our paper we mainly use top-k as the characterization of “overlap”, that is, we use to measure the overlap between the heads of the two specific tasks. We do this to assess the similarity of the two on the top-ranked, more important heads, and to show similarity in mechanism rather than in number, as far as possible through causal experiments. Baek instead describes the overlap using Jaccard similarity computed over successive score bands, which measures set-level similarity more. They can therefore easily reach the conclusion that “the overlap between the so-called OCR heads and retrieval heads is low” (since the intersection is divided by the size of the union). Our conclusions are not actually in conflict; they are simply two ways of expressing. In fact, if we measure the overlap by the number of attention heads in a manner similar to Baek's work, the OCR heads identified by our method also exhibit the sparsity reported in their work (the number of heads activated by OCR is far smaller than that for retrieval). To pin down the difference between the two further, we also computed Jaccard values as a supplement; on text recognition, our Jaccard value is close to theirs. The results are as follows:
| Model | Task | # OCR heads | # Retrieval heads | Intersection | Fraction of OCR heads contained | Jaccard |
|---|---|---|---|---|---|---|
| Qwen2-VL-2B | Text | 13 | 31 | 12 | 92.31% | 37.50% (34.7% in Baek et al.) |
| Qwen2-VL-2B | Table | 16 | 31 | 15 | 93.75% | 46.88% |
| Qwen2-VL-2B | Formula | 7 | 31 | 6 | 85.71% | 18.75% |
Thanks for the responses
Thank you for your responses. Most of my concerns have been addressed, so I will keep my positive rating.
Comment part 1
Dear Reviewer Wjjx,
Thank you for your insightful comments, which gave us much to think about. Our point-by-point responses are below.
Question 1: Section 5.2 lacks a control check.
Answer 1: Thank you for this comment. We have added the random-control experiments and will include them in the revised version of the paper. The results are given in the tables below.
Ablating retrieval heads: OCR versus random controls
Ablating OCR heads: random controls on the retrieval task (NIAH)
These experiments show that ablating OCR heads and retrieval heads damages both the retrieval and the OCR tasks far more than ablating random attention heads in the same layers.
Question 2: In Section 5.4, the total attention weight on the target region and on the distant control location is not matched.
Answer 2: This is a good question. In the paper we have tried our best to keep the discrepancy between the target region and the distant control location from being too large, but we cannot fully eliminate the effect this has — after all, there is no way for us to directly control this model-generated variable, the total attention mass. We will explore this in future work.
Question 3: The argmax criterion has additional limitations.
Answer 3: We additionally introduced an attention-mass metric, defined in “In-context Learning and Induction Heads” (2022). For head h and generated position t, it is the sum of the attention weights that head h places at position t on the evidence patches:
Smass(h,t)=∑p∈Etah,t,p
where Et is the set of evidence patches for position t and ah,t,p is the attention weight from position t to patch p.
We first ask whether the top-20 OCR heads selected by argmax are also robust on Smass. Compared with layer-matched random attention heads, we observe behavior consistent with the argmax metric: under every condition, the attention mass of the top-20 heads selected by argmax is significantly higher than that of the layer-matched random group. The argmax criterion is therefore not incidental; other corresponding metrics show good agreement as well: