- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates which components of vision-language models support the identification of familiar entities. The authors define familiarity tiers using recognition under image perturbations, then scan attention-layer windows by replacing head outputs with their mean over the input sequence. Experiments cover four models and four entity domains: faces, logos, landmarks, and artworks. The authors find that interventions in middle decoder layers substantially reduce identity responses, with varying effects on perceptual controls. Further analysis of Qwen2.5-VL identifies five attention heads whose joint ablation sharply reduces logo recognition. The paper also examines how the intervention changes responses when visual evidence conflicts with memorized knowledge.
Critical strengths
- The experiments cover four model families and several entity domains. The systematic layer sweeps provide a useful account of where identity responses are sensitive to intervention. The authors also report the less selective results in Qwen3-VL and Gemma-4, which helps establish the limits of the findings. (Sections 3 and 4; Figure 3.)
- The edited-face experiment provides a useful control on the same images. In the highly known tier, identity preservation falls from 58.3% to 14.9%, while 99.0% of attribute responses remain unchanged. This helps rule out complete model failure as the explanation for the recognition drop. (Section 4.2; Appendix F; Figure 9.)
Critical weaknesses
-
The reference lines in Figure 3 overlap the annotations. The vertical dotted lines run through the layer labels at the top of the panels. Please move the labels or shorten the lines so that they do not intersect. This is a minor presentation issue; the methodological concerns below determine my assessment.
-
The recognition measure conflates identification with willingness to answer. The main probes require a known or unknown prefix, and the parser treats responses that do not begin with known as unknown. A decrease in the reported recognition rate can therefore reflect changes in confidence, abstention, or output compliance as well as entity identification. Appendix D describes some of these response changes, but does not establish whether the model has lost the ability to distinguish the entity. The direct naming experiment on edited faces helps address dependence on the prescribed format, although it still allows an unknown response. The discussion acknowledges this limitation, but it remains central to the proposed recognition-gate interpretation. (Section 2; Appendices B, D, and J.)
-
The localization may depend on the intervention itself. Mean replacement changes each selected head's output at every input position, including both visual and text positions. It also retains an input-dependent average contribution. This operation can alter information routing beyond the processing of entity identity. A recognition-loss peak could therefore identify layers that are particularly sensitive to this replacement procedure. I did not find a comparison establishing that the same localization holds under a different intervention. Without that check, the functional interpretation of the selected window remains uncertain. (Section 2, Equation 2; Appendix B.)
-
The controls are too coarse to establish the claimed selectivity. Most control probes concern color or the presence of text and circles. Preserving these answers does not rule out damage to the finer visual features needed to distinguish faces, logos, or landmarks. The edited-face controls mainly test color reporting and leave this possibility open. The evidence also varies considerably across models: control responses change on 13% to 31% of examples in Gemma-4 and 35% to 46% in Qwen3-VL. The authors acknowledge these differences, but the results provide limited support for a shared selective gate across all four models. (Section 4.1; Figure 3; Appendices D and F.)
-
The five-head result lacks independent validation of selectivity. The heads are selected according to the logo recognition loss caused by individual ablations, and their joint effect is reported on the same logo collection. The paper does not describe a separate selection and evaluation split. More importantly, Figure 4 reports recognition performance without corresponding perceptual-control results for this five-head intervention. The controls for ablating the entire layer window cannot establish the selectivity of the five selected heads. The drop from 96.1% to 3.1% demonstrates strong dependence of logo responses on this set, but its selectivity and transfer to other entities remain untested. (Section 4.2; Figure 4; Appendix G.)
Questions for authors
-
On the existing entity images, does the intervention still cause a large accuracy drop in a forced-choice identification task that does not allow abstention? Please randomize the answer order. This would help determine how much of the reported effect comes from impaired entity discrimination rather than a change in the threshold for answering.
-
Does the same layer window emerge under an alternative intervention that keeps visual and text positions separate? Evidence that the localization persists would help address the possibility that the current result depends on sequence-wide mean replacement.
-
Does the intervention preserve fine-grained visual discrimination when no entity name is required, for example in an image-matching task? Such a control would help distinguish disruption of identity-related visual processing from disruption of knowledge retrieval or identity reporting.
-
Were the five heads selected and evaluated on separate samples? With the head set fixed, please report recognition loss and perceptual-control changes on images excluded from selection. Any existing evaluation in other entity domains would also help assess whether this is a stable functional component.
Thank you for highlighting the systematic sweeps, edited-face controls, and reporting of differences across models.
Experiments 1–5 are described in our general comment.
Addressing the weaknesses:
We will move the layer labels above the panels and shorten the reference lines so they no longer overlap.
We agree that the format and option to abstain complicate the interpretation; While appendix F does not fully resolve this, we are working on forced-choice identification without abstention. We will post the results once ready and verified. For now, we describe the effect as a loss of explicit naming, without assuming loss of identification ability.
In the complete sweeps (Exp 2), zero ablation at all positions and mean ablation at text positions both recover layers 16–17 on faces and logos. Zero ablation there gives 100% naming loss with 13% control change on faces and 5% on logos. Final-position interventions peak later for faces, and connection blocking can peak elsewhere.
We added face matching using different photographs, visually similar negatives, and no names. Accuracy is 88.2% intact, 88.7% with five-head ablation, and 84.9% with layer ablation (Exp 1). This harder control supports the naming result, especially for the five heads; we also report the larger layer-ablation drop for highly known people. Model differences remain at matched widths (Exp 5). We will limit selectivity claims to the tested models and abilities, without claiming a shared selective gate.
On 235 new brands excluded from selection, the fixed heads give 100% naming loss among 160 initially correct cases, with 1.7% control change (Exp 3). The largest loss among 25 random sets with the same layer allocation is 15.6%. We also test the same heads on faces and landmarks, with direct visual controls, and add face matching in Exp 1.
Addressing the questions:
These experiments are still in progress, and we will post the results once ready and verified.
Text-only mean ablation recovers layers 16–17, with naming loss / control change of 85% / 12% on faces and 100% / 4% on logos. Visual-only ablation at that window gives only 5% and 2% naming loss, respectively. Final-position interventions peak at layers 19–20 for faces. Exp 2 therefore supports the same window under text-only intervention while showing that the positions we target matter.
Five-head matching accuracy is 88.7%, compared with 88.2% intact. Layer ablation gives 84.9%, with a larger drop for highly known people. Exp 1 gives the setup and results. This supports a separation between naming and matching, most clearly for the five heads.
The submission used the same logos for selection and evaluation. We now test the original heads on new brands and other domains without reselecting them. Naming loss / control change is 91% / 7.7% on landmarks and 58% / 10.8% on faces. Exp 3 also reports a separate selection/evaluation split and comparisons with random head sets.