- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines whether VLMs generalize novel visual concepts through object similarity or abstract relational structure. Using controlled Relational Match-to-Sample (RMTS) tasks, the authors find that larger models and scenes with more objects favor relational matching, while visual noise weakens it. Layer-wise RSA reveals a shift from object-aligned to relation-aligned representations. Activation patching identifies attention heads that influence relational decisions, and the authors test their contribution to ARC-AGI-1.
Critical strengths
-
Well-motivated experimental design. The paper adapts a cognitive science paradigm to VLMs, using controlled stimuli to distinguish object-based from relational generalization.
-
Consistent behavioral and representational findings. Experiments across model families reveal systematic trends, while RSA tracks the effects of object count and visual noise.
-
Causal interventions and careful documentation. Head-level interventions complement the RSA analysis, and the appendices provide useful controls, reproducibility details, and limitations.
Critical weaknesses
-
The evidence does not establish two competing circuits. RSA shows how representational similarity changes across layers, but the transition could arise from successive stages of a single computation. The authors patch attention heads only at the prediction position. These interventions may identify heads that read out an existing decision rather than compute relational structure. The paper locates signals associated with relational decisions without explaining how the model constructs those relations or demonstrating competition between distinct circuits.
-
The experimental controls leave important alternative explanations open. Increasing object count also changes pattern redundancy and the appearance of the object-match candidate. The shape-coherence control in Appendix H does not establish whether the trend persists at larger scene sizes. Furthermore, averaging over both answer orderings makes a model that always selects A indistinguishable from one that chooses randomly, complicating the interpretation of chance-level results.
-
The ARC-AGI-1 results do not establish relational specificity. The identified heads lie primarily in late layers, but the random-ablation baseline samples heads across all layers. The larger performance drop could therefore reflect the greater influence of late-layer heads. A depth-matched baseline and statistical tests are needed before claiming that these heads support relational reasoning beyond RMTS.
-
The novelty needs clearer positioning. Previous studies have applied RMTS and related relational-abstraction tasks to neural networks and VLMs, including Campbell et al. (2024), Webb et al. (2023), and Puebla and Bowers (2024). The authors should distinguish their preference-based design and causal analyses from this prior work rather than presenting RMTS itself as a new diagnostic for VLMs.
Questions for authors
-
Can the authors distinguish heads that compute relational structure from heads that merely read out the answer, perhaps by patching image tokens while keeping answer positions fixed?
-
Does the object-count effect persist under shape-controlled stimuli at larger values of n?
-
Can the authors report consistency across answer orderings to distinguish genuine matching preferences from positional bias?
-
Does the ARC-AGI-1 effect remain under depth-matched random ablation?
-
How do the authors reconcile the proposed relational circuit with the finding that CoT restores performance after its strongest heads are ablated?
4 comments
Author Response to Reviewer oCti (2/2)
Q3
Can the authors report consistency across answer orderings to distinguish genuine matching preferences from positional bias?
Yes. With Y/Z as answer letters, for which the models have no prior:
| % of trials | relational in both orders | object in both orders | same letter in both orders |
|---|---|---|---|
| Gemma-4-26B | 99 | 0 | 1 |
| Gemma-4-31B | 100 | 0 | 0 |
| InternVL3-38B | 100 | 0 | 0 |
These models choose the relational match whichever position it is in, so their preference is not a positional bias. We report this in Section 2.2 (page 4) and Appendix N.7, Table 17 (page 33).
Q4
Does the ARC-AGI-1 effect remain under depth-matched random ablation?
Yes. We reran the baseline with 20 seeds of random heads from the same layers. Ablating the relational heads gives 0.408, which is below every seed (0.536 ± 0.052; Wilcoxon test paired over tasks, p = 7×10⁻⁵). The drop grows with the number of ablated heads (k = 1 / 3 / 5: 0.512 / 0.408 / 0.367). A non-relational control, where the model only copies the same grids, is not affected (0.937 → 0.944). We replaced Figure 9 (page 9), updated Section 5 (page 10) and added Appendix N.4, Table 13 (page 32). Thank you for the suggestion. The ARC result now has proper statistics.
Q5
How do the authors reconcile the proposed relational circuit with the finding that CoT restores performance after its strongest heads are ablated?
The three-stage circuit gives a simple explanation. The relation is computed at the sample and candidate tokens in the early and middle layers (stages 1 and 2). The relational heads are the last stage and turn it into an immediate answer. Ablating them does not remove the relation that was computed earlier. When the model writes out a chain of thought, it can read that representation again and state the comparison in words, and this brings the performance back. We added this explanation to the CoT paragraph of Section 4.3 (page 9).
W4 (novelty)
The authors should distinguish their preference-based design and causal analyses from this prior work rather than presenting RMTS itself as a new diagnostic for VLMs.
We agree that RMTS itself is not our contribution, and we should have said so more clearly. Webb et al. (2023) compare LLMs with humans on analogy tasks at the behavioural level. Campbell et al. (2024) document VLM failures and relate them to binding. Puebla & Bowers evaluate object-centric models that are trained on visual reasoning tasks. All of these ask whether a model can solve a relational task. We ask a different question, in two parts: (i) with no training on the task, which kind of generalisation, object-based or relational, does a generative VLM prefer; and (ii) how is that choice made inside the model, which we trace causally from the samples to the candidates to the answer and test on ARC-AGI-1. We rewrote the positioning in Section 5 (page 10) and Appendix P (page 34-35).
Thank you again for the detailed comments. Please tell us if anything is still unclear, and we will be happy to explain further.
Thank you for the reply. The new experiments and results address my main concerns about the causal analysis, experimental controls, and ARC-AGI generalization.
I encourage the authors to include these results and clarify the contribution relative to prior work.
My main concerns have been addressed, and I like the concept of this paper. I will maintain my recommendation.
Thank you for reading our response and the revised paper. We are glad that the new experiments address your main concerns. The new results and the clarified contribution relative to prior work are both in the revised version, and we will keep them in the final version. Thank you again for your support.
Author Response to Reviewer oCti (1/2)
Thank you for the positive and constructive review. We answer each point below and would be glad to hear whether this resolves your concerns. All changes in the revised paper are marked in red.
Q1
Yes. We did what you suggest: we patched the image tokens while keeping the answer positions fixed.
The data. Each item of a new set, which we call the interchange set, gives two prompts, c1 and c2. The two candidate images, their positions and the answer letters are the same in both. Only the samples differ, and with them the relation that decides which candidate is correct. Figure 18 (page 30) shows an example, and Appendix N.1 (pages 29–31) describes the construction.
What we do. We copy residual-stream activations from the run on c2 into the run on c1, separately for three groups of tokens: the sample tokens (image tokens of the two samples), the candidate tokens (image tokens of the two candidates) and the final tokens (the text after the last image, up to the position where the answer is predicted). Since only the samples differ between the prompts, any effect is caused by the relation they show.
Result: the relational circuit has three stages (Gemma-4-26B, 30 layers; Figure 8, page 9).
The experiment separates the two roles: the relation is computed in stages 1 and 2, and the heads we identified are stage 3 of the same circuit. The results are in Section 4.3 (page 9), Figure 8 (page 9) and Appendix N.1–N.2 (pages 29–31; Figures 18–19). Thank you for the suggestion. It took us from a list of heads to a circuit we can follow through the prompt.
W1
We ran ablations restricted to the tokens where each set of heads acts. Ablating object heads at the image tokens makes the relational margin larger (Gemma-4-26B +0.83, Gemma-4-31B +1.38, InternVL3-14B +0.32). Random heads from the same layers do not. Ablating relational heads at the final tokens makes it smaller in all five models that solve the task. If these were successive stages of one computation, removing the earlier stage should not help the later one. The two sets of heads push in opposite directions. The results are in Section 4.3 (page 9) and Appendix N.3, Tables 11–12 (page 32). Thank you for pressing us on this. The competition claim now has causal evidence behind it.
Q2
We made stimuli for n from 3 to 20 that leave out the "all different" pattern, so the pattern rule does not change with n, and we combined them with shape-controlled candidates (both in the same unseen shape). With the shape control, Gemma-4-26B chooses the relational match in at least 95% of trials at every n, and Gemma-4-31B in at least 94% for n ≥ 4, up to n = 20. This is in Section 3.3 (page 5) and Appendix N.6, Table 16 (page 33).