acceptodds
Withdrawn from ICLR 2027

Rosetta Neurons Across Vision and Language

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Can independently trained vision and language models develop individual neurons that respond to the same concepts? We identify these cross-modal rosetta neurons: pairs of neurons in separate vision and language models whose activations correspond across images and text. We extend neuron matching to paired images and captions by correlating each neuron's mean activation over the corresponding inputs. We find such matches across CLIP, DINOv2, MAE, and a Stable-Diffusion UNet paired with language models, including vision encoders trained without text. We recover the same neuron pairs in two separately sampled sets of image–caption pairs. Blinded model judges and human raters support concept agreement among selected high-correlation pairs. The matched population grows with model size, following sublinear power-law scaling across language models and text-free vision encoders. These findings suggest that independently trained models can represent corresponding visual and linguistic concepts at the level of individual neurons. We use these correspondences for text-indexed image retrieval: a concept named in text selects a language neuron, whose matched vision neuron retrieves images of that concept without additional training.

Primary area:
interpretability and explainable AI
Keywords:
Rosetta Neurons, Vision-Language, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gQeeOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether independent vision and language models develop neurons that respond to corresponding concepts. It identifies cross-modal neuron pairs by correlating their mean activations over paired image–caption data and demonstrates such correspondences across multiple models, including vision encoders trained without text. The findings are supported by reproducibility tests and human and model-based evaluations, with the number of matched neurons exhibiting sublinear power-law scaling with model size.

Critical strengths
  1. The work investigates neurons that encode similar semantic concepts across vision-language models.
  2. The work conducts experiments across multiple model architectures.
  3. The work provides multiple visualizations and analysis.
Critical weaknesses
  1. The proposed method lacks novelty, as it primarily collects neuron activations and performs statistical correlation analyses.
  2. The method lacks theoretical guarantees and relies entirely on the neuron correspondences between the original models. Therefore, its effectiveness cannot be guaranteed for arbitrary model pairs.
  3. The paper lacks discussion and comparison with prior methods, such as VL-SAE [1], which has mapped multimodal representations from CLIP and MLLM models into a unified concept space and leveraged this alignment to improve model performance.
  4. The proposed method does not demonstrate its potential to further improve model performance. Compared with existing methods [1, 2], the contribution of this work is limited to visualization and statistical analysis, without demonstrating tangible improvements in model capabilities.

[1] Shen, S., Sun, J., Huang, Q., & Wang, S. (2026). Vl-sae: Interpreting and enhancing vision-language alignment with a unified concept set. Advances in Neural Information Processing Systems, 38, 45235-45265.

[2] Renzulli, R., Spadaro, G., Gowda, S., Mazouz, A. E., & Nguyen, V. T. (2026). TORINO: Token Reduction via Interpretable Concept Overlap in Vision-Language Models. arXiv preprint arXiv:2607.04593.

Questions for authors

Overall, the paper lacks discussion and comparisons with highly relevant prior work, offers limited methodological novelty, and focuses solely on analyzing the internal properties of models without demonstrating practical improvements. Therefore, I have reservations about the significance and contribution of this work.

Reviewer_TCAHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether neurons that respond to the same concepts (Rosetta neurons) exist across independently trained vision and language models. Previous studies have demonstrated the existence of Rosetta neurons across different vision models and across different language models, but whether such neurons also exist across models of different modalities has remained unknown. In this paper, the authors identify Rosetta neurons based on correlations between neuron activations in vision and language models using image-caption paired datases. Furthermore, as an application of Rosetta neurons, the authors demonstrate that text-to-image retrieval can be performed using independently trained vision and language models without additional training.

Critical strengths
  • S1: The experimental analysis is rigorous.
Critical weaknesses
  • W1: The analysis methodology itself offers limited novelty, as it is similar to that of Dravid et al. (2026).
  • W2: The diversity of language models evaluated is limited compared to that of vision models. Do corresponding neurons also exist in other representative language models, such as DeepSeek, Qwen, and Gemma?
  • W3: The increase in the matched population with model size seems expected, as larger models naturally provide more candidate neurons for matching and more neurons with similar responses.
  • W4 (minor):
    • The authors should appropriately distinguish between the \citep and \citet commands to improve readability. For example, \citep should be used in cases such as "CLIP (Radford et al., 2021) is ...", whereas \citet should be used in cases such as "Radford et al. (2021) proposed CLIP."
    • The authors should provide more precise mathematical descriptions of the analysis methodology, including how activations and matching scores are computed and how the retrieval procedure is performed.
Questions for authors
  • Q1: Is image-to-text retrieval also possible?
Reviewer_q619OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends rosetta neurons across modalities. Using COCO image-caption pairs as the shared samples, the authors mean-pool each MLP neuron over image patches or caption tokens, correlate vision and language neurons, and keep mutual top-K pairs that reappear in two separate data draws. They find about a thousand such pairs between CLIP ViT-B/16 and Pythia-160M, and also with text-free encoders such as DINOv2 and MAE. Model judges and human raters find that the top pairs share concepts, the number of pairs grows sublinearly with model size, and the pairs can be used for simple text-to-image retrieval.

Critical strengths
  1. The question is interesting, and the text-free vision encoders make the result more than a CLIP artifact. Finding matched units between DINOv2 or MAE and a separately trained language model is a nice observation.

  2. The empirical work is careful and open. Caption-permutation nulls, stability across two draws, the disjoint gallery sweep, rescoring with other captions and a human study all go in the right direction, and the full pipeline is released.

Critical weaknesses
  1. The null does not rule out a much simpler explanation. Shuffling captions removes all image-caption correspondence, so beating this null only shows that both models encode what kind of scene a pair shows, which is already well known. Because activations are mean-pooled over the whole image and the whole caption, each neuron acts as a scene-level detector, and COCO is dominated by a few scene types. The largest clusters in Fig. 4b are exactly kitchens, food, trains, ball sports and bathrooms. The matching is also far from one-to-one: the 1,103 pairs involve only 612 language units, and the strongest language units each have several vision partners. So the result may just be that both models have units for common COCO scene categories, which is weaker than the neuron-level correspondence the paper claims. A null that shuffles captions only among images with the same COCO labels would test this directly.

  2. The scaling and application results are thin. Each power-law fit uses three to seven points, one of the five ladders fails the authors' own criterion, and the exponents are small, so the counts are close to flat. The paper also does not show what scaling the matching rule produces on its own, for example with random-weight models of the same sizes. For retrieval, the main text highlights the six best categories, while the median AP over all 80 categories is 0.045, close to the random-neuron control at 0.030 and far below CLIP at 0.533.

Minor issues:

  • Concept agreement and the human study only cover the top 100 pairs by correlation, so they say little about the other thousand pairs.
  • The method and the scaling analysis closely follow Dravid et al. (2023; 2026), so the main new step is pooling over paired image-caption data.
Questions for authors

See Weaknesses. Also,

  1. Do the matches survive a null that shuffles captions only among images sharing the same COCO object labels, or after regressing these labels out of both activation matrices?
  2. What scaling does the same pipeline give for random-weight models of the same sizes?
  3. If you sample pairs at random from the 1,103 instead of taking the top 100, how often do the judges find a shared concept?

Comments on the paper

1 comment

We thank the reviewers for their time and detailed feedback. We have read the reviews carefully and appreciate the questions and suggestions. We plan to continue developing this work and will consider the points raised as we prepare the next iteration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.