- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Proposed a novel research perspective and systematically analyzed the relationship between text/image representations in VLMs and cross-modal alignment with the human brain. Adopted a rigorous 2×2 neural encoding framework and validated the text-to-image cross-modal advantage across multiple models, subjects, and robustness settings. Residual analysis and embedding prediction provided further analysis of the sources of representation asymmetry. ulative.
Critical strengths
- Proposed a novel research perspective and systematically analyzed the relationship between text/image representations in VLMs and cross-modal alignment with the human brain.
- Adopted a rigorous 2×2 neural encoding framework and validated the text-to-image cross-modal advantage across multiple models, subjects, and robustness settings.
- Residual analysis and embedding prediction provided further analysis of the sources of representation asymmetry. Interpreting linear predictability as cross-modal semantic alignment may be too strong.
Critical weaknesses
-
Interpreting linear predictability as cross-modal semantic alignment may be too strong. The analysis mainly tests whether one representation can linearly predict another representation or brain activity. But being linearly predictable does not necessarily mean that the two modalities are semantically aligned. It can also depend on representation geometry, dimensionality, or covariance structure. Therefore, linear prediction alone does not justify the conclusion that one modality contains more cross-modal semantic information.
-
The source of the observed asymmetry is not fully separated from the inherent differences between images and captions. Images and captions are naturally asymmetric: captions are usually compressed semantic descriptions of visual scenes, while images contain much richer visual detail. As a result, the text-to-image advantage may partly come from the information imbalance between the two modalities, rather than purely from properties of VLM representations.
-
The residualization analysis relies on a narrow definition of shared information. The paper defines modality-shared information as the component that can be linearly predicted across modalities. However, shared semantic information may also exist in nonlinear forms. The current analysis may capture only the linearly predictable part and may not fully reflect the complete cross-modal shared structure.
-
The connection to the broader representation learning community is still somewhat limited. The paper provides an interesting analysis of VLM representations using neural data, but it does not show how the discovered asymmetry can improve model training, evaluation, or downstream VLM capabilities. Its practical implications for future representation learning therefore remain speculative.
-
The problem addressed by this work is meaningful and relevant. However, the proposed method does not demonstrate sufficient methodological novelty for ICLR, as it lacks a sufficiently compelling and well-validated approach to solving the problem.
Questions for authors
See the weaknesses
4 comments
4. + 5. Connection to representation learning. As part of this submission to the “applications to neuroscience & cognitive science” primary area, improvements of VLM capabilities are out of the scope of this paper. Our main novelty is the presented 2x2 cross-modal encoding framework, and the asymmetry across the cross-modal prediction settings it allowed us to discover. We argue that this cross-modal comparison of similarities/differences in human neural representations and VLM-based ones is a meaningful contribution to the research community at the intersection of neuroscience and AI (NeuroAI), and to representation learning more broadly. Moreover, the additional analyses performed as part of this discussion period further strengthen the validity of the approach. Nonetheless, we now explicitly name potential directions for future research on improving VLMs in the discussion: “For VLM research, the asymmetry exposes a measurable shortcoming of the tested image encoders, and our paired paradigm offers a brain-grounded benchmark for future work on training objectives that aim to close it.”
References Naselaris, T., Kay, K. N., Nishimoto, S., & Gallant, J. L. (2011). Encoding and decoding in fMRI. NeuroImage, Multivariate Decoding and Brain Reading, 56(2), 400–410.
Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021). The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45).
Doerig, A., Kietzmann, T. C., Allen, E., Wu, Y., Naselaris, T., Kay, K., & Charest, I. (2025). High-level visual representations in the human brain are aligned with large language models. Nature Machine Intelligence, 7(8), 1220–1234.
Thank you for your thoughtful response. I will make my decision in the next step.
Thank you for acknowledging our response, we appreciate your time!
We thank the reviewer for their helpful comments, please find our point-to-point reply below.
1. + 3. Nonlinearity. Our primary choice of linear prediction for the quantification/isolation of cross-modal information is further supported by Appendix A.9, in which we tested a non-linear prediction setting alongside the linear one, and only found very minimal differences (median absolute difference in Pearson r of 0.013). We hypothesize this may partly reflect vision-language alignment during pretraining, which could make cross-modal information linearly accessible. In line with most prior studies on neural encoding (e.g., Naselaris et al. (2011), Schrimpf et al. (2021), Doerig et al. (2025)), we used a linear neural encoding setting to investigate the representation geometry of the vision/text embeddings themselves, rather than the capability of a (potentially powerful) non-linear encoder, which would present a major confounder for making claims about the image/text embeddings themselves. Nevertheless, we now make this limitation of focusing on linear predictability more explicit in the discussion: “However, factors beyond linear predictability may explain semantic alignment between modalities, such as the geometry, dimensionality, or covariance structure of the representations.”
2. Imbalanced information content across text and images. We agree that inherent differences between shorter captions and more detailed scenes could indeed be a limiting factor in our study design, and we now explicitly acknowledge this in the discussion: “This difference in information content also applies within our dataset: captions describe only a subset of the objects, actions, and relations depicted in a scene, so unequal semantic coverage could contribute to the asymmetry.”. To reduce the information imbalance across text and images, we used OLMo 3.1 32B Instruct to generate more detailed captions. The instruction for the caption generation was to describe all actions in the provided COCO-Actions scene graph for a given image (see Appendix A.15 for more details). Then, we use the more detailed captions to repeat the cross-modal embedding prediction (research question RQ3) and the text embedding-based neural encoding experiments (RQ1): text embeddings -> text fMRI, and text embeddings -> image fMRI. For the cross-modal embedding prediction (RQ3), the detailed captions reduce the asymmetry between text=>image and image=>text embedding prediction (difference in r between the modalities is reduced from 0.093 to 0.048). For neural encoding (RQ1), detailed captions predict image-evoked activity slightly better (0.045 vs. 0.040, see Table 8).
Neural encoding (RQ1, Appendix A.15, Table 8). The table shows the mean r/r_NC across the 14 models with a text encoder; in parentheses, the number of models that are significant. With detailed captions, text embeddings predict image-evoked activity slightly better, so the encoding asymmetry becomes larger rather than smaller. Text-evoked activity, which was elicited by the original captions, is predicted much less well.
Embedding prediction (RQ3, Appendix A.15, Table 9). Detailed captions roughly halve the difference in cross-modal embedding prediction. Richer captions therefore may reduce the asymmetry in the embeddings themselves, but not in neural encoding. Image embeddings, which already contain rich scene content, still predict caption-evoked activity only weakly.
Importantly, if missing information in the captions was the key reason for asymmetric cross-modal neural encoding, we argue that we would predict the opposite asymmetry than the one we observe. If a caption describes only a subset of its scene, the image contains everything the caption conveys, so image embeddings should predict caption-evoked activity at least as well as caption embeddings predict image-evoked activity. We observe the reverse; text embeddings predict image fMRI, whereas image embeddings predict text fMRI to a far lesser extent, so we believe that incomplete captions cannot explain the asymmetry.