acceptodds
Under review as a conference paper at ICLR 2027

Cross-Modal Neural Encoding Is Asymmetric: Evidence From Vision-Language Models and Paired Text-Image Functional MRI

Abstract

Vision-language models (VLMs) have been shown to predict neural responses to visual stimuli better than unimodal models, which has been interpreted as evidence that VLMs and the human brain converge on shared, cross-modal representations. However, existing studies supporting this claim have critical limitations due to the functional magnetic resonance imaging (fMRI) datasets they use, because they often present a single modality, bind modalities inseparably (e.g., a video with simultaneous audio), present mismatched stimuli across modalities, or restrict paired stimuli to isolated concepts. We address these limitations by using a recent phrase- and scene-level text-image fMRI dataset built from paired stimuli: each stimulus exists as both a caption and an image, but any given participant encounters it in only one form. Therefore, cross-modal prediction cannot be attributed to simultaneous presentation, enabling a direct test of whether VLMs and the brain share cross-modal representations. Across multiple VLMs and unimodal baselines, we find a consistent asymmetry: text embeddings reliably predict image-evoked brain activity, whereas image embeddings are largely unable to predict text-evoked brain activity above chance level. We then use a residualization analysis to show that removing the cross-modally shared information from VLM embeddings selectively reduces cross-modal encoding performance to a larger extent than within-modality encoding performance. Mapping the embedding spaces onto each other directly, we find this same asymmetry: text embeddings predict image embeddings more reliably than vice versa. Taken together, these results indicate that VLMs and the brain do share cross-modal information, but not symmetrically: cross-modal information may be more directly accessible from text embeddings than from image embeddings.

Primary area:
applications to neuroscience & cognitive science
Keywords:
neural encoding, functional magnetic resonance imaging, vision-language models, cross-modality

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vkkhOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Proposed a novel research perspective and systematically analyzed the relationship between text/image representations in VLMs and cross-modal alignment with the human brain. Adopted a rigorous 2×2 neural encoding framework and validated the text-to-image cross-modal advantage across multiple models, subjects, and robustness settings. Residual analysis and embedding prediction provided further analysis of the sources of representation asymmetry. ulative.

Critical strengths
  1. Proposed a novel research perspective and systematically analyzed the relationship between text/image representations in VLMs and cross-modal alignment with the human brain.
  2. Adopted a rigorous 2×2 neural encoding framework and validated the text-to-image cross-modal advantage across multiple models, subjects, and robustness settings.
  3. Residual analysis and embedding prediction provided further analysis of the sources of representation asymmetry. Interpreting linear predictability as cross-modal semantic alignment may be too strong.
Critical weaknesses
  1. Interpreting linear predictability as cross-modal semantic alignment may be too strong. The analysis mainly tests whether one representation can linearly predict another representation or brain activity. But being linearly predictable does not necessarily mean that the two modalities are semantically aligned. It can also depend on representation geometry, dimensionality, or covariance structure. Therefore, linear prediction alone does not justify the conclusion that one modality contains more cross-modal semantic information.

  2. The source of the observed asymmetry is not fully separated from the inherent differences between images and captions. Images and captions are naturally asymmetric: captions are usually compressed semantic descriptions of visual scenes, while images contain much richer visual detail. As a result, the text-to-image advantage may partly come from the information imbalance between the two modalities, rather than purely from properties of VLM representations.

  3. The residualization analysis relies on a narrow definition of shared information. The paper defines modality-shared information as the component that can be linearly predicted across modalities. However, shared semantic information may also exist in nonlinear forms. The current analysis may capture only the linearly predictable part and may not fully reflect the complete cross-modal shared structure.

  4. The connection to the broader representation learning community is still somewhat limited. The paper provides an interesting analysis of VLM representations using neural data, but it does not show how the discovered asymmetry can improve model training, evaluation, or downstream VLM capabilities. Its practical implications for future representation learning therefore remain speculative.

  5. The problem addressed by this work is meaningful and relevant. However, the proposed method does not demonstrate sufficient methodological novelty for ICLR, as it lacks a sufficiently compelling and well-validated approach to solving the problem.

Questions for authors

See the weaknesses

4 comments

We thank the reviewer for their helpful comments, please find our point-to-point reply below.

1. + 3. Nonlinearity. Our primary choice of linear prediction for the quantification/isolation of cross-modal information is further supported by Appendix A.9, in which we tested a non-linear prediction setting alongside the linear one, and only found very minimal differences (median absolute difference in Pearson r of 0.013). We hypothesize this may partly reflect vision-language alignment during pretraining, which could make cross-modal information linearly accessible. In line with most prior studies on neural encoding (e.g., Naselaris et al. (2011), Schrimpf et al. (2021), Doerig et al. (2025)), we used a linear neural encoding setting to investigate the representation geometry of the vision/text embeddings themselves, rather than the capability of a (potentially powerful) non-linear encoder, which would present a major confounder for making claims about the image/text embeddings themselves. Nevertheless, we now make this limitation of focusing on linear predictability more explicit in the discussion: “However, factors beyond linear predictability may explain semantic alignment between modalities, such as the geometry, dimensionality, or covariance structure of the representations.”

2. Imbalanced information content across text and images. We agree that inherent differences between shorter captions and more detailed scenes could indeed be a limiting factor in our study design, and we now explicitly acknowledge this in the discussion: “This difference in information content also applies within our dataset: captions describe only a subset of the objects, actions, and relations depicted in a scene, so unequal semantic coverage could contribute to the asymmetry.”. To reduce the information imbalance across text and images, we used OLMo 3.1 32B Instruct to generate more detailed captions. The instruction for the caption generation was to describe all actions in the provided COCO-Actions scene graph for a given image (see Appendix A.15 for more details). Then, we use the more detailed captions to repeat the cross-modal embedding prediction (research question RQ3) and the text embedding-based neural encoding experiments (RQ1): text embeddings -> text fMRI, and text embeddings -> image fMRI. For the cross-modal embedding prediction (RQ3), the detailed captions reduce the asymmetry between text=>image and image=>text embedding prediction (difference in r between the modalities is reduced from 0.093 to 0.048). For neural encoding (RQ1), detailed captions predict image-evoked activity slightly better (0.045 vs. 0.040, see Table 8).

Neural encoding (RQ1, Appendix A.15, Table 8). The table shows the mean r/r_NC across the 14 models with a text encoder; in parentheses, the number of models that are significant. With detailed captions, text embeddings predict image-evoked activity slightly better, so the encoding asymmetry becomes larger rather than smaller. Text-evoked activity, which was elicited by the original captions, is predicted much less well.

ConditionOriginal captionsDetailed captions
Text embeddings → image fMRI0.040 (14/14)0.045 (14/14)
Text embeddings → text fMRI0.032 (14/14)0.008 (11/14)
Image embeddings → text fMRI (reference, 10 VLMs)0.005unchanged

Embedding prediction (RQ3, Appendix A.15, Table 9). Detailed captions roughly halve the difference in cross-modal embedding prediction. Richer captions therefore may reduce the asymmetry in the embeddings themselves, but not in neural encoding. Image embeddings, which already contain rich scene content, still predict caption-evoked activity only weakly.

CaptionsText⇒imageImage⇒textDifference
Original0.3140.220+0.093
Detailed0.3060.258+0.048

Importantly, if missing information in the captions was the key reason for asymmetric cross-modal neural encoding, we argue that we would predict the opposite asymmetry than the one we observe. If a caption describes only a subset of its scene, the image contains everything the caption conveys, so image embeddings should predict caption-evoked activity at least as well as caption embeddings predict image-evoked activity. We observe the reverse; text embeddings predict image fMRI, whereas image embeddings predict text fMRI to a far lesser extent, so we believe that incomplete captions cannot explain the asymmetry.

4. + 5. Connection to representation learning. As part of this submission to the “applications to neuroscience & cognitive science” primary area, improvements of VLM capabilities are out of the scope of this paper. Our main novelty is the presented 2x2 cross-modal encoding framework, and the asymmetry across the cross-modal prediction settings it allowed us to discover. We argue that this cross-modal comparison of similarities/differences in human neural representations and VLM-based ones is a meaningful contribution to the research community at the intersection of neuroscience and AI (NeuroAI), and to representation learning more broadly. Moreover, the additional analyses performed as part of this discussion period further strengthen the validity of the approach. Nonetheless, we now explicitly name potential directions for future research on improving VLMs in the discussion: “For VLM research, the asymmetry exposes a measurable shortcoming of the tested image encoders, and our paired paradigm offers a brain-grounded benchmark for future work on training objectives that aim to close it.”

References Naselaris, T., Kay, K. N., Nishimoto, S., & Gallant, J. L. (2011). Encoding and decoding in fMRI. NeuroImage, Multivariate Decoding and Brain Reading, 56(2), 400–410.

Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021). The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45).

Doerig, A., Kietzmann, T. C., Allen, E., Wu, Y., Naselaris, T., Kay, K., & Charest, I. (2025). High-level visual representations in the human brain are aligned with large language models. Nature Machine Intelligence, 7(8), 1220–1234.

Thank you for your thoughtful response. I will make my decision in the next step.

Thank you for acknowledging our response, we appreciate your time!

Reviewer_urnEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Previous studies have shown that multimodal models can predict brain activity. However, many datasets either present stimuli from only one modality or present images and sounds simultaneously, making it difficult to determine whether model–brain correspondence reflects information shared across modalities. This study uses paired scene images and textual descriptions presented separately to distinguish within-modal from cross-modal prediction. The paper addresses an important question. The paired-stimulus design is valuable, the four experimental conditions are clearly organized, and the analyses across models and in the appendix strengthen confidence in the findings. Its main contribution is the identification of an asymmetry in cross-modal prediction that warrants further investigation. However, the current residualization analyses, condition comparisons, and embedding-mapping results do not adequately distinguish shared semantics from alternative explanations, including differences in target-space predictability, information loss during feature extraction, and unequal information content across stimulus modalities. In particular, the inconsistency between the absolute effects of residualization and the claims in the main text, potential pseudoreplication across correlated models, and direct comparisons across different target embedding spaces weaken the central argument.

Critical strengths

1.The experimental design is clear. The four combinations of within-modal and cross-modal prediction provide a systematic framework for examining asymmetry between prediction directions.

2.Paired images and texts are presented separately, and individual participants do not encounter both versions of the same content. This reduces confounds associated with simultaneous presentation and repeated exposure.

3.The asymmetry is observed across multiple vision–language model families and unimodal baselines. Additional analyses examining voxel thresholds, common voxel sets, individual participants, and layer depth in selected models further support the robustness of the main finding.

Critical weaknesses

1.The introduction motivates the study through modality invariance in semantic cognition and presents cross-modal encoding as a direct test of representations shared by models and the brain. However, separate encoding mappings are fitted for the four conditions. The ability of embeddings from another modality to predict brain activity therefore demonstrates the presence of information that supports prediction; it does not establish that the correspondence reflects the same semantic subspace or neural code. Moreover, the paper invokes convergence zones, semantic hubs, and embodied cognition without specifying the distinct predictions these accounts make for the four conditions. Consequently, the current experiment offers limited discrimination among these theoretical explanations. Without direct evidence concerning neural representational geometry or semantic specificity, the main conclusions should be restricted to cross-modal predictive correspondence.

2.Although the design addresses simultaneous presentation, captions typically describe only a subset of the objects, actions, and relations present in a scene. Images contain substantial information that is not captured by their descriptions, and the authors acknowledge their richer semantic content in the discussion. The observed directional asymmetry could therefore reflect selective description, information compression, or unequal semantic coverage. Furthermore, preventing participants from seeing both versions of the same content reduces repeated-exposure effects but also means that the study does not obtain paired neural responses to the same content across modalities within individual participants. The authors should provide further details about the reading and image-viewing tasks and the associated behavioral results, quantify caption coverage of objects, actions, and relations, and repeat the analyses after stratifying stimuli by semantic alignment and scene complexity.

3.The authors state that unimodal models were included to separate the contribution of joint vision–language training. However, the same asymmetry is also observed in unimodal visual and language models. This broadens the empirical scope of the phenomenon but does not show that it depends on joint training. The comparisons additionally differ in architecture, parameter count, training data, and training objectives, making attribution to a particular training mechanism difficult. The paper should clarify whether its primary contribution concerns a general asymmetry in prediction or a specific effect of joint training on model–brain correspondence. If the latter is central, stronger controls are needed, such as unimodal and jointly trained models with matched architectures or comparisons of the same model before and after joint training, together with a direct test of the interaction between model type and prediction direction.

4.The main text states that removing components of image embeddings predictable from text affects cross-modal prediction significantly more than within-modal prediction. However, Figure 21 shows a substantially larger absolute reduction in the image–image condition than in the image–text condition. The results therefore do not support a general claim that cross-modal performance decreases more strongly for both input modalities. Moreover, the large relative reduction in the image–text condition is sensitive to its near-zero baseline. Small denominators inflate percentage changes, particularly when correlations cross zero after residualization. The authors should prespecify the primary measure of change and, separately for each input modality, test the difference between cross-modal and within-modal performance reductions, reporting effect sizes and confidence intervals. The figure already includes tests of performance changes; the concern is their interpretation and the unit of statistical inference. Differences in final scores after residualization should not be treated as differences in performance reductions.

5.The paper depicts the component linearly predictable from the other modality as a shared “intersection” and describes the residuals as containing only modality-specific information. However, residuals obtained from ridge regression with a fixed regularization parameter may retain shared components that are insufficiently fitted, as well as nonlinear correspondences. Their composition also depends on feature scaling, the variance spectrum, and effective rank. Residualization additionally alters the input covariance structure and the regularization conditions of the subsequent encoding model. Preserving dimensionality therefore does not ensure that performance changes exclusively reflect the removal of shared information.

6.Retaining a fixed set of 50 principal components preserves directions of high variance, which are not necessarily those most informative for predicting activity in language or semantic brain regions. Averaging spatial patch representations may also attenuate information about object relations and local semantic structure. The layer-depth analysis is useful, but it covers only Qwen3.5-VL 9B. The common-voxel analysis addresses differences in evaluation voxels, whereas the nonlinear MLP analysis concerns prediction between embedding spaces. None directly tests information loss introduced by PCA or pooling. The authors should compare different PCA dimensionalities, explained-variance thresholds, and ridge regression using the full feature space. Where applicable, they should also compare pooling strategies, examine layer selection across model families, and include analyses of predefined semantic and language regions of interest. All such choices should be made using training or validation data.

7.The finding that most models do not achieve statistical significance in the image–text condition does not demonstrate that this direction contains no predictable information. Indeed, the main text reports three significant models. Given the limited number of participants, the precision of the effect estimates also requires clarification. Similarly, counting significant models cannot substitute for comparing the amount of predictable information. The authors should report confidence intervals for the key effects. If they wish to argue that prediction in one direction is negligible, they should define a smallest effect size of interest in advance and conduct an equivalence test. Comparisons of residual predictive information should use continuous effect sizes, and the findings should be validated in an independent fMRI dataset with paired image and text stimuli. Until such validation is available, the conclusions should remain limited to the models, stimuli, and analytical procedures examined here.

Questions for authors

Retaining a fixed set of 50 principal components preserves directions of high variance, which are not necessarily those most informative for predicting activity in language or semantic brain regions. Averaging spatial patch representations may also attenuate information about object relations and local semantic structure.

3 comments

We thank the reviewer for their thoughtful and engaging review, please find our point-by-point reply below.

1. Narrowing the scope of the paper. First, to clarify, we do not intend to offer a precise delineation between the theoretical explanations (convergence zones, semantic hubs, and embodied cognition) with our experiments, we rather list those for a general contextualization of our work. We therefore agree that the framing of the contribution of our work can be narrowed down to cross-modal predictive correspondence. We have now made this more explicit in the related work section: “These accounts provide complementary background to our research question: rather than discriminating between the different theories, we leverage parallel text-image fMRI data to test in how far the information linearly shared across text and image embeddings supports the predictive correspondence between VLMs and brain activity.”, and further mentioned this in the discussion: “Accordingly, our results do not discriminate between convergence-zone, semantic-hub, and grounded-cognition accounts of how the brain integrates modalities, but show which VLM embeddings carry information that predicts brain activity across modalities.”.

2. Behavioral results and imbalanced information content across text and images. Details on the behavioral task of the original study and its results are now provided in Appendix A.10. To verify that participants were actively processing the presented stimuli, they completed a yes/no recognition task after each session, distinguishing 50 of the presented stimuli from 50 lure stimuli (25 texts and 25 images each). Every participant performed significantly above chance (mean recognition accuracy 0.844 ± 0.039; p<0.001). Moreover, we agree that inherent differences between shorter captions and more detailed scenes could indeed be a limiting factor in our study design, and we now explicitly acknowledge this in the discussion: “This difference in information content also applies within our dataset: captions describe only a subset of the objects, actions, and relations depicted in a scene, so unequal semantic coverage could contribute to the asymmetry.”. To reduce the information imbalance across text and images, we used OLMo 3.1 32B Instruct to generate more detailed captions. The instruction for the caption generation was to describe all actions in the provided COCO-Actions scene graph for a given image (see Appendix A.15 for more details). We quantify the caption coverage of annotated objects, actions and person-object interactions for the original and the detailed captions:

Caption coverage (Appendix A.15, Table 7).

CaptionsMedian wordsObject categoriesActionsPerson-object interactions
Original949%13%6%
Detailed13994%100%99%

Then, we use the more detailed captions to repeat the cross-modal embedding prediction (research question RQ3) and the text embedding-based neural encoding experiments (RQ1): text embeddings->text fMRI, and text embeddings->image fMRI. For the cross-modal embedding prediction (RQ3), the detailed captions reduce the asymmetry between text=>image and image=>text embedding prediction (difference in r between the modalities is reduced from 0.093 to 0.048). For neural encoding (RQ1), detailed captions predict image-evoked activity slightly better (0.045 vs. 0.040, see Table 8).

Neural encoding (RQ1, Appendix A.15, Table 8). The table shows the mean r/r_NC across the 14 models with a text encoder; in parentheses, the number of models that are significant. With detailed captions, text embeddings predict image-evoked activity slightly better, so the encoding asymmetry becomes larger rather than smaller. Text-evoked activity, which was elicited by the original captions, is predicted much less well.

ConditionOriginal captionsDetailed captions
Text embeddings → image fMRI0.040 (14/14)0.045 (14/14)
Text embeddings → text fMRI0.032 (14/14)0.008 (11/14)
Image embeddings → text fMRI (reference, 10 VLMs)0.005unchanged

Embedding prediction (RQ3, Appendix A.15, Table 9). Detailed captions roughly halve the difference in cross-modal embedding prediction. Richer captions therefore may reduce the asymmetry in the embeddings themselves, but not in neural encoding. Image embeddings, which already contain rich scene content, still predict caption-evoked activity only weakly.

CaptionsText⇒imageImage⇒textDifference
Original0.3140.220+0.093
Detailed0.3060.258+0.048

3. Clarifying that we are not making specific claims about the effect of joint vision-language training on neural encoding. Our primary contribution concerns a general asymmetry in cross-modal prediction rather than a specific effect of joint training on model–brain correspondence. Since the latter is not central to our claims, unimodal and jointly trained models with matched architectures and training data are not required to support them, and training such models is out of scope for this paper. We now make this distinction more explicit throughout the paper. In the Introduction, we now mention: “to identify which embeddings, if any, carry information that predicts brain activity across modalities, irrespective of whether they stem from jointly trained or unimodal models.” In the Methods (see Section 3.2), we now clarified the purpose of the unimodal baselines: “To test whether the asymmetry is specific to jointly trained vision-language models or not, we additionally evaluate unimodal image embeddings [...] and unimodal language embeddings [...].”. Lastly, in the discussion we state: “Moreover, our unimodal baselines differ from the VLMs in architecture, size, training data, and objective: they show that the asymmetry is not specific to jointly trained models, but attributing it to joint vision-language training would require matched models or a comparison before and after joint training. Our primary contribution is thus a general asymmetry in cross-modal prediction rather than a specific effect of joint training on model–brain correspondence.”

4. + 7. Prespecified measure of change and statistical conclusions. We agree that relative changes are unstable for near-zero baselines such as the image->text condition, and we now pre-specify the absolute change as the primary measure in the Methods (Section 3.4): “Importantly, we pre-specify the absolute change (delta r/r_NC = residualized - baseline) as the primary measure of change, because relative changes are unstable for baselines close to zero.”. We treat this absolute change as a continuous effect size in units of encoding accuracy, which we report for every model and condition, together with its variability across participants, in Appendix A.8. We now revised our claims about residualization throughout the paper, and reported relative changes in Appendix A.8 for completeness only. Regarding the image->text direction, we clarify that we do not claim that it contains no predictable information, only that image embeddings “predict text-evoked brain activity only weakly”. We also clarify the limitations in the discussion: “Non-significant effects should not be read as evidence of absence, comparisons across models describe the consistency of an effect rather than supporting population inference, and subject-level tests are limited by the number of participants.”

5. Nonlinear correspondences. Our comparison of linear and non-linear models for cross-modal prediction (Appendix A.9) yielded minimal differences: The mean accuracies of the two tested directions are nearly unchanged (0.469 vs. 0.457 and 0.378 vs. 0.379), suggesting that non-linearly predictable information between the two modalities may only play a minor role. Nonetheless, we now acknowledge factors beyond linear predictability in the discussion: “However, factors beyond linear predictability may explain semantic alignment between modalities, such as the geometry, dimensionality, or covariance structure of the representations. In addition, semantic correspondences may be non-linear, although a non-linear predictor changes our embedding prediction results only minimally (see Appendix A.9). Likewise, residualization removes linearly shared variance whether or not it is semantic, may leave non-linear or under-fitted shared information, and alters the covariance of the encoding input, so the drop in encoding accuracy may not be a pure measure of shared semantic information.”

6. PCA, pooling, ROI-based analyses. For the sake of feasibility, we restricted the PCA, pooling and layer analyses to one representative model, Qwen3.5-VL 9B. In Appendix A.13, we now varied the number of principal components as well as the explained-variance threshold (50%, 75%, 90% and 95%), and additionally encoded from the full feature space (see Appendix A.13, Figure 26, the explained variance of all models is shown in Figure 25). The asymmetry is preserved in every setting: Within-modality and text->image accuracies are stable between 10 and 50 components and similar in the full feature space. Image->text accuracy is highest with few components (0.013–0.015), but stays far below text->image accuracy.

PCA analyses (Appendix A.13).

ConditionRange across settingsFull feature space
Text embeddings → image fMRI0.021–0.0400.039
Image embeddings → text fMRI0.006–0.0150.010

For image representations, we now compared mean pooling over patch tokens (main analysis), max pooling, and a 2×2 spatial grid that retains the coarse spatial layout of the scene. For text representations, we now compared mean pooling (main analysis), the last token, and max pooling. The asymmetry holds for every strategy, including the spatial grid-based one.

Pooling (Appendix A.14).

ConditionRange across pooling strategies
Text embeddings → image fMRI0.028–0.041
Image embeddings → text fMRI0.002–0.014

In Appendix A.12, we replicate the main analysis in five predefined regions of interest (ROIs), a subset of the visual and language regions analyzed in previous studies (Oota et al., 2023; 2025). More specifically, we added an ROI-based analysis using five predefined bilateral regions from the Glasser HCP-MMP1 parcellation, defined as in Oota et al. (2023, 2025): early visual cortex (EVC), lateral occipital complex (LOC), posterior temporal lobe (PTL), angular gyrus (AG), and the anterior temporal lobe (ATL). Importantly, each ROI includes all voxels with a positive noise ceiling, different from a top-%-based choice. The ROI-specific results highlight that the reported asymmetry from the main analysis (research question RQ1) is present in every ROI (see Appendix A.12, Figure 24). Text embeddings predict image-evoked activity at above-chance level for all 14 models in EVC, LOC, PTL and AG. However, image embeddings predict text-evoked activity at above-chance level for at most one of 14 models in any ROI. As a result, we provide clear evidence for the failure of image embeddings to predict text-evoked activity not being an artifact of a predominantly occipital voxel selection.

ROI analysis (RQ1, Appendix A.12, Figure 24). The table shows the mean r/r_NC across the 14 models per ROI; in parentheses, the number of models that are significant (BH-FDR correction across ROI x model x condition tests).

ROIImage emb. → image fMRIImage emb. → text fMRIText emb. → image fMRIText emb. → text fMRI
EVC0.159 (14/14)0.009 (0/14)0.055 (14/14)0.046 (14/14)
LOC0.201 (14/14)-0.003 (0/14)0.131 (14/14)0.034 (14/14)
PTL0.050 (14/14)0.003 (1/14)0.024 (14/14)0.055 (14/14)
AG0.080 (14/14)-0.003 (0/14)0.051 (14/14)0.057 (14/14)
ATL0.013 (9/14)0.001 (0/14)0.010 (2/14)0.056 (14/14)

References Oota, S. R., Gupta, M., & Toneva, M. (2023). Joint processing of linguistic properties in brains and language models. Advances in Neural Information Processing Systems, 36, 18001–18014.

Oota, S. R., Pahwa, K., Marreddy, M., Singh, M. K., Gupta, M., & Surampudi, B. R. (2025). Multi-modal brain encoding models for multi-modal stimuli. Proceedings of the Thirteenth International Conference on Learning Representations.

Reviewer_CKgnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This study uses a paired text-image fMRI dataset to test cross-modal encoding with VLM embeddings, and reports that text embeddings predict image-evoked responses but not the reverse.

Critical strengths

The paired dataset, where no subject sees both forms of a stimulus, is a good fit for testing cross-modal encoding.

Critical weaknesses

From figures 9-16, it appears that most of the selected voxels for both text and image modalities are in the occipital areas. For text this is likely a reflection of visually evoked responses to reading the text caption. But when image features are used to predict this response, the matched image (rather than the image of the caption) is used to extract the features and naturally it will not be able to predict the responses because the selected voxels are responding to something else. The authors should provide results for other selections of the voxels, including ones that are focused on specific regions and in general better justify their choices.

Also the authors should provide the details on the noise ceiling for all the subjects and modalities and present not only normalized encoding accuracies but also the raw r values. Even the normalized encoding accuracies are very small so it’s good to see what the raw values are. This raises another concern that the dataset size may be just too small and the accuracies are in general too low to make any strong claims.

The abstract starts with “Vision-language models (VLMs) have been shown to predict neural responses to visual stimuli better than unimodal models …” What exactly is the evidence for this? If you look at the winning entries from the 2023 Algonauts challenge on predicting visually evoked fMRI response, the winning entries used vision models such as DINO to achieve SOTA results.

Questions for authors

See above.

4 comments

We thank the reviewer for their comments, please find our point-by-point reply below.

1. Voxel selection and regions of interest. Alternative voxel selection choices are provided in Appendices A.3 and A.4, including the top 40% and top 60% most reliable voxels, and a single voxel set shared by the two fMRI modalities. The results indicate that the analyses are robust to the choice of voxels, in particular, the shared voxel set shows that our findings are not a result of text- versus vision-specific voxel selection procedures. We now also better justify the choice of using voxels with the highest noise ceiling in the main analysis (see Section 3.3): “We select voxels by their noise ceiling because it identifies the voxels with the most reliable stimulus-driven signal. The selected voxels, shown in Appendix A.5 for each subject, are dominated by occipital and occipitotemporal cortex. In Appendix A.12, we therefore additionally replicate the main analysis in five predefined regions of interest (ROIs), a subset of the visual and language regions analyzed in previous studies (Oota et al., 2023; 2025).”. More specifically, we added an ROI-based analysis using five predefined bilateral regions from the Glasser HCP-MMP1 parcellation, defined as in Oota et al. (2023, 2025): early visual cortex (EVC), lateral occipital complex (LOC), posterior temporal lobe (PTL), angular gyrus (AG), and the anterior temporal lobe (ATL). Importantly, each ROI includes all voxels with a positive noise ceiling, different from a top-%-based choice. The ROI-specific results highlight that the reported asymmetry from the main analysis (research question RQ1) is present in every ROI (see Appendix A.12, Figure 24). Text embeddings predict image-evoked activity at above-chance level for all 14 models in EVC, LOC, PTL and AG. However, image embeddings predict text-evoked activity at above-chance level for at most one of 14 models in any ROI. As a result, we provide clear evidence for the failure of image embeddings to predict text-evoked activity not being an artifact of a predominantly occipital voxel selection.

ROI analysis (RQ1, Appendix A.12, Figure 24). The table shows the mean r/r_NC across the 14 models per ROI; in parentheses, the number of models that are significant (BH-FDR correction across ROI x model x condition tests).

ROIImage emb. → image fMRIImage emb. → text fMRIText emb. → image fMRIText emb. → text fMRI
EVC0.159 (14/14)0.009 (0/14)0.055 (14/14)0.046 (14/14)
LOC0.201 (14/14)-0.003 (0/14)0.131 (14/14)0.034 (14/14)
PTL0.050 (14/14)0.003 (1/14)0.024 (14/14)0.055 (14/14)
AG0.080 (14/14)-0.003 (0/14)0.051 (14/14)0.057 (14/14)
ATL0.013 (9/14)0.001 (0/14)0.010 (2/14)0.056 (14/14)

2. Noise ceiling per participant and raw r values. We now report the modality-specific noise ceiling for all subjects in Appendix A.11 (Table 5). Across participants, the mean noise ceiling of the selected voxels is 0.630 for image-evoked responses and 0.661 for text-evoked responses.

Noise ceilings per participant (Appendix A.11, Table 5).

SubjectImage: voxelsImage: mean r_NCImage: max r_NCText: voxelsText: mean r_NCText: max r_NC
sub-0212,3260.6030.87213,6250.6460.898
sub-0311,3530.6360.96111,6420.6690.960
sub-048,9680.6090.93510,3120.6500.952
sub-0512,9660.6490.96414,3980.6830.974
sub-0611,6890.6030.93612,3550.6300.963
sub-0711,0810.7020.96910,5080.7280.973
sub-088,1330.5760.8928,8870.5970.912
sub-0910,5230.6600.96310,3800.6840.965
Mean10,8800.6300.93611,5130.6610.950

Moreover, the raw r values for the main analysis are now listed in Appendix A.11 (Table 6), for every model and condition:

Raw r values (Appendix A.11, Table 6).

ConditionRaw rr/r_NC
Image embeddings → image fMRI0.0560.079
Text embeddings → image fMRI0.0280.040
Text embeddings → text fMRI0.0230.032
Image embeddings → text fMRI0.0030.004

These values are in the range reported in prior work. For example, Tang et al. (2023) list cross-modal transfer accuracies of r=0.023 (story->movie) and r=0.010 (movie->story) in a larger naturalistic dataset. Further, we now better emphasize the limitations of the dataset in the discussion: “It relies on a single dataset of eight participants and 252 stimuli with low absolute encoding accuracies, so our conclusions are restricted to the models, stimuli, and analyses examined here.”

3. Prior work on unimodal models and VLMs for neural encoding. Our original claim was based on prior work reporting advantages of multimodal over unimodal models in neural encoding, as stated in the introduction (e.g., Tang et al., 2023; Oota et al., 2025; Bavaresco et al., 2026). We agree on adding nuance to this claim, which in any case does not invalidate the motivation for our study, and have weakened it in the abstract and introduction, which now also acknowledge the strong performance of vision-only models, e.g., in the Algonauts 2023 challenge. The abstract now states: “Vision-language models (VLMs) have been reported to predict neural responses to visual stimuli better than unimodal models in some settings, which has been interpreted as evidence that VLMs and the human brain converge on shared, cross-modal representations.”, and the introduction states: “Multimodal neural encoding studies have begun to examine this question, motivated by findings that VLMs can match or outperform unimodal baselines in neural encoding (Tang et al., 2023; Oota et al., 2025; Bavaresco et al., 2026), although vision-only models remain highly competitive for predicting responses to visual stimuli (Conwell et al., 2024). For instance, the top-performing models of the Algonauts 2023 challenge are built on vision-only backbones (Gifford et al., 2023; Adeli et al., 2023).”.

References Tang, J., Du, M., Vo, V., Lal, V., & Huth, A. (2023). Brain encoding models based on multimodal transformers can transfer across language and vision. Advances in Neural Information Processing Systems, 36, 29654–29666.

Oota, S. R., Gupta, M., & Toneva, M. (2023). Joint processing of linguistic properties in brains and language models. Advances in Neural Information Processing Systems, 36, 18001–18014.

Oota, S. R., Pahwa, K., Marreddy, M., Singh, M. K., Gupta, M., & Surampudi, B. R. (2025). Multi-modal brain encoding models for multi-modal stimuli. Proceedings of the Thirteenth International Conference on Learning Representations.

Bavaresco, A., de Heer Kloots, M., Pezzelle, S., & Fernández, R. (2026). Vision-Language Models Align with Human Neural Representations in Concept Processing. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), 3255–3274.

Conwell, C., Prince, J. S., Kay, K. N., Alvarez, G. A., & Konkle, T. (2024). A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. Nature Communications, 15(1), 9383.

Gifford, A. T., Lahner, B., Saba-Sadiya, S., Vilas, M. G., Lascelles, A., Oliva, A., Kay, K., Roig, G., & Cichy, R. M. (2023). The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenes (arXiv:2301.03198). arXiv.

Adeli, H., Minni, S., & Kriegeskorte, N. (2023). Predicting brain activity using Transformers (p. 2023.08.02.551743). bioRxiv.

I thank the authors for providing the raw r values and the other analyses.

My concern was that maybe this specific dataset does not lend itself to studying the asymmetry you are interested in.

In the image case, the subject saw the images, and therefore the text features describing the visual input can predict the activity. 

In the caption case, the subject read the captions, so their brain response was dominated by the visual features of the caption. You can see this even from the fact that text embeddings cannot even predict the text fMRI as well predicting image fMRI.

So if there is some asymmetry effect in predicting neural activity from the other modality, albeit very small as is shown in the raw r values, how are we sure this is not due to the way the experimetns were conducted as I pointed out above.

We thank the reviewer for following up on our response, and would like to address the mentioned concern as follows:

1. The observed asymmetry holds outside the visual cortex, too. As shown in the ROI-specific analyses in the previous response, text embeddings predict text-evoked fMRI, while image embeddings fail to predict text-evoked fMRI in the PTL and AG. (And in the PTL, text embeddings do predict text-evoked fMRI better than image-evoked fMRI.) A response dominated by the visual form of the captions would not be able to explain the asymmetry in these regions.

2. The resulting raw r values are in the same order of magnitude as in previous work. While the number of stimuli is smaller compared to other naturalistic fMRI datasets, the similar range of raw r values likely stems from the increased number of repetitions (6x) per stimulus, which enhances data quality substantially.

3. (Clarification) Every participant saw either an image or its caption, but never both. This deliberate study design choice rules out confounding imagery/memorization effects, which reviewer urnE highlights as a key strength.

Taken together, we therefore believe that the employed dataset is well suited to study the asymmetry.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.