- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether varying the instructions given to instruction-tuned multimodal large language models (IT-MLLMs) elicits different representations of the same naturalistic stimulus, and how these representations predict activity in different brain regions. The authors evaluate several video instructions and audio instructions. Brain activity is predicted from hidden states associated with generated response tokens.
The evaluation includes three reported matched pairs of pretrained and instruction-tuned models, alongside a broader comparison involving several baselines. The authors report higher brain predictivity for instruction-tuned models, instruction-conditioned representational structure, and different regional prediction patterns across instructions.
Critical strengths
- The reported matched comparisons within Qwen-2-VL, Qwen-2.5-VL, and InternVL3 provide a stronger basis for studying differences associated with instruction tuning than comparisons between heterogeneous models alone.
- The paper evaluates video and audio IT-MLLMs alongside pretrained models, multimodal reference models, and unimodal baselines.
- The paper presents instruction-specific regional prediction patterns, cortical distributions of the most predictive model layers, and participant-level consistency analyses.
Critical weaknesses
W1: The findings that instruction tuning can improve brain predictivity, and that different instructions or semantic descriptions of the same stimulus can yield different regional prediction patterns, are not new in themselves. For example, the following three studies are already cited in the manuscript.
- Aw et al. (COLM 2024) compared pretrained and instruction-tuned LLMs and reported improvements in brain alignment across multiple fMRI datasets. Thus, the general observation that instruction tuning can improve brain alignment has already been established.
- Oota et al. (ICLR 2025) applied different instructions to the same images, predicted brain activity from representations of generated responses, and compared the best-performing instructions across voxels. They reported different regional prediction patterns for captioning and image-understanding instructions in higher-level and early visual areas. Therefore, instruction-dependent regional differentiation was already observed in prior unimodal-stimulus work.
- Nakagi et al. (EMNLP 2024) examined annotations of the same naturalistic videos at multiple semantic levels and identified distinct regional encoding patterns. Although that study did not manipulate prompts in the same manner, it provides a precedent for relating different semantic descriptions of the same stimulus to different brain regions.
The contribution therefore needs to be established through what the extension to naturalistic video and audio reveals beyond these precedents, rather than through these basic observations themselves.
W2: I acknowledge the empirical extension of applying diverse instructions to multiple video and audio models and evaluating their correspondence with naturalistic-viewing fMRI. However, a combination of models, instructions, and stimuli not having been evaluated previously does not, by itself, identify an important unresolved scientific problem. The paper broadly asks which instructions predict which regions, but it is unclear which competing explanations these comparisons distinguish or how the results update our understanding of brains or models. Much of the interpretation consists of explaining the observed maps in terms of already established functional differentiation.
W3: The low RSA correlations with MiniLM/MPNet show weak correspondence with those encoders’ semantic geometry; they do not establish independence from instruction wording. The MLLMs may still depend on their own representations of the instruction text. Comparing instruction-only and video-conditioned representations within each evaluated checkpoint would therefore be an informative control, while accounting for differences in generated response content. Furthermore, Table 4 assigns one prompt to each task, confounding task identity with prompt identity. The within-video task-label permutation test detects structure associated with the chosen instruction conditions beyond video identity, but does not test generalization across different wordings of the same task.
Questions for authors
Q1: Please identify the specific scientific conclusion established for the first time in this paper relative to the previous studies.
Q2: Beyond evaluating a broader set of instructions and models, which hypothesis can be supported or rejected for the first time on the basis of this paper’s results?
Q3: Does the weak correspondence between instruction-only and video-conditioned representations in IT models persist when the instruction-similarity matrix is computed using each evaluated MLLM checkpoint itself, rather than MiniLM/MPNet? Relatedly, do the representational structure and brain-prediction patterns remain stable across multiple paraphrases of the same task?
7 comments
W2/Q2. Beyond evaluating a broader set of instructions and models, which hypothesis can be supported or rejected for the first time on the basis of this paper’s results?
We thank the reviewer for this comment. Our central hypothesis is that instruction-tuning enables linguistic task instructions to organize representations of identical audiovisual stimuli in ways that predict distinct patterns of brain activity, beyond generic prompting or generated-response effects. Our results challenge three competing explanations that were not directly tested in the cited studies.
(1) Generic prompting alone--insufficient. If brain alignment gains reflected generic prompting, random prompts would produce comparable effects. Task-specific instructions improve brain alignment over the no-prompt baseline (0.746 vs. 0.639 for Qwen2.5-VL), whereas random prompts do not reproduce the layer-wise cortical hierarchy.
(2) Stimulus identity or architecture alone--insufficient. Across three matched pretrained–instruction-tuned model pairs, significant task-conditioned representational structure beyond stimulus identity emerges only after instruction-tuning (FDR \leq). This supports a role for instruction-tuning, although the contribution of post-training data cannot be isolated.
(3) Generated responses alone--insufficient. Task-conditioned structure persists across prompt paraphrases and before response generation. Pre-generation representations also retain regional differences in brain predictivity, whereas these differences are substantially attenuated with teacher-forced identical responses ($(|\Delta| \leq 0.014$); see our response to Reviewer CFKc, W4). Generated responses alone therefore cannot explain the observed effects.
Together, these results support task-conditioned internal representational organization associated with regionally differentiated brain predictivity, beyond explanations based solely on generic prompts, fixed stimulus identity, or generated text. We do not claim that passive viewers perform the instructed tasks or that regional predictivity identifies task-specific neural computations.
W3/Q3. Does the weak correspondence between instruction-only and video-conditioned representations in IT models persist when the instruction-similarity matrix is computed using each evaluated MLLM checkpoint itself, rather than MiniLM/MPNet? Relatedly, do the representational structure and brain-prediction patterns remain stable across multiple paraphrases of the same task?
We thank the reviewer for this suggestion. To examine whether MiniLM/MPNet inadequately capture instruction semantics, we repeated the RSA using instruction representations extracted from the evaluated checkpoints for two matched pairs: Qwen-2.5-VL-7B and InternVL3-8B, each instruction-tuned versus pretrained.
-
For each model, we built a text-only similarity matrix over the 13 task instructions from hidden states at the final prompt position without video input, and compared it with the video-conditioned similarity matrix from mean-pooled generated-response representations at each layer (Fig. 12 and Fig. 13, App. H).
-
The weak correspondence is not specific to MiniLM/MPNet.
- For Qwen-2.5-VL-7B, the instruction-tuned model shows low correspondence with its own instruction geometry (mean RSA r = 0.19; range 0.12–0.26; 0.13–0.16 with MiniLM/MPNet), whereas the pretrained model shows higher correspondence (mean r = 0.39; peak about 0.55 at layer 12) and exceeds the instruction-tuned model at every layer.
- InternVL3-8B shows the same pattern (r = 0.22 instruction-tuned vs. 0.43 pretrained). The pretrained values are lower than the MiniLM/MPNet estimates (0.69–0.78), so the contrast between pretrained and instruction-tuned models holds in direction with a smaller magnitude.
- Both sets of values are reported in the revised manuscript (L278–284 and App. H).
-
A MLLM's text-only instruction representation encodes the requested task as well as its wording. The 78-instruction analysis (13 tasks × 6 paraphrases; see Reviewer CFKc W1) separates the two: task-conditioned organization holds across paraphrases, including formulations without shared content words, and under response-length, teacher-forced and pre-generation controls.
-
Brain-prediction stability. A paraphrased Narrative Understanding instruction gives regional brain alignment matching the original in AG, PCC and dmPFC (absolute differences ≤ 0.042, within participant SEM; see our response to Reviewer CFKc, W2).
Thank you for the detailed responses and additional analyses. I now understand the main claimed novelty to concern how instruction tuning shapes task-conditioned representations, rather than a new conclusion about task-specific neural computations.
Although I am not a specialist in task/function-vector research, my understanding is that the literature on task-conditioned representations in LLMs/VLMs is considerably more extensive than the corresponding brain-encoding literature. Relevant examples include earlier work on internal task representations [1,2], as well as more recent studies of instruction-derived task representations in VLMs [3] and instruction-based function vectors across post-training stages [4]. However, I could not find an explicit discussion of this literature in the manuscript.
While I recognize the differences in methods and settings, could you clarify which specific scientific conclusion goes beyond these prior findings, and what additional understanding the brain-encoding analyses provide? This would help me assess the novelty emphasized in your response.
[1] Hendel et al. (2023). In-Context Learning Creates Task Vectors. Findings of EMNLP.
[2] Todd et al. (2024). Function Vectors in Large Language Models. ICLR.
[3] Luo et al. (2025). Vision-Language Models Create Cross-Modal Task Representations. ICML.
[4] Davidson et al. (2025). Do different prompting methods yield a common task representation in language models? NeurIPS.
W1/Q1. Please identify the specific scientific conclusion established for the first time in this paper relative to the previous studies.
We thank the reviewer for this comment. The two observations the reviewer lists are established, and we already cited the previous studies. Our contribution is the following three results, none of which is reported in the cited studies.
Relation to prior work.
Together, these results show how linguistic instructions organize multimodal representations, how instruction semantics influence brain predictivity, and which effects persist independently of generated responses. The revised Introduction (Lines 75-77) and Related Work (Lines 942-947) will state these contributions explicitly.