acceptodds
Under review as a conference paper at ICLR 2027

Task-conditioned probing of instruction-tuned multimodal LLMs: Region-specific brain alignment patterns under naturalistic stimuli

Abstract

Multimodal large language models (MLLMs) align better with brain activity than unimodal models. Instruction-tuned (IT) MLLMs, in which a language instruction governs how visual and auditory input is processed, produce task-specific representations. Yet their brain alignment has been evaluated only under unimodal stimuli, while studies using multimodal stimuli have relied on non-instruction-tuned models. It therefore remains unclear whether, under naturalistic multimodal input, instruction-conditioned representations capture task-related structure beyond the surface semantics of the instruction text. We address this by predicting fMRI responses recorded during naturalistic movie watching (video with audio) from MLLM representations. Across three matched pairs of pretrained and IT MLLMs given identical instructions (the pretrained models prompted via in-context learning (ICL)), IT models align better with brain activity at every layer. In all three pairs, ICL representations track instruction-text semantics (–), whereas IT representations are largely decoupled from wording (–) and instead exhibit significant task-conditioned organization throughout the network. This does not mean IT models ignore the instruction: they respond to which task is asked, not to how it is worded. Surprisingly, for identical movie stimuli, different instructions yield distinct regional patterns of brain predictivity. Narrative understanding in the angular gyrus and posterior cingulate cortex, spatial understanding in parietal and scene-selective regions, and emotion understanding in the middle frontal gyrus. These preferences are broadly consistent across participants. Eight video and two audio IT-MLLMs replicate this instruction-dependent regional structure. Together, these findings show that instructions with IT-MLLMs offer a controlled way (model organisms) to derive many task-specific representations from a single stimulus, revealing which cortical regions are best predicted by which task. Because the neural data are held fixed, this provides a low-cost, model-side source of hypotheses to complement task manipulation in neuroimaging, which requires separate acquisitions per task.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain encoding, fMRI, multimodal instruction-tuned models, Video LLMs, Audio LLMs, multi-modal stimuli, Transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6LNnOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether varying the instructions given to instruction-tuned multimodal large language models (IT-MLLMs) elicits different representations of the same naturalistic stimulus, and how these representations predict activity in different brain regions. The authors evaluate several video instructions and audio instructions. Brain activity is predicted from hidden states associated with generated response tokens.

The evaluation includes three reported matched pairs of pretrained and instruction-tuned models, alongside a broader comparison involving several baselines. The authors report higher brain predictivity for instruction-tuned models, instruction-conditioned representational structure, and different regional prediction patterns across instructions.

Critical strengths
  • The reported matched comparisons within Qwen-2-VL, Qwen-2.5-VL, and InternVL3 provide a stronger basis for studying differences associated with instruction tuning than comparisons between heterogeneous models alone.
  • The paper evaluates video and audio IT-MLLMs alongside pretrained models, multimodal reference models, and unimodal baselines.
  • The paper presents instruction-specific regional prediction patterns, cortical distributions of the most predictive model layers, and participant-level consistency analyses.
Critical weaknesses

W1: The findings that instruction tuning can improve brain predictivity, and that different instructions or semantic descriptions of the same stimulus can yield different regional prediction patterns, are not new in themselves. For example, the following three studies are already cited in the manuscript.

  • Aw et al. (COLM 2024) compared pretrained and instruction-tuned LLMs and reported improvements in brain alignment across multiple fMRI datasets. Thus, the general observation that instruction tuning can improve brain alignment has already been established.
  • Oota et al. (ICLR 2025) applied different instructions to the same images, predicted brain activity from representations of generated responses, and compared the best-performing instructions across voxels. They reported different regional prediction patterns for captioning and image-understanding instructions in higher-level and early visual areas. Therefore, instruction-dependent regional differentiation was already observed in prior unimodal-stimulus work.
  • Nakagi et al. (EMNLP 2024) examined annotations of the same naturalistic videos at multiple semantic levels and identified distinct regional encoding patterns. Although that study did not manipulate prompts in the same manner, it provides a precedent for relating different semantic descriptions of the same stimulus to different brain regions.

The contribution therefore needs to be established through what the extension to naturalistic video and audio reveals beyond these precedents, rather than through these basic observations themselves.

W2: I acknowledge the empirical extension of applying diverse instructions to multiple video and audio models and evaluating their correspondence with naturalistic-viewing fMRI. However, a combination of models, instructions, and stimuli not having been evaluated previously does not, by itself, identify an important unresolved scientific problem. The paper broadly asks which instructions predict which regions, but it is unclear which competing explanations these comparisons distinguish or how the results update our understanding of brains or models. Much of the interpretation consists of explaining the observed maps in terms of already established functional differentiation.

W3: The low RSA correlations with MiniLM/MPNet show weak correspondence with those encoders’ semantic geometry; they do not establish independence from instruction wording. The MLLMs may still depend on their own representations of the instruction text. Comparing instruction-only and video-conditioned representations within each evaluated checkpoint would therefore be an informative control, while accounting for differences in generated response content. Furthermore, Table 4 assigns one prompt to each task, confounding task identity with prompt identity. The within-video task-label permutation test detects structure associated with the chosen instruction conditions beyond video identity, but does not test generalization across different wordings of the same task.

Questions for authors

Q1: Please identify the specific scientific conclusion established for the first time in this paper relative to the previous studies.

Q2: Beyond evaluating a broader set of instructions and models, which hypothesis can be supported or rejected for the first time on the basis of this paper’s results?

Q3: Does the weak correspondence between instruction-only and video-conditioned representations in IT models persist when the instruction-similarity matrix is computed using each evaluated MLLM checkpoint itself, rather than MiniLM/MPNet? Relatedly, do the representational structure and brain-prediction patterns remain stable across multiple paraphrases of the same task?

7 comments

W1/Q1. Please identify the specific scientific conclusion established for the first time in this paper relative to the previous studies.

We thank the reviewer for this comment. The two observations the reviewer lists are established, and we already cited the previous studies. Our contribution is the following three results, none of which is reported in the cited studies.

  • (1) Task-conditioned representations emerge through instruction-tuning. Across three matched pretrained–instruction-tuned model pairs, we find significant task-conditioned representational structure beyond stimulus identity in the instruction-tuned models (FDR q ≤ 0.004), but not in their pretrained counterparts. This shows that instruction-tuning shapes how the same audiovisual stimulus is represented under different task instructions, beyond simply improving overall brain alignment.
  • (2) Meaningful task instructions matter beyond generic prompting. For Qwen-2.5-VL, task-specific prompting achieves higher whole-brain alignment than the no-prompt baseline (0.746 vs. 0.639; +16.7%). Furthermore, random prompts do not reproduce the observed layer-wise cortical alignment hierarchy, suggesting that the effects are not solely explained by the presence of additional prompt tokens.
  • (3) Task-dependent representations persist beyond generated responses (See response to Reviewer CFKc W1/Q1). Our additional controls show that task-conditioned representational differences remain observable across prompt paraphrases and before response generation. Pre-generation representations retain substantial whole-brain alignment (0.662 ± 0.048 at the best-performing layer), while teacher-forced representations retain alignment (0.626 ± 0.053) but show no measurable regional differences between instructions (|Δ| ≤ 0.014). These results distinguish task-conditioned internal representations from effects associated with generated response content.

Relation to prior work.

  • Aw et al. (2024) compared vanilla and instruction-tuned language-model embeddings (only forward pass, no matched architectures) without any prompts and systematically manipulating task prompts.
  • Oota et al. (2025) investigated task-dependent regional alignment using static images (no prompt controls, no matched architectures), whereas we examine language instructions applied to dynamic audiovisual stimuli, including video- and audio-oriented tasks.
  • Nakagi et al. (2024) investigated different semantic descriptions of naturalistic videos rather than directly manipulating task instructions within an MLLM.
  • Our study additionally includes matched architectural comparisons, modality controls, random- and no-prompt baselines, and response-generation controls.

Together, these results show how linguistic instructions organize multimodal representations, how instruction semantics influence brain predictivity, and which effects persist independently of generated responses. The revised Introduction (Lines 75-77) and Related Work (Lines 942-947) will state these contributions explicitly.

W2/Q2. Beyond evaluating a broader set of instructions and models, which hypothesis can be supported or rejected for the first time on the basis of this paper’s results?

We thank the reviewer for this comment. Our central hypothesis is that instruction-tuning enables linguistic task instructions to organize representations of identical audiovisual stimuli in ways that predict distinct patterns of brain activity, beyond generic prompting or generated-response effects. Our results challenge three competing explanations that were not directly tested in the cited studies.

(1) Generic prompting alone--insufficient. If brain alignment gains reflected generic prompting, random prompts would produce comparable effects. Task-specific instructions improve brain alignment over the no-prompt baseline (0.746 vs. 0.639 for Qwen2.5-VL), whereas random prompts do not reproduce the layer-wise cortical hierarchy.

(2) Stimulus identity or architecture alone--insufficient. Across three matched pretrained–instruction-tuned model pairs, significant task-conditioned representational structure beyond stimulus identity emerges only after instruction-tuning (FDR qq \leq0.004 0.004). This supports a role for instruction-tuning, although the contribution of post-training data cannot be isolated.

(3) Generated responses alone--insufficient. Task-conditioned structure persists across prompt paraphrases and before response generation. Pre-generation representations also retain regional differences in brain predictivity, whereas these differences are substantially attenuated with teacher-forced identical responses ($(|\Delta| \leq 0.014$); see our response to Reviewer CFKc, W4). Generated responses alone therefore cannot explain the observed effects.

Together, these results support task-conditioned internal representational organization associated with regionally differentiated brain predictivity, beyond explanations based solely on generic prompts, fixed stimulus identity, or generated text. We do not claim that passive viewers perform the instructed tasks or that regional predictivity identifies task-specific neural computations.

W3/Q3. Does the weak correspondence between instruction-only and video-conditioned representations in IT models persist when the instruction-similarity matrix is computed using each evaluated MLLM checkpoint itself, rather than MiniLM/MPNet? Relatedly, do the representational structure and brain-prediction patterns remain stable across multiple paraphrases of the same task?

We thank the reviewer for this suggestion. To examine whether MiniLM/MPNet inadequately capture instruction semantics, we repeated the RSA using instruction representations extracted from the evaluated checkpoints for two matched pairs: Qwen-2.5-VL-7B and InternVL3-8B, each instruction-tuned versus pretrained.

  • For each model, we built a text-only similarity matrix over the 13 task instructions from hidden states at the final prompt position without video input, and compared it with the video-conditioned similarity matrix from mean-pooled generated-response representations at each layer (Fig. 12 and Fig. 13, App. H).

  • The weak correspondence is not specific to MiniLM/MPNet.

    • For Qwen-2.5-VL-7B, the instruction-tuned model shows low correspondence with its own instruction geometry (mean RSA r = 0.19; range 0.12–0.26; 0.13–0.16 with MiniLM/MPNet), whereas the pretrained model shows higher correspondence (mean r = 0.39; peak about 0.55 at layer 12) and exceeds the instruction-tuned model at every layer.
    • InternVL3-8B shows the same pattern (r = 0.22 instruction-tuned vs. 0.43 pretrained). The pretrained values are lower than the MiniLM/MPNet estimates (0.69–0.78), so the contrast between pretrained and instruction-tuned models holds in direction with a smaller magnitude.
    • Both sets of values are reported in the revised manuscript (L278–284 and App. H).
  • A MLLM's text-only instruction representation encodes the requested task as well as its wording. The 78-instruction analysis (13 tasks × 6 paraphrases; see Reviewer CFKc W1) separates the two: task-conditioned organization holds across paraphrases, including formulations without shared content words, and under response-length, teacher-forced and pre-generation controls.

  • Brain-prediction stability. A paraphrased Narrative Understanding instruction gives regional brain alignment matching the original in AG, PCC and dmPFC (absolute differences ≤ 0.042, within participant SEM; see our response to Reviewer CFKc, W2).

Thank you for the detailed responses and additional analyses. I now understand the main claimed novelty to concern how instruction tuning shapes task-conditioned representations, rather than a new conclusion about task-specific neural computations.

Although I am not a specialist in task/function-vector research, my understanding is that the literature on task-conditioned representations in LLMs/VLMs is considerably more extensive than the corresponding brain-encoding literature. Relevant examples include earlier work on internal task representations [1,2], as well as more recent studies of instruction-derived task representations in VLMs [3] and instruction-based function vectors across post-training stages [4]. However, I could not find an explicit discussion of this literature in the manuscript.

While I recognize the differences in methods and settings, could you clarify which specific scientific conclusion goes beyond these prior findings, and what additional understanding the brain-encoding analyses provide? This would help me assess the novelty emphasized in your response.

[1] Hendel et al. (2023). In-Context Learning Creates Task Vectors. Findings of EMNLP.
[2] Todd et al. (2024). Function Vectors in Large Language Models. ICLR.
[3] Luo et al. (2025). Vision-Language Models Create Cross-Modal Task Representations. ICML.
[4] Davidson et al. (2025). Do different prompting methods yield a common task representation in language models? NeurIPS.

Reviewer_wdFPOpenReview ↗
Rating
2
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how task instructions affect multimodal LLM representations and their alignment with fMRI responses during movie watching. Three matched model pairs show improved brain predictivity after instruction tuning, while different instructions yield distinct regional prediction patterns.

Critical strengths
  • Varying model instructions while holding stimuli and neural recordings fixed provides a useful way to generate hypotheses about regional brain–model correspondence.

  • The matched pretrained–instruction-tuned comparisons, broader model evaluation, and participant-level analyses strengthen the empirical findings.

Critical weaknesses
  • Task identity remains confounded with prompt wording and generated-response characteristics. Each task uses one instruction, and neither low RSA correlations with external text encoders nor the task-label permutation test establishes robustness across paraphrases. Controls for response length and format are particularly relevant because representations are averaged over generated tokens.

  • The regional maps identify winning instructions without establishing their prediction margins over alternatives. The bootstrap intervals and participant-level agreement are useful, but do not directly establish reliable differences in predictivity. Independent selection and evaluation of instructions and layers would strengthen these conclusions.

Questions for authors
  1. Do the findings hold across paraphrases of each task and after controlling response length and format?

  2. Are preferred instructions and layers selected independently of evaluation data, and are their prediction advantages over alternatives statistically reliable?

6 comments
Area_Chair_dMLLOpenReview ↗

Wrong review posted

Dear reviewer

The authors note that this review is for the wrong paper. Please correct ASAP.

Thanks, Area Chair

AuthorsOpenReview ↗

Reminder: wrong review posted

Dear reviewer wdFP

We noticed that your review appears to have been submitted for a different paper. Could you please update your review asap?

Thanks, Authors

W1/Q1. Paraphrases and response controls.

We thank reviewer for this question. As per reviewer suggestion, we ran the different prompt controls. Using Qwen-2.5-VL-7B-Instruct, we evaluated all 13 tasks with six instructions each (78 instructions; the original, three paraphrases, and two paraphrases sharing no content words with the original).

  • Multiple Paraphrases. instructions for the same task are far more similar than instructions for different tasks (within = 0.445, between = −0.039; gap = 0.484; significant at every layer, 10,000 permutations). Paraphrases with no shared content words are assigned to the correct task in 66% of cases, against 7.7% chance.
  • Response length and format. with responses constrained to a common length and format, the separation is unchanged (gap = 0.450).
  • Identical response text. with the same response teacher-forced under all instructions, the separation holds (gap = 0.577).
  • Pre-generation. it is already present at the final prompt position, before any response token exists (gap = 0.392).

Overall, task-conditioned organization is therefore robust to paraphrasing and does not depend on generated-response characteristics. These analyses are in the revised manuscript at L298–315, with Fig. 14 and Table 13 in Appendix I; Please refer our full response to Reviewer CFKc (W1/Q1).

W2/Q2. Are preferred instructions and layers selected independently of evaluation data, and are their prediction advantages over alternatives statistically reliable?

We thank reviewer for this question. We added a leave-one-participant-out analysis (N = 6) in which the best layer for each instruction and the winning and runner-up instructions for each ROI are selected from five participants and evaluated in the sixth, so neither layers nor instructions are chosen with the evaluation data. We make the following observations.

  • The winning instruction beats the next higher instruction in the held-out participant in all 12 ROIs (Δ = +0.030 to +0.141).
  • The advantage is significant at p < 0.05 in seven ROIs (dmPFC, Early Visual, PCC, PTL, AG, MFG, Face Visual); six have q = 0.062 after Benjamini–Hochberg correction across ROIs.
  • Consistency with the original brain maps: the independently selected winner matches the group-level winner in the manuscript in 11 of 12 ROIs.

The same result holds when only the instruction is selected on held-out participants (positive margin in all 13 ROIs). Both analyses are in the revised manuscript at L466–473, with Tables 17 and 18 in Appendix J.1; full details are in our response to Reviewer CFKc (W2 and Q2).

Reviewer_CFKcOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how task-specific instructions affect the alignment between MLLMs and brain activity during naturalistic movie watching. The authors extract representations from instruction-tuned video and audio models under different task instructions and use voxel-wise encoding models to predict fMRI responses. They find that instruction-tuned models generally achieve better brain alignment than pretrained models and that different instructions produce distinct regional alignment patterns. The authors argue that instruction-tuned MLLMs can serve as useful tools for studying task-related functional organization in the brain.

Critical strengths

I think the idea of using different instructions to probe brain alignment under identical naturalistic stimuli is interesting. It provides a relatively inexpensive way to explore how different model representations relate to regional brain activity without collecting new fMRI data. I appreciate the broad experimental evaluation across multiple video and audio MLLMs. In particular, the three matched comparisons between pretrained and instruction-tuned checkpoints make the improvements in brain predictivity more convincing than comparisons across unrelated model architectures alone.

Critical weaknesses
  • I am not convinced that the method successfully separates task-specific representations from prompt-related effects. The paper argues that instruction-tuned models capture task-specific information rather than the surface semantics of the instructions. However, each task is represented by a single instruction, making task identity inseparable from prompt wording and response format. The authors use RSA and a task-label permutation test to support this conclusion. I think these analyses establish that different prompts produce distinguishable representations, but they do not demonstrate that the differences reflect the underlying cognitive tasks. For example, models may generate substantially different response lengths, linguistic structures, or levels of detail depending on the instruction. I would expect experiments using multiple paraphrases of the same task, as well as controls for response length and format, before concluding that the representations are genuinely task-conditioned rather than prompt-conditioned.

  • The authors assign each voxel to whichever instruction achieves the highest prediction correlation. I find this approach potentially misleading because every voxel is assigned a preferred task, even when the differences between tasks are very small. For example, a voxel may be labeled as preferring narrative understanding even if its prediction accuracy is nearly identical under video understanding or another instruction. The resulting cortical maps could therefore suggest stronger functional specialization than the evidence supports.

    Although the paper provides bootstrap confidence intervals and participant-level consistency analyses, I would like to see direct statistical comparisons between the winning instruction and competing instructions, ideally with independent data for selecting and evaluating task preferences. I also wonder whether the reported preferences remain stable across different prompt formulations.

  • Participants passively watched movies and were never instructed to perform the tasks used to probe the models. Therefore, the observed regional differences reflect which model-generated representations best predict a fixed neural response, not how the brain changes its processing under different task demands. The authors acknowledge this distinction, which I appreciate.

    However, I still find the broader interpretation of task-related functional specialization too strong. An instruction that improves prediction in a particular brain region may simply emphasize stimulus features already represented there. For example, better alignment from a narrative instruction does not necessarily indicate that the corresponding region specifically implements narrative understanding. I think the paper needs to distinguish more carefully between regional predictivity and evidence for task-specific neural computations.

  • The authors average hidden states over generated response tokens to construct each task-specific representation. I am not sure whether this is the most appropriate representation for studying brain alignment, because the resulting embeddings depend on the generated text, including its length, content, and potentially hallucinated information.

    This is particularly concerning for tasks such as narrative understanding and linking events, which require temporal context extending beyond an individual video clip. The paper does not sufficiently establish whether the temporal context available to the models supports the intended task distinctions.

    I would like to see a more controlled comparison using representations extracted before response generation, together with clearer analysis of how stimulus windows are aligned with fMRI time points.

  • In Appendix P, the authors compute shared and unique contributions using inclusion–exclusion operations on normalized brain-alignment scores, which are based on prediction correlations. However, correlations are not equivalent to explained variance, and applying variance-partitioning arithmetic directly to these scores does not necessarily provide a valid decomposition of neural variance. I think this is a substantive issue because the resulting quantities are interpreted as unique and shared variance attributable to different tasks. The authors should justify this calculation or repeat the analysis using an appropriate held-out explained-variance metric.

Questions for authors
  • How do you distinguish genuine task-conditioned representations from prompt-specific effects? Have you tested multiple semantically equivalent instructions for each task?
  • How robust are the regional task preferences to small differences in prediction accuracy? Are the winning instructions significantly better than the alternatives, and are the best layers and task assignments selected independently of the test data?
  • To what extent do generated response characteristics influence brain alignment? Have you controlled for response length, linguistic content, or output format, or compared against representations extracted before response generation?
  • How are video features temporally aligned with fMRI responses, particularly for narrative and event-linking tasks? What temporal context is available to the MLLMs?
  • Could the authors clarify the variance-partitioning formulation in Appendix P? Specifically, why is inclusion–exclusion applied to normalized prediction correlations rather than explained variance, and how does this affect the interpretation of unique and shared task contributions?
11 comments

W1/Q1. How do you distinguish genuine task-conditioned representations from prompt-specific effects? Have you tested multiple semantically equivalent instructions for each task?

We thank the reviewer for raising this concern. Following the reviewer's suggestion, we ran additional paraphrase and response-control experiments using the same Qwen2.5-VL-Instruct checkpoint across all 13 tasks. Each task was evaluated with six semantically equivalent instructions: the original prompt, three paraphrases, and two additional paraphrases designed to avoid shared content words. In total, we evaluated 78 instructions across five video clips sampled from the movie, under three conditions: natural response generation, length- and format-controlled generation, and teacher-forced identical responses. The Narrative Understanding instructions are shown as an example; all 78 will be listed in the Appendix I Table 13. We assessed the within-task vs. between-task similarity gap using 10,000 task-label permutations across the 78 instructions, correcting across layers using a maximum-statistic procedure.

  • P1: Summarize the main storyline of the movie. What is the central conflict, and how is it resolved? (Original prompt)
  • P2: Describe the central story, including its main conflict and resolution.
  • P3: Explain the plot of the video, focusing on the principal conflict and how it is resolved.

Multiple paraphrases. We computed mean-centered cosine similarity between task-conditioned representations, averaged across layers 1–29 (Appendix I, Fig. 14).

  • Under natural generation, within-task similarity was substantially higher than between-task similarity (0.445 vs. −0.039; gap = 0.484). The separation was significant at all 29 layers after multiple-comparison correction and positive across all five clips (t(4) = 31.8, p < 0.001).
  • Importantly, paraphrases without shared content words were assigned to their intended tasks in 66% of cases, substantially above chance (7.7%, p < 0.001).
  • The weakest separation occurred among semantically related tasks involving scene descriptions, including Video Understanding, Action Recognition, Video Captioning, Temporal Ordering, and Object and Scene Recognition.

Response-length and format controls.

  • Constraining responses to a common length and format reduced response-length variability (mean token count: 86.4 -> 50.2; SD: 24.5 -> 8.9), while task separation persisted (within = 0.414, between = −0.035; gap = 0.450), remaining significant at all 29 layers.
  • Although differences in response length were weakly associated with lower same-task similarity (r = −0.15 under natural generation; r = −0.23 under controlled generation), adjusting for token-count differences had minimal effect on task separation (0.484 -> 0.468 for natural responses; 0.450 -> 0.446 for controlled responses).

Teacher-forced identical responses. As a stronger control, we teacher-forced identical response text across instructions, ensuring that response tokens, their positions, and response length were held constant.

  • Task separation persisted (within = 0.537, between = −0.040; gap = 0.577), with significant separation across all 78 task pairs.
  • Thus, differences in generated response content, length, or format are not necessary for the observed instruction-conditioned representational organization.

Pre-generation representations. We additionally examined hidden states at the final prompt position, before any response tokens were generated.

  • Task separation was already present under natural prompting (within = 0.359, between = −0.033; gap = 0.392) and length/format-controlled prompting (within = 0.410, between = −0.037; gap = 0.447), with significant separation at all 29 layers in both conditions.
  • Furthermore, paraphrases without shared content words were assigned to their intended tasks with accuracies of 61% and 80%, respectively, compared with 7.7% chance. These findings demonstrate that instruction-conditioned representational differentiation emerges before response generation and does not depend solely on shared lexical content or generated text.

Taken together, these experiments demonstrate that semantically equivalent instructions produce relatively consistent representations, whereas different task instructions yield distinguishable representations even when the video input and response-token sequence are held fixed. This organization persists across all 13 tasks, multiple paraphrases, response-length and format controls, teacher-forced identical responses, and pre-generation representations. These findings provide stronger evidence that the observed differentiation reflects task-related instruction semantics rather than being explained solely by prompt wording or generated response characteristics.

In the revised manuscript, these analyses are reported in the RQ1 section (L298–315), with Fig. 14 and the full list of 78 instructions (Table 13) in Appendix I.

W2.1/Q2 I would like to see direct statistical comparisons between the winning instruction and competing instructions, ideally with independent data for selecting and evaluating task preferences.

We thank the reviewer for this suggestion. To address reviewer's question, we ran the comparison for a direct statistical test of each region's winning instruction against its closest competitor, with selection and evaluation on independent data.

Held-out winner vs. runner-up

  • Independent selection. We used leave-one-participant-out evaluation across six participants, 29 layers and 13 instructions. For each held-out participant, the best layer for each instruction and the winning and runner-up instructions for each ROI were selected from the other five participants only, and then evaluated in the held-out participant. Neither the layer nor the instruction is chosen with the evaluation participant's data.

Appendix J.1 Table 17. Winning vs. runner-up instruction, evaluated on held-out participants (N = 6; one-sample t-test, two-sided; p-values uncorrected).

ROIWinnerRunner-upΔ (held-out)95% CIp-value
AGNUSU+0.052[−0.020, +0.123]0.121
ATLVQVU+0.020[−0.065, +0.105]0.572
PTLSUVQ+0.031[−0.030, +0.091]0.245
IFGVRSU+0.042[−0.041, +0.124]0.248
MFGESAVQ+0.077[+0.016, +0.139]0.024
IFGOrbSUVC+0.034[−0.021, +0.088]0.170
PCCNUSU+0.081[+0.019, +0.142]0.020
dmPFCNUVQ+0.064[+0.001, +0.127]0.047
EVCVUSU+0.031[+0.005, +0.057]0.028
LOCSUVR+0.038[−0.030, +0.105]0.207
OFASUVR+0.048[−0.006, +0.102]0.071
PPASUVR+0.040[−0.020, +0.099]0.145
MTSUNU+0.077[+0.020, +0.135]0.018
  • The held-out winner-runner-up differences were positive across all 13 ROIs (Δ = +0.020 to +0.081). Five regions showed nominally significant advantages: PCC (Narrative Understanding vs. Spatial Understanding: Δ = +0.081, p = 0.020), MFG (Emotion Analysis vs. Video Question Answering: Δ = +0.077, p = 0.024), Middle Temporal (Spatial Understanding vs. Narrative Understanding: Δ = +0.077, p = 0.018), dmPFC (Narrative Understanding vs. Video Question Answering: Δ = +0.064, p = 0.047), and Early Visual (Video Understanding vs. Spatial Understanding: Δ = +0.031, p = 0.028).

  • In the revised pdf, we updated Table 17 and included in [RQ3] Held-out validation of regional instruction preferences (Lines 453-460).

W2.3/Q2 whether the reported preferences remain stable across different prompt formulations.

  • To assess whether brain alignment depends on the exact wording of the instruction, we repeated the encoding analysis for Narrative Understanding with a paraphrased instruction (P2: "Describe the central story, including its main conflict and resolution"). Regional brain alignment is unchanged within participant variability (mean ± SEM, six participants): AG 0.722 ± 0.030 vs. 0.680 ± 0.035; PCC 0.738 ± 0.029 vs. 0.779 ± 0.070; dmPFC 0.676 ± 0.068 vs. 0.695 (all differences ≤ 0.042).
  • The regional predictivity of Narrative Understanding is not tied to the original wording, in line with the 78-instruction analysis in W1/Q1, which shows that the representations of all 13 tasks are stable across paraphrases.

W3. I think the paper needs to distinguish more carefully between regional predictivity and evidence for task-specific neural computations.

We thank the reviewer for this comment. We state at L420–422 that these analyses "characterize which model-side instructions best predict each region's responses; they do not imply that participants performed the corresponding tasks", and again in the Limitations (App. V, L2832–2838).

  • Our two questions are

    • (1) whether different task instructions elicit distinct MLLM representations of the same audiovisual stimulus and
    • (2) whether these representations differ in their alignment across cortical regions. Neither requires, or claims, task-specific neural computation.
  • Our new controls test the reviewer's alternative directly.

    • Instruction-dependent regional differences are present before response generation, and when the response text is held fixed across instructions the regional differences are no longer measurable (|Δ|≤ 0.014; see W4).
    • Instructions act by changing which stimulus information the model's representation captures, which is the account the reviewer describes.
  • In the revision we will use "instruction-dependent regional predictivity" throughout and replace the wording that reads as specialization: "probes of functional specialization" (L109), "selectively align with distinct functional regions" (L104) and "preferentially engage distinct higher-order ROIs in ways consistent with known functional neuroanatomy" (L449–450).

W4.1/Q3. The resulting embeddings depend on the generated text, including its length, content, and potentially hallucinated information.

To investigate response length, content, and potentially hallucinated information, we conducted additional representational and brain-encoding experiments using the Qwen2.5-VL-Instruct model.

  • As discussed in W1/Q1, task-conditioned organization persists under length and format control, with identical teacher-forced response text, and before any response is generated.

  • To directly assess whether these controls also preserve brain-predictive information, we conducted additional voxel-wise encoding experiments using Qwen2.5-VL-7B-Instruct for one representative task “Narrative Understanding”.

  • We evaluated three representation-extraction conditions:

    • (i) length- and format-controlled generation,
    • (ii) teacher-forced identical responses, and
    • (iii) pre-generation representations extracted at the final prompt position before response generation.
  • At the best-performing layer, whole-brain normalized alignment is 0.635±0.024 for the original generated-response representations, 0.662±0.048 before generation, 0.626±0.053 with teacher forcing and 0.594±0.056 with length and format control.

  • Regional alignment is likewise retained: AG, PCC and dmPFC reach 0.680, 0.779 and 0.695 with the original representations, 0.741, 0.705 and 0.802 before generation, and 0.695, 0.705 and 0.739 with teacher forcing.

  • Pre-generation representations contain no generated text, so response length, content and hallucinated information cannot contribute to their alignment. Brain predictivity does not depend on the generated response.

Task-dependent regional differences. We extended the encoding analysis to Spatial Understanding and compared it with Narrative Understanding in 13 ROIs (Table 19, App. J.1; N = 6; averaged across layers 1–29).

  • Under pre-generation extraction, Narrative Understanding showed higher alignment in 11 of 13 ROIs, most strongly in MFG (Δ = +0.116), dmPFC (+0.048) and PCC (+0.022); Spatial Understanding was higher in PTL (−0.037) and PPA (−0.048).
  • The directions in AG, dmPFC, PCC, PTL and Scene Visual (PPA) are consistent with the instruction preferences in our main analysis. The Spatial preferences in IFGOrb and in the LOC,OFA, and MT visual regions were not reproduced.
  • Under teacher-forced extraction, both instructions retained strong alignment but did not differ in any ROI (|Δ|≤ 0.014).

Table 19 (App. J.1 of the revised manuscript). Narrative Understanding versus Spatial Understanding: ROI-level difference in normalized brain alignment under pre-generation and teacher-forced extraction. Δ = Narrative minus Spatial; positive values indicate higher alignment for Narrative.

ROIPre-generation ΔTeacher-forced Δ
AG+0.029+0.008
ATL+0.032−0.002
PTL−0.037−0.004
IFG+0.014+0.002
MFG+0.116−0.010
IFGOrb+0.018−0.004
PCC+0.022−0.010
dmPFC+0.048−0.001
EVC+0.024+0.004
LOC+0.018+0.013
OFA+0.018+0.012
PPA−0.048+0.008
MT+0.042+0.014

Layer-wise profile. With teacher forcing, the difference is near zero in every region at every layer. Before generation it is structured across depth: Narrative is favoured in dmPFC and MFG across the middle layers and in PCC in the late layers (about 21–27); Spatial is favoured in Scene Visual from about layer 17 and in PTL across layers 20–28.

  • Two conclusions follow.
    • First, the regional preferences are not produced by response length or hallucinated content: both are absent before generation, where the direction of the preference is already reproduced in six of eight ROIs.
    • Second, in response-token representations the instruction acts through the content it elicits: with identical response text, the two instructions give the same regional alignment.

Both extraction conditions are reported in the revised manuscript in Sec. 4 (L404–417 and L474–482), with Table 19 and Fig. 20 in App. J.1.

W4.2/Q4. How are video features temporally aligned with fMRI responses, particularly for narrative and event-linking tasks?

  • Temporal context. Each model input is the video segment for one TR (1.49 s; L119–120). The five-delay FIR model (about 7.5 s; L215–216) accounts for the hemodynamic delay; it does not give the model additional context. Our Narrative Understanding and Linking Events conditions are therefore narrative-oriented and event-linking-oriented instructions applied to local audiovisual content, and our results show how these instructions change the representation of that content and its regional predictivity. We make no claim that the model reconstructs the storyline or links distant events. The revised Limitations (App. V, L2835–2837) state that individual audiovisual segments are approximately 1.5 seconds long, which limits conclusions about long-range narrative comprehension and event reasoning.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. We summarize the main additions below.

Overall, our main findings hold under the additional controls and validations (78 prompt formulations across 13 tasks, response-length and format controls, teacher-forced identical responses, pre-generation representations, MLLM's-specific instruction representations, held-out selection of layers and instructions, and a selection-free ROI-by-instruction interaction test): instruction-tuned MLLMs exhibit robust task-conditioned representations of identical audiovisual stimuli, achieve higher brain alignment than matched pretrained counterparts, and show statistically significant global regional differentiation in instruction predictivity. These additional analyses further strengthen our central conclusions and clarify which regional claims the evidence supports.

1. Robustness to prompt paraphrases and response generation (CFKc, 6LNn, wdFP; Sec. 4 RQ1, App. I, Fig. 14, Table 13).

  • We evaluated all 13 tasks with six prompt formulations each (78 prompts), including paraphrases sharing no content words with the original. Representations of the same task remain substantially more similar than those of different tasks (within = 0.445, between = -0.039; gap = 0.484; significant at every layer). This separation persists under response-length and format controls (gap = 0.450), teacher-forced identical responses (0.577), and pre-generation extraction (0.392). Moreover, lexically distinct paraphrases are assigned to their intended tasks with 66% accuracy, compared with 7.7% chance, demonstrating robust task-conditioned organization beyond prompt wording and generated-response effects.

2. New baselines and controls.

  • MLLM-specific instruction representations (6LNn; Sec. 4 RQ1, App. H, Figs. 12–13). Using each checkpoint’s own instruction representations instead of MiniLM/MPNet, instruction-tuned models show weaker coupling to instruction-text semantics than their matched pretrained counterparts in both pairs tested (Qwen-2.5-VL-7B: rr=0.19 vs. 0.39; InternVL3-8B: rr=0.22 vs. 0.43), confirming that this distinction is not specific to external text encoders.

  • Held-out validation of regional preferences (CFKc, wdFP; Sec. 4 RQ3, App. J.1, Tables 17–18). Under leave-one-participant-out evaluation (N=6), the winning instruction outperformed the runner-up on held-out participants across all ROIs. With both layers and task-independently selected on training participants, all 12 ROIs showed positive held-out margins (+0.030 to +0.141), with seven reaching nominal significance (p<0.05p<0.05), although none survived BH-FDR correction (minimum $q\approx$0.062).

  • ROI-by-instruction interaction (CFKc, wdFP; Sec. 4 RQ3, App. J.1). A selection-free analysis across 13 instructions and 12 ROIs shows significant global regional differentiation: interaction patterns are consistent across participants (mean pairwise rr=0.17, permutation p=0.0001p=0.0001), and the Narrative-minus-Spatial advantage is larger in AG/PCC/dmPFC than in PTL/PPA in all six participants (Δ=+0.004\Delta=+0.004, p=0.035p=0.035).

  • Brain encoding under response-generation controls (CFKc, wdFP; Sec. 4 RQ3, App. J.1, Tables 19-20, Fig. 20). Across all 13 instructions, pre-generation representations retain 6/12 original ROI winners (median margin 0.044 vs. 0.062), while teacher-forced identical responses retain 2/12 (median margin 0.005). For Narrative Understanding, whole-brain alignment remains comparable under both controls (0.662±0.048 and 0.626±0.053 vs. 0.635±0.024 originally), showing that brain-predictive information persists despite changes in representation extraction.

  • Temporal context (CFKc; App. J.1, Table 21). Extending the video input from 1.5 to 24s improves brain alignment for Narrative Understanding in AG (+0.084), PCC (+0.098), and dmPFC (+0.075), supporting the benefit of longer temporal context.

3. Reframing and clarifications (CFKc, 6LNn, wdFP).

  • The Abstract, Results, and Conclusion now distinguish robust task-conditioned representations, significant global regional differentiation, and descriptive individual ROI preferences.
  • Clarified that regional patterns reflect task-dependent brain predictivity, not functional specialization, and that participants watched movies passively (Sec. 4 RQ3; App. V).
  • Clarified the single-TR (~1.5 s) input window and its limitations for narrative-oriented instructions (App. V).
  • Added comparisons with prior work on task and function vectors in LLMs and VLMs (Related Work, Lines 961–971).
  • Clarified the competing explanations tested: instruction semantics, generic prompting, and generated-response effects (Introduction).

We believe these additional analyses strengthen our conclusions on task-conditioned representations and brain alignment, provide significant evidence for global regional differentiation, and clarify the limited stability of individual ROI preferences.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.