acceptodds
Withdrawn from ICLR 2027

How does longer temporal context enhance multimodal narrative video processing in the brain?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding how humans and artificial intelligence systems process complex narrative videos is a fundamental challenge at the intersection of neuroscience and machine learning. We investigate how the temporal context length of video clips (3–24 s long) and the narrative-task prompting shape brain-model alignment during naturalistic movie watching. Using fMRI recordings from participants viewing full-length movies, we examine how brain regions sensitive to narrative context dynamically represent information over varying timescales and how these neural patterns align with model-derived features. We find that increasing clip duration substantially improves brain alignment for multimodal large language models (MLLMs), whereas unimodal audio/video models show little to no gain. Further, shorter temporal windows align with perceptual and early language regions, while longer windows preferentially align higher-order integrative regions, mirrored by a layer-to-cortex hierarchy in MLLMs. Finally, experiments with four narrative-task prompts show that they elicit task-specific, region-dependent brain alignment patterns and context-dependent shifts in clip-level tuning in higher-order regions. Our work positions long-form narrative movies as a principled testbed for studying long-timescale temporal integration in long-context MLLMs and its relationship to cortical responses during narrative comprehension.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Multimodal brain encoding, language models, multimodal models, video, Audio, fMRI, Narrative understanding, Temporal context

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_A4HXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents an analysis concerning the predictability of fMRI data with computational models on narrative long form video content. Therefore it extends the body of evidence of previous work that largely focused on temporally shorter content. This analysis reveals that increasing content duration increases brain alignment with models only for the tested MLLMs, while unimodal models show little improvement. Furthermore some ROIs are more aligned with shorter time windows while others are aligned with longer ones. Additionally they compare how different narrative prompts influence brain alignment across different ROIs and find that there are differences in brain predictability across prompts but also that individual voxels within ROIs are heterogenous. And finally, they show that the clips that elicit the highest predicted activity for a given voxel are more consistent for visual voxels, while those for language-related voxels vary more across context lengths and prompts. These findings are based on the Movie10 video dataset where participants watched different movies, and they essentially train voxel prediction via regression from model responses on these movies and then evaluate on unseen movies. They also estimate cross-participant prediction accuracy by predicting each participant’s brain responses from those of the other four and use this estimate to normalise model performance.

Critical strengths

The paper presents a natural extension to previous work and adds useful observations for investigating model brain alignment when varying timescales. The most interesting part is that these models seem to show context-length preferences that correspond to the cortical hierarchy of timescales. For me, the main value of this finding is what it tells us about the models’ representations. Also the observation that those model representations are somewhat well adapted to representing different narrative contexts in a brain like manner shows some value of narrative prompting in the pursuit of studying model–brain alignment.

Critical weaknesses

What is less clear, is the broader significance of these findings in regards to the brain. The hierarchical organisation was already established in previous works such as Hasson et al. (2008) and Lerner et al. (2011). The participants were not given different viewing tasks, so the prompt comparisons do not establish how those tasks would change brain responses. But following from this I find the title not necessarily a good representation of the content of the paper.

Questions for authors

How sensitive are the A4 results to the choice of top k? And how do the predicted highest clips compare to the the actual highest brain activity clips ?

Minor:

L.153 -160 what is the reason for choosing this sampling strategy if the ablation with one 1fps preserves the main trend? Why do you not just use 1 fps instead of this heterogeneous sampling strategy 9, 9, 12, 16, 24, and 30?

The dataset description and Table 2 caption refer to five participants, but the participant-preference row reports agreement as 3/4, 2/4, and 4/4. Does this analysis include only four participants?

Reviewer_hx2UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how temporal context and narrative-task prompts affect brain–model alignment during movie viewing, using pretrained MLLM features to predict fMRI responses. It reports improved predictivity with longer windows and context-, task-, and layer-dependent alignment patterns across cortical regions.

Critical strengths
  1. The systematic comparison of context lengths, narrative prompts, and model layers provides a useful characterization of MLLM–brain alignment.

  2. Within-model modality and prompting ablations help distinguish these factors from differences between model families.

  3. Cross-movie evaluation, participant-level analyses, and an additional held-out film strengthen the evidence for the overall long-context benefit.

Critical weaknesses
  1. The source of long-context gains remains partly unresolved. Longer windows provide more audiovisual information and may increase temporal smoothness. The constant-fps ablation still increases frame count with duration, leaving uncertainty about whether the gains specifically reflect narrative integration. (Sec. 3, A1(d), Appendix I)

  2. The reliability of specific regional preferences could be better established. Winner-based maps identify preferred conditions, but some best–runner-up margins are small. Moreover, the paraphrase analysis measures embedding similarity rather than the stability of brain-prediction results. Further validation would strengthen confidence in the reported ROI-by-condition differences. (A2–A3, Tables 2–3)

  3. Positioning relative to prior work could be clearer. Closely related work already examines instruction-conditioned MLLM encoding and cortical layer/task correspondences. The additional insights from jointly studying context and prompting therefore deserve clearer emphasis. Related work is important for assessing this contribution; although Table 1 provides an overview, the detailed discussion is in Appendix A. I would encourage the authors to reorganize the main text to include a focused related-work discussion. (Oota et al., 2025)

  4. Generalization of the detailed alignment patterns remains less established. The Hidden Figures evaluation supports the overall context-length benefit, while regional and task-specific findings are primarily evaluated on Life. As an optional extension, reproducing these patterns on Hidden Figures—and, if feasible, an independent naturalistic dataset such as CineBrain (https://huggingface.co/datasets/Fudan-fMRI/CineBrain) — would strengthen their generalizability across stimulus content and acquisition settings. (A1(a), A3)

Questions for authors
  1. Could the authors compare joint long-window encoding with aggregated short-window features from the same content, or include a content-preserving temporal-order control, to assess whether the gains depend on narrative structure?

  2. Are the main regional preferences supported by direct condition contrasts or ROI-by-condition interaction tests? Please clarify how preferred conditions were selected and whether these preferences persist on independent data and under paraphrased prompts.

  3. Does context length systematically change the relative effectiveness of narrative tasks across ROIs? Does “complementary probes” refer to two analytical perspectives, or to complementary brain-relevant information?

Reviewer_aVTtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how video context duration and narrative-task prompts affect the correspondence between model representations and fMRI responses during naturalistic movie viewing. Using five participants from Movie10, the authors extract layer-wise representations from Qwen-2.5-Omni and DATE over 3–24 s windows and fit participant-specific voxelwise ridge-encoding models. Qwen receives audiovisual inputs and task text; DATE receives video and task text. Comparisons include three unimodal baselines, within-model modality and prompting ablations, and a frame-sampling sensitivity analysis. The paper reports substantial longer-context gains for the two MLLMs, ROI-dependent preferred windows and prompts, a layer-to-cortex hierarchy, and changes in model-predicted top-ranked clips. Validation uses a disjoint test movie, a second test movie for Qwen, participant-level summaries, and repeat-based reliability estimates.

Critical strengths
  1. The broad context-length finding has several complementary sources of support. Table 8 shows positive 3-to-24 s gains in all five participants for both MLLMs, with mean normalized alignment increasing from approximately 0.674 to 0.953 for Qwen and 0.674 to 0.899 for DATE. Qwen shows a similar trend on Hidden Figures (p. 5). The within-Qwen modality and prompting controls and within-DATE prompting comparison reduce confounding from cross-family differences. This makes the empirical longer-context benefit a meaningful contribution.
  2. The paper provides useful evidence about the scope and variability of its results. Tables 9–10 and 12–13 report ROI preferences and model differences, while Tables 4–5 complement cross-subject normalization with repeat-based reliability. PCC and dmPFC prefer 24 s in all five participants for both MLLMs, whereas other ROIs are more heterogeneous. The authors explicitly acknowledge dominant tendencies, small margins, and participant variability. This transparency is consequential for interpreting a small-participant naturalistic fMRI study.
Critical weaknesses
  1. The specific temporal-hierarchy and task-dissociation interpretations need stronger support than the descriptive winner maps provide. Figures 3 and 5 select the maximum across six windows or four prompts. The reported margins, SEMs, and participant preferences are helpful, but condition-wise SDs do not establish uncertainty in paired best-versus-runner-up differences or the stability of the selected winner.
  2. The mechanistic interpretation of the overall gain also needs qualification: the 1 fps control still increases frame count with duration, so it tests sampling-scheme sensitivity without separating input amount from integration across the full 3–24 s sweep. The matched nine-frame 3/6 s comparison provides useful local evidence, but does not settle that full-range interpretation. Focused contrast analyses or consistently narrower claims would address these concerns.
  3. Given that there are only 5 subjects, does this imply the potential for a significant data bias?
Questions for authors
  1. Please reconcile the Qwen–DATE p-values with Table 8 and provide the participant-level inputs, aggregation, test, raw and adjusted p-values, and FDR family for each principal comparison.
  2. For the main window and prompt contrasts underlying the hierarchy/dissociation claims, can the authors provide paired uncertainty estimates and account for how the contrasts or winners were selected? Please distinguish robust tendencies from near-ties and account for temporal dependence if resampling movie time points.
  3. What evidence separates temporal integration from increased input amount over the full context sweep? The 1 fps experiment changes frame count with duration, while the matched nine-frame comparison covers only 3 versus 6 s. A fixed-frame-count or temporal-order control would strengthen the mechanistic claim.
Reviewer_7i18OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the effects of temporal context length and narrative-task prompting on the ability of multimodal model features in predicting fMRI responses for naturalistic movie watching. Authors extract representations over overlapping 3-24 s video windows, and condition the models on four tasks namely – character motivation, event boundary detection, multi-scene summary, and narrative summary, and fit participant-specific encoding models. Their core held-out evaluation uses Life with TimeSFormer, VideoMAE, and an audio-only model as a unimodal baseline.

The central claim is that longer windows appreciably improve the average brain alignment for MLLMs, while unimodal baseline shows comparatively small gains. Authors further report that the preferred window length differs by cortical region (short contexts favor sensory and earlier language regions while longer contexts favor higher-order semantic and narrative regions), and argue that this relates this to layer-wise differences in MLLM representations. Additionally authors also use task-conditioned encoding and clip retrieval to argue that narrative prompts reveal distinct regional preferences. Evaluation on a second held-out movie supports the aggregate context-length trend.

Critical strengths
  • The research question posed is very interesting and timely as the focus goes beyond ‘whether or not long-context multimodal representations improve brain prediction’ to ‘which cortical regions and model layers benefit from them’.
  • The coverage of two MLLMs vs audio/video baselines, within-model modality, prompting ablations and a frame-sampling control is appreciably wide.
  • ROI-wise window analysis, clip retrieval, task-preference maps, and layer-wise comparisons give complementary views suggesting verifiable hypotheses.
  • Hidden Figures provides an initial cross-movie check of the main trend, and subject-level trajectories and best vs runner-up values are also reported.
Critical weaknesses
  1. Long-context integration is not cleanly separated from temporal smoothing as 3-24 s windows overlap heavily (due to 1.49 s strides) which produces slowly varying representations that may predict the slow BOLD signal without requiring narrative integration at all and the constant-frame-density ablation does not address this. Controls should have included averaging short-window features over the same 24 s span and shuffling frame order within long windows.

  2. Absence of a text-only baseline. Hidden states are averaged over generated tokens, so improvements could easily reflect longer verbal descriptions instead of better multimodal temporal representations. An encoding model using only the generated descriptions, context-matched transcript baselines with analyses controlling for output length are imperative for isolating the source of the gains.

  3. Model comparisons and the code provided with the supplementary zip raise reproducibility concerns. Extraction scripts load Qwen/Qwen2.5-Omni-3B and Qwen/Qwen2.5-VL-7B-Instruct with DATE-related modifications, while the paper describes two 36-layer MLLMs. The scripts use different system prompts and incompatible feature aggregation – Omni keeps layer-wise representations, while the DATE script averages across all layers and tokens into a single vector, this does not match the reported DATE layer-wise analyses. The DATE timestamp-insertion branch requires a timestamps argument which is not passed in the extraction script (although a positional-encoding override is installed), hence the active timestamp mechanism requires clarification. The alignment script hardcodes ws9_str1, assumes a layer axis that the DATE extractor does not export, accepting only subjects 1, 2, 3, and 5 (4 in total) while the paper reports five participants. It also applies an unreported PCA to 1024 dimensions, searches ridge penalties over 10-1000 (while paper reports 10^-1 to 10^3), and truncates the Life test set to 2,013 TRs (while 2,028 is reported in the paper). How the fixed-input unimodal baselines process these 18-24 s windows is also not described. These observations concern the supplied code and would need clarifications if authors used some additional scripts not part of what's in the zip.

  4. The reported values for reliability show concerning regularities that require necessary explanation. In Table 4 the step-to-step differences in vertices(%) retained across S1->S2->S3->S5->S6 are the same five values (+4.25, -5.60, +6.90, +3.25, -8.80) just cyclically shifted, in 13 of 14 ROIs. Several rows equal another row plus a constant for all five subjects (as IFGOrb = AG - 0.004 in r and - 3.14 in %; AC = EVC - 0.001 in r). Table 5 shows exactly the same pattern (as AC EV = EVC EV - 0.044 for every subject and the IFGOrb EV row equals the AG row with S1/S2 swapped). Table 2 reports participant agreement out of 4 (3/4, 2/4 etc) although five participants are analyzed and then Table 3 reports a PCC "tie (3/5 each)", which is not possible with five participants. Provided code is completely amiss for the ceiling or EV estimates. As these tables supply the ceiling that's used to normalize every reported result, the abovementioned issues affect confidence in all normalized findings.

  5. Statistical support for ROI- and task-specific claims are not sufficient, and tests are inconsistent. Best vs runner-up gaps are often small (0.005-0.159) and below the across-subject SDs (for ex. PCC .963 +/- .301, in Table 3), and no paired subject-level tests or uncertainty are given on these differences. Appendix F describes Wilcoxon signed-rank tests (the minimum exact two-sided p-value with five paired observations is 0.0625), whereas Section 5 reports paired t-tests. Section 5 states that Qwen-DATE differences are nonsignificant at all windows (p >= 0.435), but paired t-tests computed from papers own Table 8 give p ~ 0.011 at 18 s and p ~ 0.0025 at 24 s (uncorrected). All these test definitions, correction families and reported p-values need to be coherently recaliberated.

  6. Cross-movie validation is limited and effect sizes remain unclear The ROI and task-preference analyses use only Life (a narrated documentary). Hidden Figures verifies only the aggregate trend and some normalized alignment values exceed 1 (Table 8) which is not necessarily an error for a cross-subject ceiling, but raw Pearson correlations, ceiling estimates, and uncertainty are needed for a clear interpretation of these scores.

Minor issues

  • Disagreement in prose and tables: Section [A3] says Narrative Summary dominates PTL, however Table 3 selects Character Motivation. Claims regarding unanimous MFG/dmPFC preferences disagree with Table 12 (4/5 each) and the statement "eight of 12 ROIs agree in at least three of five participants" also conflicts with Table 12 (12/12).
  • Prose in section [A2] says 9/12 ROIs agree in at least four participants (while Table 9 has 8/12) and cites EVC among the most consistent ROIs (Table 9 has 3/5). Appendix G's 9/12 in at least three participants conflicts Table 9 (11/12).
  • Table 11 contradicts the claim of monotonic improvement (0.957 at 18 s vs. 0.943 at 24 s under 1 fps), and differences between sampling schemes (0.002-0.039) are also smaller than the stated 0.03-0.06.
  • The framing that prior multimodal studies used windows "often under 2 s" or "only up to ~1.49 s" should be reconciled with the paper's own reporting of 4 s and 35 s settings listed in Table 1.
  • Extraction code samples during generation (Omni: do_sample=True, top_k=2; DATE: top_k=1, effectively greedy). And it does without a stated seed or repeated-run variability analysis. The decoding choices are also not described in the paper.
  • The claim that the four tasks form complementary and irreducible components of narrative understanding is not empirically proven.
Questions for authors
  1. If temporal-smoothing and frame-order-shuffle controls retain the long context gains?
  2. What's the performance of generated-text-only and transcript-only baselines after controlling for output length?
  3. Which checkpoints (DATE timestamp mechanism, layer-wise feature definitions, and long-window unimodal preprocessing) produced the reported results? Can authors provide runnable scripts for all six window lengths and all five subjects?
  4. Can raw correlations, paired ROI/task comparisons be reported with confidence intervals along with the exact statistical tests and multiple-comparison corrections used?
  5. Does ROI-specific window and prompt preferences replicate on Hidden Figures?
  6. Can the authors provide the code and per-subject outputs underlying Tables 4 & 5 and all of S6 results, and explain the concerning regularities described in Weakness 4?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.