- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents an analysis concerning the predictability of fMRI data with computational models on narrative long form video content. Therefore it extends the body of evidence of previous work that largely focused on temporally shorter content. This analysis reveals that increasing content duration increases brain alignment with models only for the tested MLLMs, while unimodal models show little improvement. Furthermore some ROIs are more aligned with shorter time windows while others are aligned with longer ones. Additionally they compare how different narrative prompts influence brain alignment across different ROIs and find that there are differences in brain predictability across prompts but also that individual voxels within ROIs are heterogenous. And finally, they show that the clips that elicit the highest predicted activity for a given voxel are more consistent for visual voxels, while those for language-related voxels vary more across context lengths and prompts. These findings are based on the Movie10 video dataset where participants watched different movies, and they essentially train voxel prediction via regression from model responses on these movies and then evaluate on unseen movies. They also estimate cross-participant prediction accuracy by predicting each participant’s brain responses from those of the other four and use this estimate to normalise model performance.
Critical strengths
The paper presents a natural extension to previous work and adds useful observations for investigating model brain alignment when varying timescales. The most interesting part is that these models seem to show context-length preferences that correspond to the cortical hierarchy of timescales. For me, the main value of this finding is what it tells us about the models’ representations. Also the observation that those model representations are somewhat well adapted to representing different narrative contexts in a brain like manner shows some value of narrative prompting in the pursuit of studying model–brain alignment.
Critical weaknesses
What is less clear, is the broader significance of these findings in regards to the brain. The hierarchical organisation was already established in previous works such as Hasson et al. (2008) and Lerner et al. (2011). The participants were not given different viewing tasks, so the prompt comparisons do not establish how those tasks would change brain responses. But following from this I find the title not necessarily a good representation of the content of the paper.
Questions for authors
How sensitive are the A4 results to the choice of top k? And how do the predicted highest clips compare to the the actual highest brain activity clips ?
Minor:
L.153 -160 what is the reason for choosing this sampling strategy if the ablation with one 1fps preserves the main trend? Why do you not just use 1 fps instead of this heterogeneous sampling strategy 9, 9, 12, 16, 24, and 30?
The dataset description and Table 2 caption refer to five participants, but the participant-preference row reports agreement as 3/4, 2/4, and 4/4. Does this analysis include only four participants?