- Rating
- 2(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper compares the brain alignment of vision-language models, action-specialized multimodal models, and traditional game-model baselines using fMRI recordings from 32 participants playing six Atari-style games. Model features extracted from gameplay frames under action-focused, reasoning-focused, and no-prompt conditions are used in subject-specific ridge-regression models to predict held-out brain responses. The authors report higher predictive correlations for VLMs and LAMs than for the selected baselines, larger prompt-related gains in higher-order than early visual regions, and different action-versus-reasoning variance decompositions across model families. An additional Qwen3.5 analysis examines representations read from reasoning traces and final answers.
Critical strengths
- Studying brain alignment during active gameplay connects model representations with perception and goal-directed human behavior.
- Held-out-run evaluation, feature-dimension matching, and prompt controls address several important alternative explanations for the observed alignment.
- Multiple model families and complementary regional, layer-wise, and variance-decomposition analyses provide a broad characterization of the measured effects.
Critical weaknesses
- The scientific meaning of the alignment differences remains unclear because the analyses neither distinguish specific cognitive explanations nor establish a convincing link to behavioral competence.
- The introduction's claim that reasoning representations are less aligned under both readouts contradicts the numerical reversal after mean pooling (0.036 versus 0.031, compared with 0.012 versus 0.031 for last-token readout), requiring revised wording and a direct statistical comparison.
- Because both prompts combine scene analysis with action recommendation, the purported action-unique and reasoning-unique variance components require validation under more independent manipulations or interpretation as prompt-specific predictive information.
Questions for authors
- What evidence justifies interpreting the unique variance components as action and reasoning information, given that both prompts require scene analysis and action recommendation?
- Does a direct statistical comparison of mean-pooled reasoning and final-answer representations support the claim of weaker reasoning alignment despite the numerical reversal (0.036 versus 0.031)?
5 comments
W3. Because both prompts combine scene analysis with action recommendation, the purported action-unique and reasoning-unique variance components require validation under more independent manipulations or interpretation as prompt-specific predictive information
We thank the reviewer for this suggestion. The components are defined by prompt condition, and we have renamed them action-prompt-unique and reasoning-prompt-unique variance throughout manuscript (Sec. 4).
To directly address the concern, we conducted new variance-partitioning experiments with more independent prompt manipulations: Goal (identify objectives, no motor commitment) and Action-bare (select a movement key, no justification).
- For UI-TARS, Action-bare contributes significantly more unique variance than Goal under the primary unthresholded analysis (15.4% vs. 2.9%; ΔR² = +0.0020, p = 0.037; paired t-test across 32 participants). The same direction holds at r > 0 (16.2% vs. 8.1%, p = 0.046) and r > 0.05 (24.1% vs. 14.4%, p = 0.054, marginal).
- For the untuned Qwen2-VL backbone, both unique contributions are negative (−21.8% Action-bare; −16.2% Goal), indicating that the joint model does not outperform either single-prompt model. The Action-bare versus Goal difference is not significant (p = 0.154). Thus, the base model shows no statistically significant action dominance, whereas its action-fine-tuned derivative, UI-TARS, exhibits a significant action-bare preference (p = 0.037).
This replicates the dissociation with prompts that share less content and that isolate motor commitment from goal identification. It supports reading the action-prompt-unique component as information tied to committing to a movement, rather than to scene analysis or goal identification, which both prompt pairs share.
- Changes in the revised PDF:
- components defined as action-prompt-unique / reasoning-prompt-unique in Sec. 4 (L248–252) and used in Sec. 5, RQ3 (L461–472);
- Goal vs. Action-bare analysis in App. N.3 (L1493–1511) with a summary in RQ3 (L467–472).
Q1. What evidence justifies interpreting the unique variance components as action and reasoning information, given that both prompts require scene analysis and action recommendation?
- Please see our response to W3. We clarify that the variance components quantify prompt-specific brain-predictive information, rather than isolated cognitive processes. Shared variance captures overlapping predictive information, while unique components measure what each prompt contributes beyond the other.
Four findings support their functional interpretation:
- Independent prompt manipulation. Our new Goal vs. Action-bare experiment confirms action dominance in UI-TARS (15.4% vs. 2.9%, p = 0.037), but not in its Qwen2-VL backbone (p = 0.154).
- Anatomical specificity. The action-prompt-unique contribution is particularly pronounced in SMA (34% of joint explained variance for UI-TARS), with weaker contributions in early visual cortex.
- Regional replication under the new controls. Under these new controls, total prompt-unique variance is substantially greater in SMA (45.8%; Action-bare 24.1%, Goal 21.7%) than in early visual cortex (6.2%; Action-bare 0.8%, Goal 5.4%). Indicates that prompt-specific information is particularly pronounced in motor-associated cortex
- Fine-tuning comparison. Relative to Qwen2-VL, UI-TARS exhibits increased action-prompt-unique and reduced reasoning-prompt-unique contributions (t(31) = 6.20, p < 0.001), despite comparable raw brain-encoding accuracy (p = 0.64).
Together, these findings support interpreting action-prompt-unique variance as brain-predictive information associated with motor commitment. We interpret unique and shared variance as properties of predictive representations, rather than a decomposition of distinct cognitive functions.
- Changes in the revised PDF: stated in Sec. 4, “Variance partitioning” (L248–252) and App. G.
Q2. Does a direct statistical comparison of mean-pooled reasoning and final-answer representations support the claim of weaker reasoning alignment despite the numerical reversal (0.036 versus 0.031)?
Yes, The apparent numerical reversal resulted from a reporting error: 0.036 is the mean-pooled final-answer readout, whereas the mean-pooled reasoning readout is 0.009. The submitted text incorrectly attributed 0.036 to reasoning, which we have corrected.
- The direct matched comparison (paired t-test across 32 participants) gives final answer > reasoning under mean pooling (r = 0.036 vs. 0.009, p = 0.054), in the same direction as the last-token comparison (0.031 vs. 0.012, p = 0.032). The mean-pooled difference is marginal, so we now describe the reasoning-trace deficit as significant for last-token readouts and marginal for mean-pooled readouts (Sec. 5 RQ3; App. O).
Request to review our responses
Dear Reviewer iJKK,
As we approach the end of the discussion phase, we sincerely request that you review our responses. We have made every effort to carefully address all your concerns and provide detailed clarifications.
If you find our responses satisfactory, we would be grateful if you could consider reassessing your evaluation of our paper. We remain fully committed to addressing any additional questions or concerns you may have.
Thank you for your time, thoughtful feedback, and consideration.
Thanks
W1/Q1 The scientific meaning of the alignment differences remains unclear because the analyses neither distinguish specific cognitive explanations nor establish a convincing link to behavioral competence.
We thank the reviewer for this important comment. Our findings provide complementary evidence at three levels: brain predictivity, representational organization, and behavioral relevance.
First, variance partitioning reveals distinct representational organization despite comparable encoding accuracy. Our encoding models predict held-out fMRI responses across 32 participants using leave-one-run-out cross-validation, establishing measurable brain alignment during gameplay.
Second, prompt controls and regional analyses support this interpretation.
Third, behavioral analyses provide complementary evidence.
Together, these results demonstrate that brain encoding reveals measurable, regionally structured neural predictivity, while variance partitioning identifies differences in representational organization that are not apparent from overall prediction accuracy alone.
W2. The introduction's claim that reasoning representations are less aligned under both readouts contradicts the numerical reversal after mean pooling (0.036 versus 0.031, compared with 0.012 versus 0.031 for last-token readout), requiring revised wording and a direct statistical comparison.
We thank the reviewer for identifying this inconsistency. We clarify that the reported r=0.036 corresponds to the mean-pooled final-answer representation, not the reasoning representation. The submitted text incorrectly attributed this value.
We performed matched readout comparisons across 32 participants using leave-one-run-out cross-validation.
Final-answer representations show higher whole-cortex brain alignment than reasoning-span representations under both last-token readout (r = 0.031 vs. 0.012, p = 0.032) and mean-pooled readout (r = 0.036 vs. 0.009, p = 0.054).
Thus, the direction is consistent across readouts, with a statistically significant difference for last-token representations and a marginal, non-significant difference for mean-pooled representations.
Changes in the revised PDF: We corrected the
All sections now report matched comparisons, distinguish significant from marginal findings, and remove the unsupported claim that readout position drives the difference.