acceptodds
Under review as a conference paper at ICLR 2027

Brain alignment of reasoning and action representations from vision–language and action models during naturalistic gameplay

Abstract

Understanding how humans and artificial intelligence systems predict and plan by interacting with their environment is a fundamental challenge at the intersection of neuroscience and machine learning. Most brain-encoding studies focus on aligning artificial models with brain activity during language comprehension or passive visual processing, while interactive brain-alignment studies have to date been largely limited to reinforcement-learning (RL) agents and theory-based models. To address this gap, we study brain alignment of representative models from two foundation-model types, namely vision-language models (VLMs) and large-action models (LAMs), using fMRI recordings from participants playing naturalistic Atari-style video games. Specifically, we examine how action-focused and reasoning-focused prompts shape the model's internal representations and their alignment with fMRI brain activity. First, we find that both VLMs and LAMs achieve significantly higher voxel-wise encoding performance than RL baselines, with the advantage holding even under matched feature dimensionality. Second, compared to a “no-prompt” baseline, prompt-driven gains are larger in higher-order frontal-parietal and motor-planning regions than in early visual cortex, roughly – when averaged over ROI groups, although individual regions are heterogeneous. Third, variance partitioning reveals a qualitatively different representational organization. VLM representations are prompt-symmetric (12.5% unique action vs. 13.6% unique reasoning), whereas LAM is action-dominant (27% unique action vs.% unique reasoning), with the asymmetry strongest in frontal-motor cortex. Together, these results demonstrate that action-specialized fine-tuning reorganizes multimodal game-state representations in ways that change their alignment with action-associated cortical regions, a difference that is invisible at the level of raw prediction accuracy.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain encoding, fMRI, encoding models, atari games, vision-language models, action models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_iJKKOpenReview ↗
Rating
2(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper compares the brain alignment of vision-language models, action-specialized multimodal models, and traditional game-model baselines using fMRI recordings from 32 participants playing six Atari-style games. Model features extracted from gameplay frames under action-focused, reasoning-focused, and no-prompt conditions are used in subject-specific ridge-regression models to predict held-out brain responses. The authors report higher predictive correlations for VLMs and LAMs than for the selected baselines, larger prompt-related gains in higher-order than early visual regions, and different action-versus-reasoning variance decompositions across model families. An additional Qwen3.5 analysis examines representations read from reasoning traces and final answers.

Critical strengths
  1. Studying brain alignment during active gameplay connects model representations with perception and goal-directed human behavior.
  2. Held-out-run evaluation, feature-dimension matching, and prompt controls address several important alternative explanations for the observed alignment.
  3. Multiple model families and complementary regional, layer-wise, and variance-decomposition analyses provide a broad characterization of the measured effects.
Critical weaknesses
  1. The scientific meaning of the alignment differences remains unclear because the analyses neither distinguish specific cognitive explanations nor establish a convincing link to behavioral competence.
  2. The introduction's claim that reasoning representations are less aligned under both readouts contradicts the numerical reversal after mean pooling (0.036 versus 0.031, compared with 0.012 versus 0.031 for last-token readout), requiring revised wording and a direct statistical comparison.
  3. Because both prompts combine scene analysis with action recommendation, the purported action-unique and reasoning-unique variance components require validation under more independent manipulations or interpretation as prompt-specific predictive information.
Questions for authors
  1. What evidence justifies interpreting the unique variance components as action and reasoning information, given that both prompts require scene analysis and action recommendation?
  2. Does a direct statistical comparison of mean-pooled reasoning and final-answer representations support the claim of weaker reasoning alignment despite the numerical reversal (0.036 versus 0.031)?
5 comments

W1/Q1 The scientific meaning of the alignment differences remains unclear because the analyses neither distinguish specific cognitive explanations nor establish a convincing link to behavioral competence.

We thank the reviewer for this important comment. Our findings provide complementary evidence at three levels: brain predictivity, representational organization, and behavioral relevance.

  • First, variance partitioning reveals distinct representational organization despite comparable encoding accuracy. Our encoding models predict held-out fMRI responses across 32 participants using leave-one-run-out cross-validation, establishing measurable brain alignment during gameplay.

    • Qwen2.5-VL shows balanced Action- and Reasoning-prompt contributions (12.4% vs. 9.5%), whereas UI-TARS is action-dominant (25.6% vs. −8.2%; p = 0.032), most strongly in SMA (34% vs. −14%).
    • Our new backbone-controlled analysis using identical prompts and encoding procedures, Qwen2-VL, the base model of UI-TARS, also shows balanced contributions (7.6% vs. 9.9%), and the between-model asymmetry is significant (t(31) = 6.20, p < 0.001) while raw encoding accuracy is unchanged (p = 0.64).
    • The dissociation also persists under more independent Goal versus Action-bare prompts (15.4% vs. 2.9%, p = 0.037), supporting an association between action specialization and the organization of brain-predictive information.
  • Second, prompt controls and regional analyses support this interpretation.

    • Across seven additional prompt controls (App. E, Table 1; App. L, Fig. 8), scrambled, color-only and scene-only prompts align weakly (r ≈ 0.006–0.029), well below task-framed prompts (≈ 0.047), so the gains are not explained by text presence or visual description alone.
    • Centered CKA reveals distinct stimulus-dependent representations across prompts, and prompt gains are significantly greater in higher-order than in early visual cortex (VLM: Δr = +0.012, q = 0.019; LAM: Δr = +0.020, q = 0.039), with the largest LAM gains in SMA (+0.057) and MFG (+0.042).
  • Third, behavioral analyses provide complementary evidence.

    • Model features decode participants’ TR-level actions above chance under leave-one-run-out cross-validation (balanced accuracy: VLM 0.342, LAM 0.362, EMPA 0.289; mean chance 0.167; App. J, Table 6), while model–human action agreement is modest and game-dependent (App. J, Table 5). We claim behaviorally relevant information in the representations, not human-like competence.

Together, these results demonstrate that brain encoding reveals measurable, regionally structured neural predictivity, while variance partitioning identifies differences in representational organization that are not apparent from overall prediction accuracy alone.

  • Changes in the revised PDF:
    • backbone control in Sec. 5, RQ3 (L461–466) and App. N.2 (L1444–1489);
    • independent prompt controls in RQ3 (L467–472) and App. N.3 (L1493–1511);
    • interpretation separated from cognitive-mechanism claims in Sec. 4 (L248–252) and the Discussion (L511–518).

W2. The introduction's claim that reasoning representations are less aligned under both readouts contradicts the numerical reversal after mean pooling (0.036 versus 0.031, compared with 0.012 versus 0.031 for last-token readout), requiring revised wording and a direct statistical comparison.

We thank the reviewer for identifying this inconsistency. We clarify that the reported r=0.036r = 0.036 corresponds to the mean-pooled final-answer representation, not the reasoning representation. The submitted text incorrectly attributed this value.

  • We performed matched readout comparisons across 32 participants using leave-one-run-out cross-validation.

  • Final-answer representations show higher whole-cortex brain alignment than reasoning-span representations under both last-token readout (r = 0.031 vs. 0.012, p = 0.032) and mean-pooled readout (r = 0.036 vs. 0.009, p = 0.054).

  • Thus, the direction is consistent across readouts, with a statistically significant difference for last-token representations and a marginal, non-significant difference for mean-pooled representations.

  • Changes in the revised PDF: We corrected the

    • Introduction (Contribution 4, L112–116),
    • Results (L494–497),
    • Discussion (L518–522), and Appendix O (L1650–1666).

All sections now report matched comparisons, distinguish significant from marginal findings, and remove the unsupported claim that readout position drives the difference.

W3. Because both prompts combine scene analysis with action recommendation, the purported action-unique and reasoning-unique variance components require validation under more independent manipulations or interpretation as prompt-specific predictive information

We thank the reviewer for this suggestion. The components are defined by prompt condition, and we have renamed them action-prompt-unique and reasoning-prompt-unique variance throughout manuscript (Sec. 4).

To directly address the concern, we conducted new variance-partitioning experiments with more independent prompt manipulations: Goal (identify objectives, no motor commitment) and Action-bare (select a movement key, no justification).

  • For UI-TARS, Action-bare contributes significantly more unique variance than Goal under the primary unthresholded analysis (15.4% vs. 2.9%; ΔR² = +0.0020, p = 0.037; paired t-test across 32 participants). The same direction holds at r > 0 (16.2% vs. 8.1%, p = 0.046) and r > 0.05 (24.1% vs. 14.4%, p = 0.054, marginal).
  • For the untuned Qwen2-VL backbone, both unique contributions are negative (−21.8% Action-bare; −16.2% Goal), indicating that the joint model does not outperform either single-prompt model. The Action-bare versus Goal difference is not significant (p = 0.154). Thus, the base model shows no statistically significant action dominance, whereas its action-fine-tuned derivative, UI-TARS, exhibits a significant action-bare preference (p = 0.037).

This replicates the dissociation with prompts that share less content and that isolate motor commitment from goal identification. It supports reading the action-prompt-unique component as information tied to committing to a movement, rather than to scene analysis or goal identification, which both prompt pairs share.

  • Changes in the revised PDF:
    • components defined as action-prompt-unique / reasoning-prompt-unique in Sec. 4 (L248–252) and used in Sec. 5, RQ3 (L461–472);
    • Goal vs. Action-bare analysis in App. N.3 (L1493–1511) with a summary in RQ3 (L467–472).

Q1. What evidence justifies interpreting the unique variance components as action and reasoning information, given that both prompts require scene analysis and action recommendation?

  • Please see our response to W3. We clarify that the variance components quantify prompt-specific brain-predictive information, rather than isolated cognitive processes. Shared variance captures overlapping predictive information, while unique components measure what each prompt contributes beyond the other.

Four findings support their functional interpretation:

  • Independent prompt manipulation. Our new Goal vs. Action-bare experiment confirms action dominance in UI-TARS (15.4% vs. 2.9%, p = 0.037), but not in its Qwen2-VL backbone (p = 0.154).
  • Anatomical specificity. The action-prompt-unique contribution is particularly pronounced in SMA (34% of joint explained variance for UI-TARS), with weaker contributions in early visual cortex.
  • Regional replication under the new controls. Under these new controls, total prompt-unique variance is substantially greater in SMA (45.8%; Action-bare 24.1%, Goal 21.7%) than in early visual cortex (6.2%; Action-bare 0.8%, Goal 5.4%). Indicates that prompt-specific information is particularly pronounced in motor-associated cortex
  • Fine-tuning comparison. Relative to Qwen2-VL, UI-TARS exhibits increased action-prompt-unique and reduced reasoning-prompt-unique contributions (t(31) = 6.20, p < 0.001), despite comparable raw brain-encoding accuracy (p = 0.64).

Together, these findings support interpreting action-prompt-unique variance as brain-predictive information associated with motor commitment. We interpret unique and shared variance as properties of predictive representations, rather than a decomposition of distinct cognitive functions.

  • Changes in the revised PDF: stated in Sec. 4, “Variance partitioning” (L248–252) and App. G.

Q2. Does a direct statistical comparison of mean-pooled reasoning and final-answer representations support the claim of weaker reasoning alignment despite the numerical reversal (0.036 versus 0.031)?

Yes, The apparent numerical reversal resulted from a reporting error: 0.036 is the mean-pooled final-answer readout, whereas the mean-pooled reasoning readout is 0.009. The submitted text incorrectly attributed 0.036 to reasoning, which we have corrected.

  • The direct matched comparison (paired t-test across 32 participants) gives final answer > reasoning under mean pooling (r = 0.036 vs. 0.009, p = 0.054), in the same direction as the last-token comparison (0.031 vs. 0.012, p = 0.032). The mean-pooled difference is marginal, so we now describe the reasoning-trace deficit as significant for last-token readouts and marginal for mean-pooled readouts (Sec. 5 RQ3; App. O).
AuthorsOpenReview ↗

Request to review our responses

Dear Reviewer iJKK,

As we approach the end of the discussion phase, we sincerely request that you review our responses. We have made every effort to carefully address all your concerns and provide detailed clarifications.

If you find our responses satisfactory, we would be grateful if you could consider reassessing your evaluation of our paper. We remain fully committed to addressing any additional questions or concerns you may have.

Thank you for your time, thoughtful feedback, and consideration.

Thanks

Reviewer_NFdPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper study brain alignment of vision-language models (VLMs) and large-action models (LAMs), using fMRI recordings from participants playing naturalistic Atari-style video games. The aim of such experiment is to understand the connection between AI and human brain. There are some claims such as VLMs and LAMs aligns better than RL baselines, and prompt-driven models aligns better.

Critical strengths
  1. It is interesting to see the alignment of models and human brains.

  2. The overall results are easy to follow and the presentation is clear.

Critical weaknesses
  1. The evaluation seems narrow, the experiments only cover Atari-style video games, so the generalizability is not sure.

  2. I am not familiar with neuroscience, so I am not sure about the contribution on it. But in terms of building VLMs and LAMs, it is hard to get insights from the results.

Questions for authors
  1. Do the authors have some takeaways from the results? It says in the paper that "VLMs show balanced contributions from Action and Reasoning prompts, whereas action-tuned LAMs are strongly action-dominant, especially in frontal-motor regions." Isn't that expected that "action-tuned LAMs are strongly action-dominant"? And by these results, do the authors suggest more reasoning data or action data in model training?
3 comments

W1. The evaluation seems narrow, the experiments only cover Atari-style video games, so the generalizability is not sure.

We thank the reviewer for this question.

  • Our study focuses on Atari-style gameplay as an established experimental paradigm for investigating goal-directed behavior, decision-making, and planning (Tomov et al., 2023).

  • Our contribution is to extend brain-alignment research from passive listening, viewing, to interactive, goal-directed behavior.

  • Prior studies have used language models to investigate language comprehension (Schrimpf et al., 2021; Toneva & Whebe et al. 2019, Caucheteux & King, 2022) and vision/multimodal models to study perceptual processing. As VLMs and LAMs increasingly support reasoning and action-oriented tasks, our study investigates how action- and reasoning-focused prompts shape their internal representations and alignment with human brain activity during gameplay.

  • Importantly, our evaluation is not restricted to a single game or model.

    • We analyze recordings from 32 participants playing six distinct Atari-style games, including bait, chase, helper, lemmings, plaqueAttack, and zelda. These games involve different combinations of visual processing, navigation, goal identification, environmental interaction, and action selection.
    • Further, our leave-one-run-out evaluation holds out a complete scanner run corresponding to a game, training the brain-encoding model on the remaining runs. Thus, the evaluation measures whether the learned mapping between model representations and brain responses transfers to gameplay from an unseen run/game, rather than merely interpolating within the same gameplay sequence.
    • We further evaluate two VLMs and two LAMs, observing qualitatively consistent representational patterns across model comparisons.

Thus, we present a framework for studying foundation-model representations during goal-directed interaction, validated on an established experimental paradigm with available human neural data, without claiming generalization to all interactive environments.

  • Changes in the revised PDF: scope statement in Limitations, App. P (L1670–1673).

W2. I am not familiar with neuroscience, so I am not sure about the contribution on it. But in terms of building VLMs and LAMs, it is hard to get insights from the results.

We thank the reviewer for the candid comment.

  • Our primary contribution is neuroscientific: extending brain-alignment analysis of foundation models to human brain activity recorded during interactive, goal-directed gameplay, building on prior RL- and theory-based approaches. For VLM/LAM development, we identify four practical takeaways.

  • Action fine-tuning changes representational organization without detectable improvement in overall brain alignment.

    • UI-TARS and its Qwen2-VL backbone show comparable encoding accuracy (p = 0.64), yet significantly different action–reasoning variance-partitioning asymmetries (t(31) = 6.20, p < 0.001).
    • In UI-TARS, reasoning-prompt-unique contributions become negative (−8.2% whole-brain; −14% in SMA), whereas Qwen2-VL retains positive contributions from both prompts.
    • This suggests that action specialization can make reasoning-prompt information redundant with action-prompt information.
    • A testable training hypothesis is whether pairing action trajectories with explicit rationales better preserves such complementary representations.
  • Prompt conditioning affects representation geometry.

    • Centered CKA reveals low similarity between Action- and Reasoning-prompt representations, while task-oriented prompts produce stronger brain alignment than scene-only or scrambled controls. Thus, the prompt should be treated as part of the representation when extracting VLM features for downstream evaluation.
  • Representation readout matters.

    • In Qwen3.5, final-answer representations show higher alignment than reasoning-span representations under matched last-token readouts (0.031 vs. 0.012; p = 0.032), with the same direction but marginal significance under mean pooling (0.036 vs. 0.009; p = 0.054). This highlights the importance of controlling readout strategies when analyzing reasoning-trace representations.
  • Brain alignment provides a complementary diagnostic.

    • Using publicly available fMRI data and ridge-based encoding with variance partitioning, we identify representational differences that overall encoding accuracy does not reveal. We propose this as a complementary research probe alongside behavioral evaluation, rather than a replacement for conventional benchmarks.
  • Changes in the revised PDF: Discussion, findings (L503–522) and “Implications for model training” (L523–529).

Q1. Do the authors have some takeaways from the results? Isn't that expected that "action-tuned LAMs are strongly action-dominant"? And by these results, do the authors suggest more reasoning data or action data in model training?

We thank the reviewer for this question. That an action-tuned model responds more to action instructions is expected. Two things in our results are not.

  • First, action specialization changes representational organization without a detectable improvement in overall brain alignment.

    • UI-TARS and its Qwen2-VL backbone show comparable encoding accuracy (p = 0.64), yet significantly different action–reasoning variance-partitioning asymmetries (t(31) = 6.20, p < 0.001). This action-oriented pattern persists under our new Goal vs. Action-bare control (15.4% vs. 2.9%, p = 0.037).
  • Second, reasoning-prompt-specific predictive contributions become negative in UI-TARS (−8.2% whole-brain; −14% in SMA), whereas Qwen2-VL retains positive contributions from both the Action and Reasoning prompts (7.6% and 9.9%). Thus, action specialization is associated not merely with stronger action-oriented representations, but also with reduced complementary brain-predictive information from reasoning-conditioned representations.

  • Regarding training implications, our results do not establish that more reasoning or action data is preferable.

    • Notably, our Qwen3.5 analysis shows that generated reasoning traces are not better aligned than final answers, so additional reasoning supervision is not an obvious remedy either.
    • Instead, our findings motivate testing whether action trajectories paired with explicit rationales can better preserve complementary representations than action-only supervision. This remains a hypothesis, not a demonstrated training benefit.
  • Changes in the revised PDF: the two non-obvious effects and the training hypothesis are stated in the Discussion (L511–518 and L523–529).

Dear Reviewer NFdP,

As we approach the end of the discussion phase, we sincerely request that you review our responses. We have made every effort to carefully address all your concerns and provide detailed clarifications.

If you find our responses satisfactory, we would be grateful if you could consider reassessing your evaluation of our paper. We remain fully committed to addressing any additional questions or concerns you may have.

Thank you for your time, thoughtful feedback, and consideration.

Thanks

Reviewer_Ja6oOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares VLM and large-action-model representations with fMRI recordings from 32 participants during Atari-style gameplay. It reports improved encoding with task-focused prompts and different action-versus-reasoning variance contributions across model types.

Critical strengths
  1. This paper extends foundation-model brain alignment to active gameplay, with evaluation that holds out complete scanner runs.
  2. It includes useful prompt controls, matched feature dimensions, and multiple models. Variance partitioning examines representational differences beyond overall prediction accuracy.
Critical weaknesses

UI-TARS and OS-Atlas derive from Qwen2-VL, whereas the main VLM is Qwen2.5-VL. Matching architecture dimensions does not control for pretraining differences, so the observed dissociation cannot be attributed specifically to action specialization.

Questions for authors
  1. Are the representations computed separately at each fMRI time point or shared across an entire game level?
  2. EMPA is represented by eight theory-based variables, and DDQN is described through action summaries, whereas the foundation models contribute learned hidden states. Could you evaluate RL hidden-state features extracted from the same gameplay observations, using matched feature dimensions and the same encoding procedure? This would distinguish a model advantage from differences in feature selection.
4 comments

W1. UI-TARS and OS-Atlas derive from Qwen2-VL, whereas the main VLM is Qwen2.5-VL. Matching architecture dimensions does not control for pretraining differences, so the observed dissociation cannot be attributed specifically to action specialization.

We thank the reviewer for raising this important concern.

  • We addressed this directly by evaluating the untuned Qwen2-VL backbone from which both UI-TARS and OS-Atlas are derived. This yields a within-lineage comparison in which the only difference between base and fine-tuned model is action specialization.
  • We applied the same brain-encoding procedure, using leave-one-run-out cross-validation across 32 participants and a prespecified layer. The Qwen2-VL baseline achieves the following performance under Action and Reasoning prompts:
ConditionnFULLr>0r>0.05
VLM (Qwen2-VL) Action320.0268 ± 0.03750.1008 ± 0.03230.1400 ± 0.0302
VLM (Qwen2-VL) Reasoning320.0441 ± 0.06650.1117 ± 0.06100.1520 ± 0.0578
  • Raw alignment does not change with action fine-tuning.

    • UI-TARS and Qwen2-VL are statistically indistinguishable under the Action prompt (Δr = +0.006, p = 0.64 for FULL; Δr = +0.0004, p = 0.95 for r > 0.05), and no fine-tuned-versus-base comparison survives FDR correction. This matches our central observation that raw encoding accuracy does not distinguish VLMs from LAMs.
  • Representational organization does change.

    • Qwen2-VL shows balanced action- and reasoning-prompt-unique contributions (7.6% vs. 9.9%), matching the prompt-symmetric pattern of Qwen2.5-VL (12.4% vs. 9.5%).
    • UI-TARS is action-dominant (25.6% vs. −8.2%). The action–reasoning asymmetry therefore shifts from −2.3 percentage points in the base model to +33.8 in its action-specialized derivative, and the between-model difference is significant (paired t-test across 32 participants, t(31) = 6.20, p < 0.001, dz = 1.10).
  • Because base and fine-tuned model share architecture and pretraining, this comparison removes the pretraining confound the reviewer's concern about. The action-dominant pattern is absent in Qwen2-VL and present in UI-TARS: action specialization redistributes brain-predictive variance between prompts without changing overall alignment.

  • Changes in the revised PDF:

    • Qwen2-VL control introduced in Sec. 3 (L165–169);
    • backbone-controlled results in Sec. 5, RQ3 (L461–466) and App. N.2 (L1444–1489).

Q1. Are the representations computed separately at each fMRI time point or shared across an entire game level?

We thank the reviewer for this question.

  • Representations are extracted independently at each fMRI time point (TR), rather than being shared across an entire game level. Specifically, game states are reconstructed frame by frame from the behavioral logs at the native logging rate of approximately 20 Hz.

  • This reconstruction is performed separately for each participant using their individual gameplay actions recorded in plays.bson. For each TR (2 seconds), we sample four frames from the corresponding gameplay window and extract a separate model representation.

  • We employ leave-one-run-out cross-validation, where one complete game run is held out for testing and the remaining runs are used for training. Importantly, feature windows are constructed independently within each run and never cross run boundaries. We do not extract or share representations across entire game levels, thereby avoiding temporal overlap between training and test runs and reducing the risk of information leakage.

  • Thus, the representations are therefore participant-specific, TR-aligned, and evaluated under strict run-level separation.

Q2. Could you evaluate RL hidden-state features extracted from the same gameplay observations, using matched feature dimensions and the same encoding procedure? This would distinguish a model advantage from differences in feature selection.

Thank you for this suggestion.

  • Our evaluation followed Tomov et al. (2023) using the publicly released baseline features: eight theory-derived variables for EMPA and 14 summary regressors for DDQN.

    • The trained DDQN checkpoints were not publicly released, preventing direct extraction of CNN hidden-layer representations from the same gameplay observations.
    • Reproducing these representations would require retraining six game-specific agents for 25 million steps each, which was beyond the scope of the rebuttal period.
  • To address potential differences in feature dimensionality, we performed additional comparisons using PCA fitted exclusively on training folds.

    • At eight dimensions, prompted Qwen2-VL outperforms EMPA (r = 0.030 vs. 0.014, Δr = +0.016, p = 0.031). Reducing Qwen2-VL from 3,584 to eight dimensions produces no detectable change in alignment (Δr = −0.002, p = 0.67).
    • At 14 dimensions, it also outperforms the available DDQN regressors (Δr = +0.032, p = 0.018). These results indicate that feature count alone does not explain the observed advantage over the evaluated baselines.
  • Changes in the revised manuscript: Dimensionality-matched comparisons in Sec. 5 (RQ1, L274–279); DDQN baseline clarification in Sec. 3 (L175–178) and the Discussion (L503–507).

Dear Reviewer Ja6o,

As we approach the end of the discussion phase, we sincerely request that you review our responses. We have made every effort to carefully address all your concerns and provide detailed clarifications.

If you find our responses satisfactory, we would be grateful if you could consider reassessing your evaluation of our paper. We remain fully committed to addressing any additional questions or concerns you may have.

Thank you for your time, thoughtful feedback, and consideration.

Thanks

Dear Reviewer Ja6o,

As the discussion period closes in less than 24 hours, we wanted to briefly follow up on our responses and additional analyses.

For your main concern (W1), we added a backbone-controlled comparison: the untuned Qwen2-VL model shows no action dominance, UI-TARS does (t(31) = 6.20, p < 0.001), at unchanged encoding accuracy (p = 0.64). For Q2, we added PCA-matched baselines (Sec. 5, RQ1) and now describe the DDQN baseline accurately. Both are in the revised PDF.

If our responses have satisfactorily addressed your concerns, we would be grateful if you would consider updating your assessment. We would also be happy to clarify any remaining questions.

Thank you for your time and constructive feedback.

Best regards, The Authors

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. We summarize the main additions below.

Overall, our central findings hold under the additional controls and validations (backbone-controlled comparisons, independent prompt manipulations, regional analyses, matched reasoning readouts, and dimensionality controls): VLMs and LAMs predict brain activity during goal-directed gameplay better than the evaluated RL feature baselines, including under matched feature dimensionality; task-framed prompting produces regionally differentiated alignment gains; and action fine-tuning reorganizes brain-predictive representations without a detectable improvement in overall encoding accuracy. The new analyses strengthen the link between action fine-tuning and the VLM/LAM representational dissociation, with the action-oriented pattern persisting under more independent prompt manipulations. We have also clarified that variance components measure prompt-specific predictive information, rather than isolated cognitive processes.

1. Backbone-controlled validation of action fine-tuning (Ja6o, NFdP, iJKK; Sec. 5 RQ3, L461-466; App. N.2, L1444–1489). We evaluated Qwen2-VL, the original backbone of UI-TARS, using identical prompts and encoding procedures. Despite comparable raw brain-encoding accuracy (p = 0.64), the models exhibit significantly different action-reasoning variance-partitioning asymmetries (t(31)=6.20, p < 0.001). Qwen2-VL shows balanced contributions (7.6% action-prompt-unique vs. 9.9% reasoning-prompt-unique), whereas UI-TARS is action-dominant (25.6% vs. -8.2%). Because the models share architecture and pretraining lineage, this comparison strengthens the evidence linking the dissociation to action fine-tuning.

2. New baselines and controls.

  • Independent prompt validation (iJKK; Sec. 5 RQ3, L467-472; App. N.3, L1493–1511). We conducted additional variance partitioning using Goal (objective identification without motor commitment) and Action-bare (movement selection without justification). UI-TARS retains significantly greater Action-bare-unique than Goal-unique variance (15.4% vs. 2.9%, p = 0.037), whereas Qwen2-VL shows no significant difference (p = 0.154). The direction persists at r > 0 (p = 0.046) and r > 0.05 (p = 0.054, marginal).
  • Regional specificity (iJKK; App. N.3). Under the new prompt controls for UI-TARS, total prompt-unique variance is descriptively substantially greater in SMA (45.8%; Action-bare 24.1%, Goal 21.7%) than in early visual cortex (6.2%; Action-bare 0.8%, Goal 5.4%). This supports regional differentiation of prompt-specific predictive information.
  • Dimensionality-matched baselines (Ja6o; Sec. 5 RQ1, L274-279; DDQN baseline described as summary regressors in Sec. 3, L175-178). Using training-fold PCA, we reduced prompted Qwen2-VL features to match EMPA (8 dimensions) and DDQN regressors (14 dimensions). PCA-8 achieves significantly higher brain alignment than EMPA (r = 0.030 vs. 0.014; Δr=+0.016, p = 0.031), while PCA-14 achieves higher brain alignment than the evaluated DDQN regressors (r = 0.033 vs. 0.001; Δr=+0.032, p = 0.018). Reducing Qwen2-VL from 3,584 to 8 dimensions produces only a small, nonsignificant change (Δr = -0.002, p = 0.667).
  • Matched reasoning readouts (iJKK; Introduction, L112-116; Sec. 5 RQ3, L494-497; Discussion, L518-522; App. O, L1650-1666). We corrected the Qwen3.5 readout ambiguity and conducted matched comparisons across 32 participants. Final-answer representations show significantly higher brain alignment than reasoning spans under last-token readout (r = 0.031 vs. 0.012, p = 0.032), with the same direction but marginal significance under mean pooling (r = 0.036 vs. 0.009, p = 0.054).

3. Reframing and clarifications (Ja6o, NFdP, iJKK).

  • Renamed the variance components action-prompt-unique and reasoning-prompt-unique, clarifying that they measure prompt-specific predictive information rather than isolated cognitive processes (Sec. 4, L248-252).
  • Clarified participant-specific, TR-aligned feature extraction, run-level stacking, and leave-one-run-out evaluation (Sec. 3, L180-190).
  • Described the DDQN baseline as 14 summary regressors released by Tomov et al. (2023), not learned hidden-layer activations (Sec. 3, L175-178; Discussion, L503-507).
  • Expanded the Discussion to distinguish neuroscience contributions from implications for VLM/LAM development, including action specialization, prompt conditioning, reasoning readouts, and brain alignment as a complementary diagnostic (L503-529).
  • Clarified the evaluation scope across 32 participants, six games, two VLMs, and two LAMs, and acknowledged remaining generalizability and behavioral limitations (App. P).

These additional controls strengthen our conclusions on brain alignment during goal-directed gameplay, show that action fine-tuning shifts prompt-specific brain-predictive information, and clarify that these representations do not isolate cognitive processes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.