- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a methodology, BrainVLA, for neural motor decoding using a pretrained visuomotor policy. Neural representations are aligned with language representations and replace the task instruction during inference. The policy then uses these representations, visual observations, and the rcurrent state to generate actions. Experiments are conducted on two datasets. The idea is interesting, and the results are promising. However, the baseline comparisons need better justification, and the role of language alignment is not sufficiently clear.
Critical strengths
a) The problem addressed and the method introduced are interesting. While the methods used in individual components are well established, I believe applying them to neural motor decoding seems like a useful contribution.
b) The evaluation, while limited to two datasets, is well directed and considers criteria such as trajectory accuracy and task success, which seem appropriate. The ablations also provide some useful insights into how the proposed components improve performance.
Critical weaknesses
a) The baseline comparisons need better justification. The stronger baselines, such as NoMAD and NDT2, are inspired/adapted versions, as the authors explain in Appendix C.2. Furthermore, BrainVLA receives vision and state information, whereas these baselines receive only neural input. Can the authors explain whether the improvements come from the proposed method or the additional inputs? The same-input MLP comparison in Table 2 is useful, but does not fully resolve this concern.
b) Appendix B.3 averages frozen token embeddings, which loses word-order information. This matters in D2, where exchanging which finger group should flex and which should extend changes the intended action. It is unclear whether the averaged representations still distinguish these instructions. If the action loss can distinguish them, then why do we need this particular language-alignment step?
Questions for authors
Refer to the weaknesses.
3 comments
W b: Word-order information in language embedding
Thank you for raising this important concern. We agree that mean pooling of raw frozen token embeddings loses word-order information.
(1) Contextual language representations. We replaced “frozen token embeddings” with “contextual hidden states” from the 32-layer transformer, which incorporates positional information through RoPE. All other settings remain. The results are shown below:
BrainVLA (Language embedding) D1 R² D1 SR D2 R² D2 SR
--------------------------------------------------------------------------
Raw token embedding (Submission) 0.65 0.77 0.46 1.00
Contextual hidden state (Rebuttal) 0.65 0.77 0.48 1.00
Using “contextual hidden state” preserves overall performance and yields a modest improvement on D2 R² from 0.46 to 0.48.
We also report task-level results for both representations:
Raw token embedding Contextual hidden state
D2 Tasks D2 R² D2 SR D2 R² D2 SR
--------------------------------------------------------------------------
Extend index & flex MRS 0.46 1.00 0.48 1.00
Flex index & Extend MRS 0.45 1.00 0.49 1.00
Extend/flex index 0.47 1.00 0.47 1.00
Extend/flex MRS 0.47 1.00 0.48 1.00
Extend/flex Both 0.48 1.00 0.48 1.00
Raw token embedding Contextual hidden state
D1 Tasks D1 R² D1 SR D1 R² D1 SR
--------------------------------------------------------------------------
Reach 0.52 0.95 0.57 0.95
Carry 0.63 0.94 0.58 0.94
Orient 0.40 0.59 0.47 0.60
Orient2 0.51 0.60 0.52 0.61
Shape 0.68 0.71 0.66 0.70
Grasp 0.67 0.90 0.62 0.89
Release 0.53 0.57 0.55 0.57
Notably, contextual representations improve R² on the two D2 tasks “Extend index & flex MRS” and “Flex index & Extend MRS”, which contains exchange words but with different intention. This finding are consistent with the benefit of word-order, as highlighted by the reviewer. Performance remains largely comparable on other tasks.
(2) Language alignment remain beneficial. Action loss supervises movements, while language alignment provides complementary semantic regularization. As shown in Table 2, removing alignment reduces performance, supporting its benefit beyond action supervision.
Modality (Submission Table2) D1 R² D1 SR D2 R² D2 SR
--------------------------------------------------------------------------
w/o language alignment 0.56 0.68 0.41 0.87
BrainVLA 0.65 0.77 0.48 1.00
We revised Table 13-14 in Appendix B.3, and the D2 R² results in Table 1 (highlighted in blue).
We appreciate the reviewer's insightful comment, which has helped us address the word-order limitation and better clarify the role of neural-language alignment.
Dear Reviewer, we are looking forward to hearing from you during the discussion
Dear Reviewer,
Thank you for taking the time to review our manuscript and for your meaningful feedback. We have carefully addressed your concerns and provided additional clarifications and experimental results.
With the discussion period concluding in less than two days, we would greatly appreciate it if you could take a look at our responses and let us know whether we have adequately addressed your concerns. Doing so would give us sufficient opportunity to address any follow-up questions or concerns you might have.
We really appreciate your feedback and look forward to improving our work based on your suggestions.
Thank you for your thoughtful feedback and suggestions. We are excited to read that "the problem addressed and the method introduced are interesting" and “the evaluation is appropriate”.
Weaknesses:
Thank you for this important suggestion. To address the concern about baseline fidelity, we reproduced the baselines using the official NoMAD implementation and publicly released pretrained NDT2 checkpoints. The results are shown below:
These results replace the previous NoMAD-inspired and NDT2-style values in Table 1, providing a more faithful comparison with our BrainVLA.
We updated Table 1 and Appendix C (highlighted in blue). We appreciate the reviewer's suggestion, which has helped us more rigorous baseline comparison.
Thank you for this thoughtful question. To further distinguish the contributions of method or additional inputs, we provide complementary evaluations of input modalities, neural encoder architectures, and action decoder architectures.
(1) Modality ablation. The results are shown in Table 3 (BrainVLA’s D2 R² has been updated following our response to reviewer zhKn’s weakness (b)).
Removing vision and proprio weaken the performance, which shows evidences that additional inputs contribute to BrainVLA’s performance.
(2) Alternative neural encoder. We additionally evaluated LSTM and TCN neural encoders, connected to the same frozen, task-adapted VLA under matched input conditions and training budgets. The results are shown below (BrainVLA’s D2 R² has been updated following our response to reviewer zhKn’s weakness (b)).
These encoders achieve comparable performance, with the POYO-inspired encoder consistently performing better. This indicates that BrainVLA’s effectiveness is not tied to a specific neural encoder, highlighting the flexibility of the neural-to-VLA framework.
(3) Alternative action decoders. We also retained the original neural encoder but replaced the VLA policy with MLP or transformer action decoders under matched input conditions and training budgets. The results are shown below.
Both alternative decoders perform substantially worse than BrainVLA under matched input conditions, supporting the benefit of using the adapted VLA policy.
Overall, these experiments show that both additional input modalities and model architecture contribute to BrainVLA's performance.
We have appended these results to Appendix D.4 and D.5 (highlighted in blue). We appreciate the the reviewer's suggestion, which has helped us better distinguish the contributions of method and input modalities.