- Rating
- 1(−1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks whether EEG recorded while people do manipulation tasks can be used as an extra training signal for VLA models. The authors record 128-channel EEG and egocentric video from two participants on 12 tasks. During LoRA fine-tuning, the egocentric frames and instructions also pass through the backbone, and a small MLP head on the hidden state at the final prompt token predicts the EEG of the next 1-s window (MSE plus InfoNCE). The head is dropped at test time. Against robot-only fine-tuning, the reported gains are +0.3 to +6.0 points, for OpenVLA on four LIBERO suites and three RL4VLA OOD settings, and for π0 on LIBERO. On LIBERO-Long, shuffling the EEG targets brings performance back to the robot-only level. The data are not released, so the paper offers no new dataset contribution. All robot experiments are in simulation; there are no real-robot experiments.
Critical strengths
1. A new and interesting question.
The paper asks whether neural signals recorded while people act can serve as an extra training signal for robot policies (p.1–2). This question is new and interesting, and it is the main reason I think the work deserves attention.
Critical weaknesses
1. The data are too small to be a core contribution, and the paper does not say clearly how much it uses.
The paper lists the synchronized EEG–behavior data paradigm as its first contribution (p.2), but the data come from two participants on 12 tasks. The paper never clearly states how much data is used for training. The main text says Fig. 2(B) summarizes the data of both participants, but App. A.1 notes that it shows only a 689-episode subset. The 1,224 episodes and 6,291 one-second windows (about 1.75 hours of execution-phase EEG) are given only for the decoding analysis (App. A.1, A.4), and only the supplementary code shows that training uses these same episodes. Neither the size of the training set nor the amount of data per participant is reported.
What this field lacks most is data at scale. Non-invasive EEG is much easier to collect than invasive recordings, so adding participants is feasible. Yet the paper mentions no plan to expand the data and does not commit to releasing it: the Reproducibility Statement (p.10) covers only code, and the supplementary README says the data were left out for size and privacy reasons. With two participants, the results cannot support a general claim about human neural dynamics.
2. All robot results are in simulation.
Every robot experiment runs in simulation (LIBERO and RL4VLA), while the human data come from the real world. Simulated success rates can overstate real-world performance and vary from run to run, and the gains here are small: 8 of the 11 comparisons in Tables 1–2 improve by only 0.3–1.6 points. If ± is the SD over three seeds (as for the main run in the supplement), only OpenVLA on LIBERO-Long, Vision OOD and Execution OOD reach p < 0.05. Without real-robot results, the paper says little about embodied intelligence in the physical world, which is the setting its own discussion stresses (App. C).
Overall. The motivation is positive and worth exploring, but two participants and simulation-only results cannot support the paper's core contribution. I would encourage the authors to bring this work to a related ICLR workshop first, scale up the data, and use the workshop to learn from researchers in neighboring communities which data are most needed, before resubmitting. I believe this would lead to a much more solid paper.
Questions for authors
Tags: [C] = clarification; [A] = additional analysis or experiment, feasible on the existing pipeline. The questions are ordered by how much the answers matter for my assessment. Suggestions for future work are listed at the end and do not affect my rating.
Q1 [C] How much data is used, and how was it collected? (Weakness 1)
- How much data is used for training in total (episodes, hours and windows), and how much comes from each participant? Please report this in the main text.
- How do the two source datasets (App. A.1; "v1" and "ego128" in the supplied code) relate to the two participants, their sessions and their recording days? App. A.1 states that each participant completed all 12 tasks, yet five of the 12 tasks have episodes only from "v1" (
prepare_waveform.py, l.24–37). - Were tasks recorded in blocks or interleaved? How were EEG and video synchronized, and how accurately? Fig. 2(C) plots time since a "video-estimated onset".
→ The data are presented as the first contribution, so these basic numbers belong in the main text. A blocked design without session-level control would also weaken the decoding evidence in Fig. 4.
Q2 [C] Data release, scaling and ethics. (Weakness 1)
- Will the synchronized EEG–video data be released, and under what access conditions?
- Do you plan to record more participants and to test on a held-out participant?
- Did an ethics board approve the study? The Ethics Statement (p.10) mentions informed consent, de-identification and minimal risk, but no ethics-board approval.
→ A firm commitment to release and scale up the data would partly address my main concern.
Q3 [C + A] Real-robot evidence. (Weakness 2)
- Do you have any real-robot results?
- If not, which parts of your claim do you expect to carry over to a physical robot, and why? The human data are recorded in the real world, but all robot tests are in simulation.
→ Even a small real-robot study would make the claim about embodied learning much more convincing. Without any real-robot experiments, the evidence stays at the level of simulated benchmarks, which is not enough for the paper’s claims about embodied intelligence in the physical world (Fig. 1).
Q4 [C + A] Number of runs, selection and statistics.
- How many runs underlie each table entry? For example, 54.30 and 53.50 (Table 5) and 90.17 (Table 2) cannot be means of three 500-episode runs.
- Were the default loss weights chosen on the LIBERO-Long evaluation rollouts (App. B.3)? Was the robot-only baseline tuned at all?
- Please report paired per-initial-state tests (McNemar or paired bootstrap) for all comparisons in Tables 1–3, ideally with ≥5 seeds and with hyperparameters chosen on held-out initial states.
→ Significant paired gains on settings not used for tuning would strengthen the paper. Continued non-significance for the small gains (8 of the 11 comparisons in Tables 1–2) would confirm my reading that most gains are within run-to-run variation.
Q5 [A + C] Starting point and training setup.
On LIBERO-Spatial and Object, both arms score below what the OpenVLA paper reports for fine-tuned OpenVLA (Spatial: 79.53 / 80.93 vs. 84.7; Object: 66.93 / 68.53 vs. 88.4; OpenVLA paper, Table 12). That paper also evaluates 500 trials per suite on LIBERO's original initial states.
- What do the released suite-specific OpenVLA checkpoints score under your evaluation pipeline with zero additional training steps?
- Which robot-image preprocessing was used in training? The supplied
prepare_waveform.pycenter-crops robot images to 80% of their area (l.183–196), whiletrain.pyrejects such a cache (l.39–40). A train–test crop mismatch could explain the lower scores. - Which checkpoint does each arm use?
train.pyrecords "best full EEG validation MSE; R1 final step" (l.392), while the README merges step 2,000.
→ If the released checkpoints match or beat EEG-guided training on most suites, the gain may only reflect reduced damage from continued fine-tuning.
Q6 [A] Is the gain specific to aligned EEG?
On LIBERO-Long (and ideally one more setting), how does aligned EEG compare with each of the following?
- EEG shifted in time within the same episode;
- episode-mean EEG;
- InfoNCE with same-episode windows allowed as negatives (the supplied code excludes them,
sequence_contrast.pyl.20); - a non-neural auxiliary target on the same frames (e.g., random targets with matched statistics, or task and phase labels).
Also, on held-out episodes, does the EEG head predict EEG better than a per-task-mean baseline? The README notes that the current validation split uses random windows that may share episodes.
→ If aligned EEG clearly beats all of these, the "human neural dynamics" interpretation would be much better supported. If episode-mean EEG or a non-neural target matches it, the gain is more likely a generic auxiliary-task effect.
Q7 [C] Analyses in Figs. 5–6.
- Which rollouts and which pairing underlie Fig. 5A? Why do the per-task nets sum to +65 rather than the +49 implied by Table 3 (851 vs. 802 successes), and what does n = 577 count?
- How is "hidden-state change (%)" in Fig. 6 computed, what are the n = 6 samples, and how does it compare with a null comparison (e.g., two robot-only seeds)?
→ I need this before giving weight to the behavior- and representation-level claims.
Q8 [C] π0 and RL4VLA protocols.
The main text twice points to the appendix for backbone-specific details (p.5), but the appendix covers only OpenVLA on LIBERO. Please give the initialization, training data, number of steps and per-seed results for both. For π0, also give the representation S(·) and the LoRA placement; for RL4VLA, the sub-settings and the reason for the 72.99 ± 0.00 on Semantic OOD.
→ Without these, I cannot assess the cross-backbone and OOD claims.
Suggestions for future work (beyond the rebuttal; not affecting my rating).
Record more participants and evaluate on held-out participants. Test whether human data from tasks that match the robot tasks help more than data from unrelated tasks. Validate the approach on a real robot.
10 comments
Response to Reviewer yhW8 (2/7)
Q1 [C] How much data is used, and how was it collected?
How much data is used for training in total (episodes, hours and windows), and how much comes from each participant? Please report this in the main text.
We summarize the total and per-participant data (episodes, hours, and EEG windows), along with the training/validation split, in the table below. These details are now clarified in Section 3.1 and included in a new table in the Appendix.
| Subset / Split | Episodes | 1-s Windows | Duration (Hours) |
|---|---|---|---|
| Participant 1 | 749 | 3,538 | ~0.98 h |
| Participant 2 | 475 | 2,753 | ~0.76 h |
| Total Dataset | 1,224 | 6,291 | ~1.75 h |
| **VLA Training ** | — | 5,032 | ~1.40 h |
| **VLA Validation ** | — | 1,259 | ~0.35 h |
How do the two source datasets (App. A.1; "v1" and "ego128" in the supplied code) relate to the two participants, their sessions and their recording days? App. A.1 states that each participant completed all 12 tasks, yet five of the 12 tasks have episodes only from "v1" (prepare_waveform.py, l.24–37).
In the supplied code, “v1” and “ego128” denote two recording batches collected from the same two participants on different days, containing 919 and 305 episodes, respectively. The five tasks appearing only in “v1” therefore reflect differences in task coverage between batches, rather than separate participant datasets.
Were tasks recorded in blocks or interleaved? How were EEG and video synchronized, and how accurately? Fig. 2(C) plots time since a "video-estimated onset".
Tasks were randomly interleaved during recording to reduce confounding between task identity and recording time. We have clarified this in the main text.
Video frames (30 Hz) are timestamped directly on the acquisition PC master clock (t_pc), while EEG (1000 Hz) is linearly mapped onto t_pc via ParallelBox hardware triggers: hardware trigger alignment achieves ~0.48 ms median residual, while end-to-end mapping to the shared PC reception clock has a median USB/software jitter residual of ~66 ms (~2 video frames).
In Fig. 2(C), "video-estimated onset" serves ** as the display origin for this illustrative plot**, not for dataset synchronization or training: to skip ~7 s of pre-movement rest and highlight active execution-phase EEG. Model training directly uses the clock-synchronized video–EEG windows without manual onset estimation.
The data are presented as the first contribution, so these basic numbers belong in the main text. A blocked design without session-level control would also weaken the decoding evidence in Fig. 4.
Thank you for this suggestion. Clarifying these dataset statistics and experimental details greatly improves clarity and prevents misunderstandings; we have updated both Section 3.1 and Appendix A.1 accordingly. Moreover, because tasks were randomly shuffled within each recording block during data collection, the decoding evaluation in Fig. 4 is free from block-level temporal proximity bias.
**Q2 [C] Data release, scaling and ethics. **
Will the synchronized EEG–video data be released, and under what access conditions?
Yes. At the end of December, we will publicly release an expanded embodied EEG–behavior dataset covering additional newly collected scenarios, which will include the full synchronized EEG–video dataset from this paper. The data will be openly accessible to everyone without restrictions, and we have updated our Reproducibility Statement accordingly.
Do you plan to record more participants and to test on a held-out participant?
Thank you for the suggestion. Since submission, we have continued data collection, recruited additional participants, and expanded to more settings, including desks and workbenches. We have accumulated over 10 hours of recordings, and collection is ongoing.
At this stage, we prioritized repeated, longitudinal recordings from two participants under stable conditions to control additional variability arising from individual differences and investigate whether brain-signal supervision can improve embodied model performance. Dense sampling of a small participant cohort also has precedents, such as the core language-decoding experiments in Tang et al. (Nature Neuroscience, 2023). Our current results provide preliminary evidence of effectiveness. With the expanded dataset, we also hope to assess the consistency of these benefits through held-out-participant evaluation.
Response to Reviewer yhW8 (3/7)
Did an ethics board approve the study? The Ethics Statement (p.10) mentions informed consent, de-identification and minimal risk, but no ethics-board approval.
Yes, the study has been approved by our institutional ethics board. The specific details were omitted during submission to preserve double-blind anonymity. We will include the full ethics approval details in the camera-ready Ethics Statement and alongside the official open-source dataset release.
yhW8-Q3 Real-robot evidence.
Do you have any real-robot results?
Yes. We added evaluations on a physical PIPER robot using the backbone for two tabletop manipulation tasks.
We first fine-tuned on 50 real-world teleoperation demonstrations. From this checkpoint, we performed further LoRA fine-tuning using robot data alone (Robot-only) or with EEG supervision (EEG-guided). Inference remains EEG-free.
| Real-world task | baseline SR (%) | Robot-only LoRA SR (%) | EEG-guided LoRA SR (%) |
|---|---|---|---|
| Place a plate onto a dish-drying rack | 66.7 | 80.0 | 93.3 |
| Place a pen into a pen holder | 40.0 | 53.3 | 73.3 |
EEG-guided training improves success rates over Robot-only by 13.3 and 20.0 percentage points, respectively, providing evidence of benefits beyond simulation.
We have added Figure 7 to the main text and described the experimental setup and implementation details in the appendix.
yhW8-Q4 Number of runs, selection and statistics.
How many runs underlie each table entry? For example, 54.30 and 53.50 (Table 5) and 90.17 (Table 2) cannot be means of three 500-episode runs.
The submitted LIBERO results were intended to summarize three training seeds, with 500 evaluation episodes per suite per seed. We rechecked the original summaries against the raw episode-level JSON logs and identified the following reporting errors:
| Entry | Originally reported | Cause of reporting error | Original three-seed raw counts | Corrected original three-seed summary (mean ± sample SD) |
|---|---|---|---|---|
| Table 5 BrainTokenize | 54.30 ± 1.40 | Accidentally pooled six seeds rather than three, including three additional runs: 268/500, 273/500, and 270/500 | 284/500 (56.80%), 263/500 (52.60%), 271/500 (54.20%) | 54.53 ± 2.12 |
| Table 5 LaBraM | 53.50 ± 1.00 | Rounded 53.4667 ± 1.0263 to one decimal place before zero-padding | 263/500 (52.60%), 266/500 (53.20%), 273/500 (54.60%) | 53.47 ± 1.03 |
| Table 2 Object (Robot-only) | 90.17 ± 0.37 | Used a stale summary containing an incomplete run (409/456), together with 90.60% and 90.20%, and population SD | 453/500 (90.60%), 451/500 (90.20%), 437/500 (87.40%) | 89.40 ± 1.74 |
These corrections document the original three-seed results. Separately, we have expanded the main comparisons in Tables 1 and 2 to six training seeds; their revised entries report the corresponding six-seed means and sample SDs, superseding the original three-seed summaries.
We will explicitly state the number of seeds and evaluation episodes for each experiment and consistently define “±” as the sample SD across training seeds.
Were the default loss weights chosen on the LIBERO-Long evaluation rollouts (App. B.3)? Was the robot-only baseline tuned at all?
We would like to clarify how the default loss weights were determined and how the Robot-only baseline was configured.
1. Loss weights. The default weights were not selected using LIBERO-Long evaluation results. They were fixed beforehand based on the batch ratio and loss scales of the robot and EEG branches, and kept unchanged across LIBERO and RL4VLA, and OpenVLA and . The ablations in Appendix B.3 were conducted afterward to examine sensitivity to these weights.
2. Robot-only baseline. We did not perform a separate hyperparameter search for Robot-only. Both Robot-only and EEG-guided were initialized from the same official suite-specific fine-tuned checkpoints and used identical robot-policy training settings. Robot-only omitted the EEG branch.
Response to Reviewer yhW8 (1/7)
We appreciate the reviewer’s recognition of the novelty of our research question and the potential value of the synchronized EEG–egocentric video data. To our knowledge, this is the first study to investigate whether real human EEG can improve VLA manipulation performance through training-time supervision. In response to your comments, we have made five main additions and revisions to strengthen the evidence and clarify the reporting:
yhW8 -W1: The data are too small to be a core contribution, and the paper does not say clearly how much it uses.
R- yhW8-W1: Thank you for the suggestion. Our dataset was limited at submission. Since then, we have recruited additional participants and expanded collection to desks, workbenches, and dining tables, accumulating over 10 hours of recordings, including the data used in this paper. We plan to release the data publicly in December 2026 and will state this plan in the manuscript. As stated in Contribution 1, our primary contribution is a paradigm integrating synchronized EEG recording with embodied experiments, rather than dataset scale. Our initial EEG–egocentric behavior recordings provide preliminary evidence that EEG auxiliary supervision can benefit embodied models; a larger dataset will be the focus of our subsequent release.
We initially prioritized repeated, longitudinal recordings from two participants under stable conditions to investigate whether brain signals can improve embodied model performance. This approach is consistent with prior brain-tuning research: for example, Moussa et al. (ICLR 2025) fine-tuned speech models separately using individual participants’ fMRI recordings and improved downstream semantic task performance. Our design aims to reduce additional variability while exploring useful neural information. The revised manuscript will specify the training data volume, each participant’s contribution, and the relationship between the illustrated subset and the data actually used, while acknowledging the limitations of evidence from only two participants.
yhW8 -W2: All robot results are in simulation.
R - yhW8-W2: Thank you for the suggestion. We agree that real-world robot experiments are important for assessing effectiveness in physical environments. We therefore added two tasks: placing a plate onto a dish-drying rack and placing a pen into a pen holder.
We first fine-tuned π0 on 50 real-world teleoperation demonstrations to obtain the baseline checkpoint. Starting from this checkpoint, we then performed LoRA fine-tuning using robot data alone (Robot-only) or with auxiliary EEG supervision (EEG-guided). The success rates are:
EEG-guided training outperforms both the baseline and Robot-only on both tasks, providing additional evidence of effectiveness beyond simulation. We have added Figure 7 to the main text and provided implementation details in the appendix.
We also increased the number of random seeds in the simulation experiments and conducted statistical significance tests. Most comparisons show statistically significant improvements. We report small or nonsignificant gains and discuss EEG signal characteristics as possible contributing factors. The revised manuscript clarifies the scope and limitations of these benefits and avoids directly extrapolating simulation results to real-world performance.