- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces EEG2MOTION dataset constructed by associating EEG signals to motion-text pairs from HumanML3D. EEG signals are recorded while participants observe rendered human motion videos (Action Observation). EEG-conditioned Masked Motion Model (EMMM) is also proposed for synthesizing human motions from EEG signals. The alignment between EEG and motions and synthesis are evaluated by retrieval accuracy and motion quality.
Critical strengths
- The paper evaluates EEG–motion correspondence through both retrieval and motion synthesis.
- Statistical testing, channel ablation, and qualitative comparisons with ground-truth motions are provided.
- Multiple alignment modalities, including text, video, and motion, are considered.
Critical weaknesses
- The retrieval improvement over chance is too small to support the claim that motion-specific information is decoded. The result may reflect only coarse concepts such as forward motion or left/right turns.
- The weak retrieval performance may indicate either a limitation of the method or insufficient motion-detail information in the recorded EEG. The current experiments do not distinguish these possibilities.
- Three participants with failed alignment were excluded without a predefined exclusion criterion, potentially biasing the reported retrieval and generation performance.
- Channel-ablation settings differ across alignment modalities. Some relevant conditions are omitted from the motion-synthesis evaluation, such as eeg-text alignment, and the selection criteria are unclear.
- The showcased motions appear to recover only coarse actions and movement directions. They do not demonstrate reconstruction beyond information expressible in language.
Questions for authors
I am ready to raise the rating based on discussions.
- Can the method retrieve the correct motion among candidates sharing the same coarse action category?
- Does EEG provide motion information beyond text-level semantics, such as detailed trajectories, timing, speed, or coordination?
- How can the authors distinguish limitations of the method from insufficient motion information in the measured EEG?
- Were the exclusion criteria for the three participants defined before the analysis? How do the results change when all participants are included?
- Why do the channel-ablation conditions differ across alignment modalities?
- Why were EEG-video with visual-only channels and EEG-text conditions excluded from the motion-synthesis evaluation?
- Can the authors show quantitatively that EEG-conditioned generation reconstructs motion details beyond those recoverable from text alone?
4 comments
Thank you for the extensive revision and the additional datasets.
My main concern remains whether the recorded EEG contains motion information beyond what can be expressed by coarse action categories. The current results, including the within-group confusions in AO-16 and the near-chance motion-cluster classification, seem consistent with the EEG capturing only such coarse distinctions. If so, this conflicts with the "open-vocabulary" framing, which implies decoding motions beyond a small set of coarse categories.
Do the authors have any evidence, from the current experiments or additional analyses, that the EEG carries motion information beyond coarse action categories?
Without such evidence, I am currently unable to raise my rating.
Thank you for clarifying your concern. We have added two analyses in Appendix E to test whether EEG contains information beyond coarse action labels. Both use AO-Open participant S1, with four EEG trials averaged per motion to improve SNR.
First, we tested whether EEG can distinguish different motions within the same coarse category. For each EEG sample, the task was to retrieve its corresponding motion from five candidates. We restricted all five candidates to the same motion cluster, the same action family, or the same action family and intensity group. Top-1 accuracy was 30.07%, 29.75%, and 30.51%, respectively, compared with 20% chance (all p < 0.001; Table 30). These results therefore provide evidence of finer motion information within coarse categories.
Second, we tested whether the EEG feature space contains semantic components that can be reused across actions. We first examined relative motion intensity. We defined a high-versus-low intensity direction using reference motion features from other action families, then used that direction to rank EEG samples from a held-out action family. This achieved 64.53% AUC, compared with 50% chance (p = 0.0192; Table 31), supporting a shared intensity direction across actions.
We then tested a more concrete example: running/jumping × forward/in place. The test contains source motions covering all four combinations: running forward, running in place, jumping forward, and jumping in place. Using reference motion features, we defined the direction from running in place to running forward, then applied this same direction to EEG features to distinguish jumping forward from jumping in place. We also tested the reverse transfer. Thus, the displacement distinction was defined using one action and tested on EEG from another.
The average cross-action AUC was 85.42% (p = 0.0069), close to the 84.38% AUC obtained when displacement was identified with the action given (Table 31 and accompanying text). The similar AUC values suggest that the forward-versus-in-place component can transfer across actions. In other words, the representation supports separating the base action (running or jumping) from a reusable displacement component (forward or in place).
These findings provide two concrete reasons for the open-vocabulary framing: EEG features distinguish motions within a coarse category, and semantic components can transfer between actions. A fixed action label alone does not express these within-category differences or reusable components. We hope this additional evidence addresses your main concern.
Thank you for the additional analyses in Appendix E. My concern about the "open-vocabulary" framing comes from the synthesized motions in the supplementary material, which appear to me monotonous and consist mostly of coarse locomotion. I do not think either new analysis shows that the EEG carries information beyond coarse motion categories.
- Within-category retrieval among five candidates. Top-1 accuracy of about 30%, against 20% chance, shows that the EEG can weakly separate five candidates. How strong this evidence is depends on how the categories were defined. Could the authors clarify the following?
- How were the motion clusters, action families, and intensity groups defined (e.g., from text annotations or from motion features), and how many groups and motions does each contain?
- Within a group, do the candidates still differ in coarse, language-expressible attributes such as global displacement or turning direction?
If the groups are broad, the result is also consistent with the EEG capturing another coarse category that the grouping does not cover, and it does not by itself establish finer motion information. Even under fine-grained grouping, such a weak five-way discrimination is still far from what "open-vocabulary" implies. In addition, the result comes from a single participant (S1) with four-trial averaging.
- Running/jumping × forward/in place transfer. Forward versus in place is itself a coarse binary category defined by root translation, and it is one of the most salient visual differences in a rendered video. The transfer of this attribute between two locomotion actions is consistent with the EEG encoding one more coarse category, rather than motion information beyond such categories. In addition, the transfer direction is defined in the motion feature space. The separation of base action and displacement may therefore be a property of the motion encoder, and the EEG may only need to separate forward from in place.
In my view, both analyses therefore remain at the level of coarse motion categories, and the synthesized motions appear to show the same limitation. The current evidence does not yet sufficiently support the "open-vocabulary" framing, and I suggest weakening it.
Because the revision is extensive, we first summarize the main changes.
First, we collected two new datasets, AO-16 and MI-Open; the original dataset is now called AO-Open. AO-16 contains EEG recordings from 7 participants observing 16 action classes for a total of 20,000 samples. AO-Open contains EEG recordings from 9 participants observing motions from an open-vocabulary motion space for a total of 28,000 samples. MI-Open contains EEG recordings from 7 participants performing open-vocabulary motor imagery. Participants watch a 2.5-second motion and then imagine repeating it for 2.5 seconds, yielding 5,250 samples. Because our study now goes beyond the original action-observation paradigm, we changed the title to “EEG2MOTION: Towards Open-Vocabulary Human Motion Synthesis from Non-invasive Brain Signals.”
Second, while retaining the original main method, we changed the EEG preprocessing procedure, mainly by checking for bad channels, repairing them through interpolation, and preserving the original 1,000-Hz sampling rate. With these changes, nearly every result improved over the previous version. The best Top-3 recall increased from 14.4% to 15.8% (Table 5).
Third, we added more detailed results, including AO-16 classification performance under different SNR levels and with different models (Table 2 and Appendix Tables 24 and 25); contrastive-learning and motion-synthesis results for MI-Open (Tables 4 and 6); held-out-subject cross-subject results for AO-Open and MI-Open, as well as results without participant screening (Appendix Tables 13–16); a 16-class confusion matrix; and coarse motion-cluster classification results (Appendix Tables 27 and 28). The supplementary material also includes a highlighted version showing the revisions. We additionally report the mean correlation across all joints and MAE as more direct motion-evaluation metrics (Appendix Tables 21–23). To make these materials easier to review, we incorporated the former supplementary material into the paper PDF as appendices.
Response to Reviewer 5G5S
Question 1: Can the method retrieve the correct motion among candidates sharing the same coarse action category?
We added the AO-16 dataset, which contains 16 distinct actions. The results are reported in Table 2, and Appendix Table 28 provides the confusion matrix. It shows that confusions are more likely within the hand-action categories and within the foot-action categories. We also clustered the motion spaces of AO-Open and MI-Open before performing classification. The results in Appendix Table 27 are only slightly above chance.
Question 2: Does EEG provide motion information beyond text-level semantics, such as detailed trajectories, timing, speed, or coordination?
We introduced two additional metrics: mean MAE across all joints and mean correlation across all joints (Appendix Tables 21–23). The correlation is much better than the random baseline.
Question 3: How can the authors distinguish limitations of the method from insufficient motion information in the measured EEG?
Table 2 measures how different levels of EEG SNR affect the action-observation results. We believe that EEG SNR and the large differences between participants remain the main factors limiting performance, rather than a major problem with the method framework.
Question 4: Were the exclusion criteria for the three participants defined before the analysis? How do the results change when all participants are included?
Appendix Table 12 reports the individual-subject contrastive-learning results. We selected participants solely according to whether their results were significantly above chance (p < 0.05). Results without any participant screening are also reported in Appendix Tables 13–20.
Question 5: Why do the channel-ablation conditions differ across alignment modalities?
Tables 3 and 4 show that visual channels yield better recall for AO, while sensorimotor channels yield better recall for MI. This is consistent with the brain regions activated by each task. However, video features are easier to retrieve for most alignment modalities. This does not necessarily mean that the aligned information is visual rather than motor-related, because the features depend on the modality encoder used.
Question 6: Why were EEG-video with visual-only channels and EEG-text conditions excluded from the motion-synthesis evaluation?
We have supplemented Table 5. Since EEG-text performed far worse than motion and video, we did not further evaluate channel ablations for EEG-text.
Question 7: Can the authors show quantitatively that EEG-conditioned generation reconstructs motion details beyond those recoverable from text alone?
Given the extremely low SNR of EEG, we cannot claim that it provides information superior to text. Nevertheless, directly generating motion from brain signals remains a valuable direction for brain-computer interfaces.