acceptodds
Under review as a conference paper at ICLR 2027

EEG2MOTION: Synthesizing Diverse Human Motions from EEG Signals during Motion Observation

Abstract

Human motion is governed by a hierarchical motor system where the brain provides high-level intentions and lower-level structures coordinate detailed dynamics. Existing brain-computer interfaces (BCIs) typically oversimplify this into constrained classification or low-dimensional control, failing to capture the richness of natural movement. Bridging this gap to achieve diverse, full-body motion synthesis remains challenging due to the substantial cross-modal divergence between sparse neural signals and high-dimensional kinematics, as well as the lack of large-scale paired EEG-motion datasets. To address this, we introduce EEG2MOTION, the first EEG-motion-text dataset for human motion synthesis from motion-observation EEG, comprising nearly 20,000 paired samples across thousands of motions. Using this dataset, we first demonstrate via multimodal contrastive learning that non-invasive EEG embeddings can be aligned with text, video, and motion representations to decode high-level semantics. We then propose EEG-conditioned Masked Motion Model (EMMM), a generative framework that unites an EEG encoder with a motion decoder to synthesize continuous, full-body human motions directly from brain activity. Experimental results show that EMMM generates coherent and realistic motion sequences from non-invasive brain signals. To the best of our knowledge, this is the first work to generate diverse full-body human motions from non-invasive brain signals, opening a new direction toward generative and open-vocabulary motor BCIs. See our anonymous project page: https://eeg2motion-demo-2026.github.io/EEG2MOTIONdemo_page/.

Primary area:
generative models
Keywords:
EEG, Human Motion Generation, BCI

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_5G5SOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces EEG2MOTION dataset constructed by associating EEG signals to motion-text pairs from HumanML3D. EEG signals are recorded while participants observe rendered human motion videos (Action Observation). EEG-conditioned Masked Motion Model (EMMM) is also proposed for synthesizing human motions from EEG signals. The alignment between EEG and motions and synthesis are evaluated by retrieval accuracy and motion quality.

Critical strengths
  • The paper evaluates EEG–motion correspondence through both retrieval and motion synthesis.
  • Statistical testing, channel ablation, and qualitative comparisons with ground-truth motions are provided.
  • Multiple alignment modalities, including text, video, and motion, are considered.
Critical weaknesses
  • The retrieval improvement over chance is too small to support the claim that motion-specific information is decoded. The result may reflect only coarse concepts such as forward motion or left/right turns.
  • The weak retrieval performance may indicate either a limitation of the method or insufficient motion-detail information in the recorded EEG. The current experiments do not distinguish these possibilities.
  • Three participants with failed alignment were excluded without a predefined exclusion criterion, potentially biasing the reported retrieval and generation performance.
  • Channel-ablation settings differ across alignment modalities. Some relevant conditions are omitted from the motion-synthesis evaluation, such as eeg-text alignment, and the selection criteria are unclear.
  • The showcased motions appear to recover only coarse actions and movement directions. They do not demonstrate reconstruction beyond information expressible in language.
Questions for authors

I am ready to raise the rating based on discussions.

  • Can the method retrieve the correct motion among candidates sharing the same coarse action category?
  • Does EEG provide motion information beyond text-level semantics, such as detailed trajectories, timing, speed, or coordination?
  • How can the authors distinguish limitations of the method from insufficient motion information in the measured EEG?
  • Were the exclusion criteria for the three participants defined before the analysis? How do the results change when all participants are included?
  • Why do the channel-ablation conditions differ across alignment modalities?
  • Why were EEG-video with visual-only channels and EEG-text conditions excluded from the motion-synthesis evaluation?
  • Can the authors show quantitatively that EEG-conditioned generation reconstructs motion details beyond those recoverable from text alone?
4 comments

Because the revision is extensive, we first summarize the main changes.

First, we collected two new datasets, AO-16 and MI-Open; the original dataset is now called AO-Open. AO-16 contains EEG recordings from 7 participants observing 16 action classes for a total of 20,000 samples. AO-Open contains EEG recordings from 9 participants observing motions from an open-vocabulary motion space for a total of 28,000 samples. MI-Open contains EEG recordings from 7 participants performing open-vocabulary motor imagery. Participants watch a 2.5-second motion and then imagine repeating it for 2.5 seconds, yielding 5,250 samples. Because our study now goes beyond the original action-observation paradigm, we changed the title to “EEG2MOTION: Towards Open-Vocabulary Human Motion Synthesis from Non-invasive Brain Signals.”

Second, while retaining the original main method, we changed the EEG preprocessing procedure, mainly by checking for bad channels, repairing them through interpolation, and preserving the original 1,000-Hz sampling rate. With these changes, nearly every result improved over the previous version. The best Top-3 recall increased from 14.4% to 15.8% (Table 5).

Third, we added more detailed results, including AO-16 classification performance under different SNR levels and with different models (Table 2 and Appendix Tables 24 and 25); contrastive-learning and motion-synthesis results for MI-Open (Tables 4 and 6); held-out-subject cross-subject results for AO-Open and MI-Open, as well as results without participant screening (Appendix Tables 13–16); a 16-class confusion matrix; and coarse motion-cluster classification results (Appendix Tables 27 and 28). The supplementary material also includes a highlighted version showing the revisions. We additionally report the mean correlation across all joints and MAE as more direct motion-evaluation metrics (Appendix Tables 21–23). To make these materials easier to review, we incorporated the former supplementary material into the paper PDF as appendices.

Response to Reviewer 5G5S

Question 1: Can the method retrieve the correct motion among candidates sharing the same coarse action category?

We added the AO-16 dataset, which contains 16 distinct actions. The results are reported in Table 2, and Appendix Table 28 provides the confusion matrix. It shows that confusions are more likely within the hand-action categories and within the foot-action categories. We also clustered the motion spaces of AO-Open and MI-Open before performing classification. The results in Appendix Table 27 are only slightly above chance.

Question 2: Does EEG provide motion information beyond text-level semantics, such as detailed trajectories, timing, speed, or coordination?

We introduced two additional metrics: mean MAE across all joints and mean correlation across all joints (Appendix Tables 21–23). The correlation is much better than the random baseline.

Question 3: How can the authors distinguish limitations of the method from insufficient motion information in the measured EEG?

Table 2 measures how different levels of EEG SNR affect the action-observation results. We believe that EEG SNR and the large differences between participants remain the main factors limiting performance, rather than a major problem with the method framework.

Question 4: Were the exclusion criteria for the three participants defined before the analysis? How do the results change when all participants are included?

Appendix Table 12 reports the individual-subject contrastive-learning results. We selected participants solely according to whether their results were significantly above chance (p < 0.05). Results without any participant screening are also reported in Appendix Tables 13–20.

Question 5: Why do the channel-ablation conditions differ across alignment modalities?

Tables 3 and 4 show that visual channels yield better recall for AO, while sensorimotor channels yield better recall for MI. This is consistent with the brain regions activated by each task. However, video features are easier to retrieve for most alignment modalities. This does not necessarily mean that the aligned information is visual rather than motor-related, because the features depend on the modality encoder used.

Question 6: Why were EEG-video with visual-only channels and EEG-text conditions excluded from the motion-synthesis evaluation?

We have supplemented Table 5. Since EEG-text performed far worse than motion and video, we did not further evaluate channel ablations for EEG-text.

Question 7: Can the authors show quantitatively that EEG-conditioned generation reconstructs motion details beyond those recoverable from text alone?

Given the extremely low SNR of EEG, we cannot claim that it provides information superior to text. Nevertheless, directly generating motion from brain signals remains a valuable direction for brain-computer interfaces.

Thank you for the extensive revision and the additional datasets.

My main concern remains whether the recorded EEG contains motion information beyond what can be expressed by coarse action categories. The current results, including the within-group confusions in AO-16 and the near-chance motion-cluster classification, seem consistent with the EEG capturing only such coarse distinctions. If so, this conflicts with the "open-vocabulary" framing, which implies decoding motions beyond a small set of coarse categories.

Do the authors have any evidence, from the current experiments or additional analyses, that the EEG carries motion information beyond coarse action categories?

Without such evidence, I am currently unable to raise my rating.

Thank you for clarifying your concern. We have added two analyses in Appendix E to test whether EEG contains information beyond coarse action labels. Both use AO-Open participant S1, with four EEG trials averaged per motion to improve SNR.

First, we tested whether EEG can distinguish different motions within the same coarse category. For each EEG sample, the task was to retrieve its corresponding motion from five candidates. We restricted all five candidates to the same motion cluster, the same action family, or the same action family and intensity group. Top-1 accuracy was 30.07%, 29.75%, and 30.51%, respectively, compared with 20% chance (all p < 0.001; Table 30). These results therefore provide evidence of finer motion information within coarse categories.

Second, we tested whether the EEG feature space contains semantic components that can be reused across actions. We first examined relative motion intensity. We defined a high-versus-low intensity direction using reference motion features from other action families, then used that direction to rank EEG samples from a held-out action family. This achieved 64.53% AUC, compared with 50% chance (p = 0.0192; Table 31), supporting a shared intensity direction across actions.

We then tested a more concrete example: running/jumping × forward/in place. The test contains source motions covering all four combinations: running forward, running in place, jumping forward, and jumping in place. Using reference motion features, we defined the direction from running in place to running forward, then applied this same direction to EEG features to distinguish jumping forward from jumping in place. We also tested the reverse transfer. Thus, the displacement distinction was defined using one action and tested on EEG from another.

The average cross-action AUC was 85.42% (p = 0.0069), close to the 84.38% AUC obtained when displacement was identified with the action given (Table 31 and accompanying text). The similar AUC values suggest that the forward-versus-in-place component can transfer across actions. In other words, the representation supports separating the base action (running or jumping) from a reusable displacement component (forward or in place).

These findings provide two concrete reasons for the open-vocabulary framing: EEG features distinguish motions within a coarse category, and semantic components can transfer between actions. A fixed action label alone does not express these within-category differences or reusable components. We hope this additional evidence addresses your main concern.

Thank you for the additional analyses in Appendix E. My concern about the "open-vocabulary" framing comes from the synthesized motions in the supplementary material, which appear to me monotonous and consist mostly of coarse locomotion. I do not think either new analysis shows that the EEG carries information beyond coarse motion categories.

  1. Within-category retrieval among five candidates. Top-1 accuracy of about 30%, against 20% chance, shows that the EEG can weakly separate five candidates. How strong this evidence is depends on how the categories were defined. Could the authors clarify the following?
  • How were the motion clusters, action families, and intensity groups defined (e.g., from text annotations or from motion features), and how many groups and motions does each contain?
  • Within a group, do the candidates still differ in coarse, language-expressible attributes such as global displacement or turning direction?

If the groups are broad, the result is also consistent with the EEG capturing another coarse category that the grouping does not cover, and it does not by itself establish finer motion information. Even under fine-grained grouping, such a weak five-way discrimination is still far from what "open-vocabulary" implies. In addition, the result comes from a single participant (S1) with four-trial averaging.

  1. Running/jumping × forward/in place transfer. Forward versus in place is itself a coarse binary category defined by root translation, and it is one of the most salient visual differences in a rendered video. The transfer of this attribute between two locomotion actions is consistent with the EEG encoding one more coarse category, rather than motion information beyond such categories. In addition, the transfer direction is defined in the motion feature space. The separation of base action and displacement may therefore be a property of the motion encoder, and the EEG may only need to separate forward from in place.

In my view, both analyses therefore remain at the level of coarse motion categories, and the synthesized motions appear to show the same limitation. The current evidence does not yet sufficiently support the "open-vocabulary" framing, and I suggest weakening it.

Reviewer_RP23OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a representation alignment approach to enable human motion synthesis from EEG signals. The problem is motivated by the central task of Brain-Computer Interfaces (BCIs): decoding high-level brain data into low-level implementations of motions. To do this the authors use contrastive learning to align the learned representations of EEG with those created by other multimodal models, namely CLIP (image-text), VideoMAE (video-text), and Motion Encoder (motion-text) to condition the EEG representations to encode the high-level motion descriptions. The authors expand the HumanML3D dataset by recording EEG while motion samples are shown to participants in an Action Observation paradigm; evaluate the alignment of different modalities through retrieval accuracy; and train a Motion decoder conditioned on these EEG representations to synthesize the observed motions directly from participant's brain data.

Critical strengths
  • Large scale dataset with synchronized EEG and observed motion data is the first of its kinds and relevant to this type of multimodal representation alignment work.
  • Well described experimental setup, supporting reproducibility.
  • Overall well motivated problem statement and experiment design.
Critical weaknesses
  • Connection between motivation (BCIs with open vocabulary) and what is actually developed (Motion synthesis from Action Observation EEG data) is not direct, but gap is not fully discussed.
  • In section 6.1, the discussion on differences between the contributions of visual and motor channels could be improved by relating it to previous work, like the already cited (Hardwick et al. 2018).
  • The results and discussion sections need to be reevaluated taking into account the absolute performance values and effect sizes observed. While it is true that the results obtained are statistically significant, they correspond to accuracy improvements of less than 0.03, with a baseline of having the correct answer in the top 3 less than a tenth of the time.
Questions for authors
  • Subject exclusion based on unfavorable performance (section 3.2.3) needs further justification: is this lack of of contrastive alignment a limitation of the alignment approach? a consequence of some behavioral deviation from those participants during the data collection? or ultimately a limitation of the proposed approach in general? meaning that for some subjects this just won't work?
  • In the discussion a substantial performance gap is mentioned between the paper's results and other text-to-motion models, but the gap is not explicitly measured. This gap is explained in the paper as resulting from low signal-to-noise ration of EEG, but no experiments or further justification is given as to why authors make this attribution. Could you please elaborate on this?
1 comment

General response to all reviewers

We thank all reviewers for their thoughtful comments. Our work, EEG2MOTION, has undergone major improvements in the new version. Because the revision is extensive, we first summarize the main changes.

First, we collected two new datasets, AO-16 and MI-Open; the original dataset is now called AO-Open. Together, these three datasets form EEG2MOTION. AO-16 contains EEG recordings from 7 participants observing 16 action classes. Each observation sample lasts 0.6 seconds, for a total of 20,000 samples. AO-Open contains EEG recordings from 9 participants observing motions from an open-vocabulary motion space. Each sample lasts 1 second, for a total of 28,000 samples. MI-Open contains EEG recordings from 7 participants performing open-vocabulary motor imagery. Participants watch a 2.5-second motion and then recall and imagine repeating it for 2.5 seconds, yielding 5,250 samples. AO-16 supports the study of classification among a fixed set of action classes, while MI-Open provides a baseline for motion synthesis from motor imagery. Because our study now goes beyond the original action-observation paradigm, we changed the title to “EEG2MOTION: Towards Open-Vocabulary Human Motion Synthesis from Non-invasive Brain Signals.”

Second, while retaining the original main method, we changed the EEG preprocessing procedure, mainly by checking for bad channels, repairing them through interpolation, and preserving the original 1,000-Hz sampling rate. With these changes, nearly every result improved over the previous version. The best Top-3 recall increased from 14.4% to 15.8% (Table 5). Averaging every four test trials corresponding to the same action to improve the signal-to-noise ratio, a common strategy in EEG-based image synthesis, raises the best AO-Open Top-3 recall to 18.4% (Appendix Table 22), nearly twice the random level of 9.375%. Table 2 provides more details on the SNR changes from averaging.

Third, we added more detailed results, including AO-16 classification performance under different SNR levels and with different models (Table 2 and Appendix Tables 24 and 25); contrastive-learning and motion-synthesis results for MI-Open (Tables 4 and 6); held-out-subject cross-subject results for AO-Open and MI-Open, as well as results without participant screening (Appendix Tables 13–16); a 16-class confusion matrix; and coarse motion-cluster classification results (Appendix Tables 27 and 28). The supplementary material also includes a highlighted version showing the revisions. We additionally report the mean correlation across all joints and MAE as more direct motion-evaluation metrics (Appendix Tables 21–23). To make these materials easier to review, we incorporated the former supplementary material into the paper PDF as appendices.

We are grateful for the reviewers’ suggestions, which helped us improve the work. We respectfully ask you to reconsider your assessments and scores in light of the revised manuscript.

Response to Reviewer RP23

Question 1: Subject exclusion based on unfavorable performance (Section 3.2.3) needs further justification: is this lack of contrastive alignment a limitation of the alignment approach, a consequence of behavioral deviations by some participants during data collection, or a broader limitation of the proposed approach, such that it may not work for some subjects?

Appendix Table 12 reports the individual-subject contrastive-learning results. We selected participants solely according to whether their results were significantly above chance (p < 0.05). Results without any participant screening are also reported in Appendix Tables 13–20. In addition, we did not screen any participants from AO-16, and Table 2 reports the results for all seven participants. These results show extremely large differences between participants, and averaging trials from the same action to improve SNR does not reduce these differences. Appendix Table 12 also reports the motor-imagery results: only 3 of the 7 participants perform above chance, and participant-specific differences are even larger than in action observation. Therefore, for a challenging task such as open-vocabulary retrieval, we consider participant screening necessary at the current stage.

Question 2: The Discussion mentions a substantial performance gap between this paper’s results and other text-to-motion models, but the gap is not explicitly measured. The paper attributes this gap to the low signal-to-noise ratio of EEG, without providing experiments or further justification. Could the authors elaborate?

Table 2 specifically measures how improvements in EEG SNR affect performance on the 16-class action-observation task. The results show that increasing SNR substantially improves performance. For open-vocabulary action observation, Appendix Table 22 also reports results after averaging four test samples corresponding to the same test action to improve SNR. This raises the best Top-3 to 18.4%.

Reviewer_k1C6OpenReview ↗
Rating
3
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EEG2MOTION, a benchmark pairing EEG recorded during action observation with human motion sequences and textual descriptions. It evaluates cross-modal alignment between EEG and text, video, and motion representations, and proposes an EEG-conditioned masked motion model for full-body motion synthesis. The framework combines a trainable EEG encoder with a pretrained motion generator adapted through LoRA.

Critical strengths
  1. The paper introduces an interesting benchmark connecting non-invasive EEG with full-body motion and textual descriptions. This provides a useful foundation for studying neural representations of observed actions and EEG-conditioned motion generation.

  2. The research direction is promising. Moving beyond a small number of EEG classification categories toward diverse motion synthesis could motivate further work on richer neural conditioning and future BCI applications.

  3. The experiments include held-out motions, shuffled-pair and Gaussian-noise controls, channel ablations, and comparisons of training strategies. These analyses help distinguish information provided by EEG from the motion generator’s ability to produce plausible outputs.

  4. The authors acknowledge important limitations, including the dominant contribution of visual-region channels, the performance gap relative to text-conditioned generation, and the limited novelty of the generative architecture.

Critical weaknesses
  1. Participants observe motion videos but are not asked to imagine performing or controlling the movements. Therefore, the results primarily support decoding action-related visual information rather than voluntary motor intent. The strong performance of visual-region channels reinforces this distinction. A comparison between action observation and instructed motor imagery would be valuable for future research, and the current BCI-related claims should reflect this limitation.

  2. The evaluation focuses on correspondence in learned feature spaces and does not report conventional correlation measures such as Pearson’s r or Spearman’s ρ. Where temporal alignment is meaningful, these could be applied to joint trajectories, velocities, or interpretable motion descriptors to complement R-Precision and MM-Dist. If trajectory-level correspondence is outside the intended task, the authors should explain this and clarify what aspects of the target motion the model is expected to recover.

  3. The improvement attributable to EEG conditioning remains modest. The all-channel motion-aligned fine-tuning setting achieves 11.9% Top-3 retrieval, compared with 9.1% for shuffled pairing and a theoretical chance level of 9.375%. Per-subject results and clearer reporting of statistical tests and effect sizes would help establish the reliability of this improvement. Realistic motion generation should also be distinguished from faithful recovery of the observed action.

  4. Three of nine participants were excluded based on preliminary alignment performance. The authors should clarify which data were used for screening and whether validation or test outcomes influenced selection. Reporting results for all participants would improve transparency. Generalization to unseen participants also remains unestablished.

  5. The approach primarily adapts existing EEG and motion-generation architectures, so its strongest contribution is the benchmark and empirical investigation. Clear documentation of dataset access, preprocessing, participant splits, and screening decisions will be important for reproducibility and subsequent research.

Questions for authors
  1. Were any validation or test results used to decide which participants to exclude? Can results for all nine participants be provided?

  2. Can the authors report complementary temporal or kinematic correspondence measures, including Pearson’s r or Spearman’s ρ where appropriate, or explain why such measures would not fit the intended task?

  3. How consistent are the improvements over randomized controls across individual participants?

  4. What changes to the collection protocol would be needed to investigate motor imagery or voluntary movement intention, rather than action observation?

1 comment

General response to all reviewers

We thank all reviewers for their thoughtful comments. Our work, EEG2MOTION, has undergone major improvements in the new version. Because the revision is extensive, we first summarize the main changes.

First, we collected two new datasets, AO-16 and MI-Open; the original dataset is now called AO-Open. Together, these three datasets form EEG2MOTION. AO-16 contains EEG recordings from 7 participants observing 16 action classes. Each observation sample lasts 0.6 seconds, for a total of 20,000 samples. AO-Open contains EEG recordings from 9 participants observing motions from an open-vocabulary motion space. Each sample lasts 1 second, for a total of 28,000 samples. MI-Open contains EEG recordings from 7 participants performing open-vocabulary motor imagery. Participants watch a 2.5-second motion and then recall and imagine repeating it for 2.5 seconds, yielding 5,250 samples. AO-16 supports the study of classification among a fixed set of action classes, while MI-Open provides a baseline for motion synthesis from motor imagery. Because our study now goes beyond the original action-observation paradigm, we changed the title to “EEG2MOTION: Towards Open-Vocabulary Human Motion Synthesis from Non-invasive Brain Signals.”

Second, while retaining the original main method, we changed the EEG preprocessing procedure, mainly by checking for bad channels, repairing them through interpolation, and preserving the original 1,000-Hz sampling rate. With these changes, nearly every result improved over the previous version. The best Top-3 recall increased from 14.4% to 15.8% (Table 5). Averaging every four test trials corresponding to the same action to improve the signal-to-noise ratio, a common strategy in EEG-based image synthesis, raises the best AO-Open Top-3 recall to 18.4% (Appendix Table 22), nearly twice the random level of 9.375%. Table 2 provides more details on the SNR changes from averaging.

Third, we added more detailed results, including AO-16 classification performance under different SNR levels and with different models (Table 2 and Appendix Tables 24 and 25); contrastive-learning and motion-synthesis results for MI-Open (Tables 4 and 6); held-out-subject cross-subject results for AO-Open and MI-Open, as well as results without participant screening (Appendix Tables 13–16); a 16-class confusion matrix; and coarse motion-cluster classification results (Appendix Tables 27 and 28). The supplementary material also includes a highlighted version showing the revisions. We additionally report the mean correlation across all joints and MAE as more direct motion-evaluation metrics (Appendix Tables 21–23). To make these materials easier to review, we incorporated the former supplementary material into the paper PDF as appendices.

We are grateful for the reviewers’ suggestions, which helped us improve the work. We respectfully ask you to reconsider your assessments and scores in light of the revised manuscript.

Response to Reviewer k1C6

Question 1: Were any validation or test results used to decide which participants to exclude? Can results for all nine participants be provided?

Appendix Table 12 reports the individual-subject contrastive-learning results. We selected participants solely according to whether their results were significantly above chance (p < 0.05). Results without any participant screening are also reported in Appendix Tables 13–20.

Question 2: Can the authors report complementary temporal or kinematic correspondence measures, including Pearson’s r or Spearman’s ρ where appropriate, or explain why such measures would not fit the intended task?

We added two more direct metrics: mean MAE across all joints and mean correlation across all joints, reported in Appendix Tables 21–23. The correlation is much better than the random baseline, while the MAE does not show a clear difference.

Question 3: How consistent are the improvements over randomized controls across individual participants?

As in our response to Question 1, Appendix Tables 13–20 provide detailed results for each participant, as well as comparisons between single-subject and multi-subject models.

Question 4: What changes to the collection protocol would be needed to investigate motor imagery or voluntary movement intention, rather than action observation?

We added the MI-Open dataset to test motor imagery in an open-vocabulary motion space. The results are unstable: only 3 of the 7 participants perform significantly above chance. This reflects the challenges of generative motor BCIs and indicates that the paradigm needs further improvement. We consider incorporating appropriate feedback during data collection and more strictly controlling signal quality to be two important priorities.

Reviewer_QyQKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EEG2MOTION, a dataset containing approximately 19,000 EEG–motion–text pairs, and proposes EMMM, a framework for generating full-body human motions from non-invasive EEG signals. The method combines cross-modal contrastive learning with a pretrained masked motion model to align EEG representations with motion semantics and synthesize motion sequences. Experiments demonstrate that the proposed approach outperforms randomized baselines and generates realistic, diverse, and semantically aligned human motions.

Critical strengths
  1. The paper introduces EEG2MOTION, a dataset containing approximately 19,000 EEG–motion–text pairs, enabling the exploration of full-body human motion generation from non-invasive EEG signals beyond traditional constrained classification tasks.
  2. The paper evaluates cross-modal alignment across text, video, and motion modalities and conduct ablation studies on EEG channels, alignment objectives, and training strategies. These experiments provide useful insights into the contributions of different components and demonstrate the effectiveness of EEG conditioning.
Critical weaknesses
  1. The proposed framework primarily integrates existing components, including EEGConformer, contrastive learning, and a pretrained Masked Motion Model. The paper lacks a novel architecture or learning mechanism specifically designed to address the challenges of EEG-conditioned motion generation.
  2. Although the proposed method outperforms randomized baselines, the improvement remains modest (12.3% vs. 9.2% in Top-3 motion retrieval). Moreover, the stronger performance of visual EEG channels raises concerns about whether the model genuinely captures motion-related neural information or primarily relies on visual representations of observed actions.
Questions for authors
  1. Given the relatively small improvement over randomized baselines, could the authors provide stronger evidence that the generated motions are meaningfully conditioned on EEG signals rather than primarily driven by the pretrained motion model? For example, how does changing the EEG input affect the generated motion?
  2. How well does the proposed framework generalize to unseen subjects? Since subject-specific batch normalization is used, it would be helpful to evaluate cross-subject generalization and clarify whether the model requires additional calibration for new users.
3 comments

General response to all reviewers

We thank all reviewers for their thoughtful comments. Our work, EEG2MOTION, has undergone major improvements in the new version. Because the revision is extensive, we first summarize the main changes.

First, we collected two new datasets, AO-16 and MI-Open; the original dataset is now called AO-Open. Together, these three datasets form EEG2MOTION. AO-16 contains EEG recordings from 7 participants observing 16 action classes. Each observation sample lasts 0.6 seconds, for a total of 20,000 samples. AO-Open contains EEG recordings from 9 participants observing motions from an open-vocabulary motion space. Each sample lasts 1 second, for a total of 28,000 samples. MI-Open contains EEG recordings from 7 participants performing open-vocabulary motor imagery. Participants watch a 2.5-second motion and then recall and imagine repeating it for 2.5 seconds, yielding 5,250 samples. AO-16 supports the study of classification among a fixed set of action classes, while MI-Open provides a baseline for motion synthesis from motor imagery. Because our study now goes beyond the original action-observation paradigm, we changed the title to “EEG2MOTION: Towards Open-Vocabulary Human Motion Synthesis from Non-invasive Brain Signals.”

Second, while retaining the original main method, we changed the EEG preprocessing procedure, mainly by checking for bad channels, repairing them through interpolation, and preserving the original 1,000-Hz sampling rate. With these changes, nearly every result improved over the previous version. The best Top-3 recall increased from 14.4% to 15.8% (Table 5). Averaging every four test trials corresponding to the same action to improve the signal-to-noise ratio, a common strategy in EEG-based image synthesis, raises the best AO-Open Top-3 recall to 18.4% (Appendix Table 22), nearly twice the random level of 9.375%. Table 2 provides more details on the SNR changes from averaging.

Third, we added more detailed results, including AO-16 classification performance under different SNR levels and with different models (Table 2 and Appendix Tables 24 and 25); contrastive-learning and motion-synthesis results for MI-Open (Tables 4 and 6); held-out-subject cross-subject results for AO-Open and MI-Open, as well as results without participant screening (Appendix Tables 13–16); a 16-class confusion matrix; and coarse motion-cluster classification results (Appendix Tables 27 and 28). The supplementary material also includes a highlighted version showing the revisions. We additionally report the mean correlation across all joints and MAE as more direct motion-evaluation metrics (Appendix Tables 21–23). To make these materials easier to review, we incorporated the former supplementary material into the paper PDF as appendices.

We are grateful for the reviewers’ suggestions, which helped us improve the work. We respectfully ask you to reconsider your assessments and scores in light of the revised manuscript.

Response to Reviewer QyQK

Question 1: Given the relatively small improvement over randomized baselines, could the authors provide stronger evidence that the generated motions are meaningfully conditioned on EEG signals rather than primarily driven by the pretrained motion model? For example, how does changing the EEG input affect the generated motion?

Thank you for this question. As noted above, after the improvements, the best Top-3 recall reaches 18.4%, nearly twice the random level, indicating that EEG provides useful information. In addition, Appendix Table 13 reports held-out-subject cross-subject results, which are also clearly above random and demonstrate that EEG information generalizes across subjects. Appendix Table 28 provides the 16-class confusion matrix. It shows that several hand-action classes are more likely to be confused with one another, as are several foot-action classes. We believe these results provide evidence that EEG offers meaningful conditioning information.

Question 2: How well does the proposed framework generalize to unseen subjects? Since subject-specific batch normalization is used, it would be helpful to evaluate cross-subject generalization and clarify whether the model requires additional calibration for new users.

Appendix Tables 13–16 report held-out-subject results. The target subject is used only to adapt BatchNorm, with no paired-action supervision used for training, and the results remain well above random. Appendix Tables 17–20 also compare models trained separately on each subject’s data with models trained jointly across subjects.

Thank you for the detailed responses and the substantial revisions to the manuscript. The addition of two datasets, held-out-subject evaluations, and further analyses provides a more comprehensive experimental evaluation. The new results also help clarify the role of EEG information and the framework's cross-subject generalization ability.

However, my main concern regarding the methodological novelty remains. The framework still primarily combines existing components, and the empirical improvements, while encouraging, do not fully establish a sufficiently distinctive methodological contribution. Overall, I appreciate the authors' efforts in addressing the reviewers' concerns, but I would maintain my original rating.

Thank you for recognizing the substantial revisions and the additional experimental evidence. We would like to emphasize two central contributions when assessing the novelty and significance of this work.

First, our contribution is revolutionary for motor BCIs. Beyond marginal improvements from simply changing network architectures, this is the first BCI work targeting full-body motion synthesis. The fundamental problem facing current motor BCIs is that outdated paradigms of limited-category classification or regression offer only a very restricted movement space, limiting their practical value. For future invasive clinical applications, NO ONE would be willing to have electrodes implanted merely to replace mouse control. Although our work is non-invasive, it represents a critical milestone for the field.

Second, the new results in Appendix E demonstrate a capability enabled by this open-vocabulary formulation: the EEG feature space contains semantic components that can be separated and reused across actions. We identify semantic directions that transfer across actions. A concrete example is running/jumping × forward/in place. Using reference motion features, we define the direction from running in place to running forward, then apply that same direction to EEG features to distinguish jumping forward from jumping in place. We also test the reverse transfer, from jumping to running. The average cross-action AUC is 85.42% (random = 50%), almost identical to the 84.38% AUC obtained when the action is given. The representation supports transferable semantic components and their combinations, extending what can be expressed by a fixed categorical output. This is a central advantage of our open-vocabulary approach. This distinguishes our approach from all previous motor BCIs.

Incremental algorithmic improvements offer only marginal value for motor BCIs today, and our work represents one of the most important contributions to the field at this stage. We respectfully ask you to reconsider the rating in light of its milestone significance for motor BCIs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.