acceptodds
Under review as a conference paper at ICLR 2027

NeuroAction: Aligning EEG to Discrete Motion Primitives for Continuous Motion Decoding

Abstract

EEG-to-motion decoding remains challenging because noisy scalp EEG is difficult to map directly to continuous kinematics. We present NeuroAction, a framework that aligns EEG representations to discrete motion primitives for continuous motion decoding across heterogeneous paradigms, including full-body free movement, lower-limb treadmill locomotion, and upper-limb freewill reaching and grasping. A VQ-based motion prior discretizes continuous motion into reusable primitives that serve as both alignment prototypes and intermediate prediction targets. Prototypical and supervised contrastive objectives align EEG embeddings to motion-derived prototypes and organize them according to primitive assignments. An E2M Transformer predicts motion-code sequences from the learned EEG representations, while a final geometric refinement stage improves continuous trajectory reconstruction through motion-level training. Under a last-session test protocol, NeuroAction achieves the best mean reconstruction performance among the compared EEG decoding baselines across all three datasets, with the clearest gains on full-body motion in both reconstruction accuracy and distribution-level quality. Multi-subject ablations further support the contributions of EEG-to-primitive alignment, token prediction, and refinement. Code will be released publicly upon acceptance.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG decoding, brain-computer interfaces, motion reconstruction, discrete motion primitives, representation learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_PjxJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a framework to decode continuous motion from EEG. It is seperated into two steps: 1. decode the discrete motion primities, 2. a geometric refinement stage to turn discrete sequences to continous motion trajectory. Experiments on three different datasets validate the effectiveness of it.

Critical strengths
  1. Discrete codebook decoding is insightful, I see it is the first method to achieve a positive R2R^2 on FBM dataset.
  2. Paper is written clearly, figure is easy to understand.
  3. Ablation is sufficient.
Critical weaknesses
  1. Why ablation study uses different subject combination? Table 2 and Table 3 is using 4 and 2 subjects? It is confusing.
  2. Figure 6 shows the data has already started clustering quite well, does this suggest that the codebook is actually learning stronger spatiotemporal correlations—such as those associated with sessions—rather than specific action patterns?
Questions for authors
  1. Use \citep{xx} in your latex rather than \cite{xx}, ICLR template is different.
  2. I see table 4, the codebook size is not big compared to LaBraM, so I want to see the visualization of all learned codebook unit, not t-SNE, but the decoding results, such as Figure 1's motion, since it is the prototype of a motion, can it be done?
4 comments
AuthorsOpenReview ↗

Response to Reviewer PjxJ(1/2)

We thank you for the thoughtful assessment and constructive feedback. We address the comments and questions below.

Q1. Why do Tables 2 and 3 use different subject subsets?

Tables 2 and 3 serve different experimental purposes, and their subject subsets were originally selected independently, resulting in different subject combinations and sample sizes.

Table 2 presents component ablations examining the roles of alignment, E2M, and staged training, using four subjects per dataset spanning different performance levels. Table 3 presents EEG input controls using temporal reversal, random shuffling, and zero input to examine the model's dependence on EEG information and its temporal structure; the original experiments included only two subjects per dataset. The former examines how individual model components contribute to decoding, whereas the latter examines whether the model effectively uses the input signals. For input controls, perturbation effects are easier to observe in recordings containing clearer and more stable movement-related information. EEG recording quality can vary across subjects because of acquisition conditions and artifacts. We therefore initially selected two higher-performing subjects whose recordings were considered more likely to retain stable decodable information.

To improve the representativeness of Table 3, we added two lower-performing subjects per dataset, expanding each subset to four subjects. The additional experiments retain the original data splits, preprocessing, and training settings, and the results are averaged together with those of the original subjects. The revised subsets are FBM S03/S04/S05/S10, FR S05/S07/S13/S15, and TM S02/S05/S07/S08.

The expanded results continue to support the original conclusions: standard EEG achieves better mean PCC, R², and RMSE than reversed, shuffled, or zero inputs across all three datasets. Mean PCC with standard input is 0.510, 0.674, and 0.517 on FBM, FR, and TM, respectively, falling to 0.010, 0.000, and 0.003 with zero input. Temporal shuffling reduces PCC to 0.083 on FBM and 0.004 on TM. FR retains a PCC of 0.511, but its R² decreases from 0.235 to −0.162 and RMSE increases from 0.234 to 0.331. Thus, including lower-performing subjects preserves the evidence that decoding depends on informative input and its temporal structure, while retaining the original finding that FR preserves more predictive information after temporal shuffling. The results have been updated in Table 3. Section 3.3 explains the experimental purposes, and the captions of Tables 2 and 3 identify their respective subject subsets.

Q2. Could the clustering in Figure 6 reflect session-related correlations rather than specific movement patterns?

Each panel in Figure 6 contains data from only one held-out test session of one subject, so the clusters cannot arise from mixing multiple sessions. The two panels correspond to FBM S10 and FR S05, respectively. We will further clarify this in the figure caption.

The figure shows EEG-token representations colored by the codebook labels of their paired motion segments, rather than the codebook vectors themselves. The original visualization selects EEG tokens associated with the four most frequently used motion codes. It therefore provides qualitative evidence of motion-primitive-related organization in EEG representations within a single test session.

Q3. Parenthetical citations following the ICLR template

Thank you for pointing this out. We have checked the manuscript against the ICLR template's citation conventions and standardized the relevant parenthetical citations using \citep{...}.

AuthorsOpenReview ↗

Response to Reviewer PjxJ(2/2)

Q4. Given the small codebook, can the decoded movements of individual codebook entries be visualized?

First, LaBraM and NeuroAction model different signals and cover different scopes, so their codebook sizes are not directly comparable. LaBraM [1] targets large-scale EEG representation learning across multiple datasets, and its codebook represents EEG. Our codebook instead learns motion primitives from each dataset’s trajectories. Given the relatively limited scope of our movement tasks and training data, the key question is whether codebook capacity suits the motion representation and its decodability from EEG, rather than whether a larger codebook is inherently better.

Table 4 and Appendix D.1 evaluate this choice through codebook-capacity ablations. Among the tested capacities, FBM, TM, and FR achieve their best mean performance with K = 32, 16, and 12, respectively. Further increasing codebook size reduces performance, supporting the use of smaller codebooks for the current tasks.

We have added decoded outputs of eight representative FBM codebook entries in Appendix D.4, “Decoded Motion-Code Visualization.” Each code is individually supplied to the motion decoder to generate a local sequence of four frames, with the second frame displayed as the representative pose.

These poses show the outputs obtained by decoding individual codes in isolation; they do not represent unique, fixed motion templates. During trajectory reconstruction, the motion decoder applies temporal convolutions and upsampling to the complete primitive sequence, so the output associated with a code also depends on neighboring primitives. Consequently, even when the central code remains unchanged, different preceding and following codes may produce different local trajectories.

References

[1] Jiang, W.-B., Zhao, L.-M., & Lu, B.-L. (2024). Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCI. International Conference on Learning Representations. https://arxiv.org/abs/2405.18765

Reviewer_PjxJOpenReview ↗

Reply

  1. I don't recognize the anwer to Q1, I understand that the collecting quality has large influence on the results, but it is not the reason that "we omit part of subjects when we doing ablation study". In contrast, you could list all subjects results (some in appendix) to show that generally your model is good. Overall you give more results, this is good.

  2. Q2 and Q3 has been set.

  3. I appreciate the new figure give in appendix D.4, but I am not asking why your codebook size is small. I fully know that LaBraM's codebook can not be visualized, but your codebook can be visualized, so I just wonder what's the visualization results. Just curious about the probablity that "codebook learns prototype of motion in EEG".

Overall, I decide to maintain my score.

AuthorsOpenReview ↗

Response to Reply

Thank you for your constructive feedback.

Regarding Q1, our main comparisons cover all included subjects, with individual results already reported in Appendix D.6 (Tables 12, 14, and 15). Given the training cost of numerous ablation configurations, we selected subsets spanning different performance levels and listed their subject IDs. Similar designs appear in fMRI and EEG research: MindEye2 uses ten sessions from Subject 1 for its Section 3.5 ablations [2], while CAST uses four subjects spanning high to very low EEG-to-iEEG reconstruction performance [3].

Thank you again for confirming that Q2 and Q3 have been resolved, and for your positive comments on the additional results and the motion-code visualization in Appendix D.4. We appreciate your time and constructive feedback.

References

[2] Scotti, P. S., et al. (2024). MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. Proceedings of the 41st International Conference on Machine Learning, PMLR 235, 44038–44059. https://proceedings.mlr.press/v235/scotti24a.html

[3] Pham, T.-D., & Tran, X.-T. (2026). Cross-Subject Intracranial EEG Reconstruction from Scalp Recordings Using Multi-Scale Cross-Attention Transformers. arXiv preprint arXiv:2605.18897. https://arxiv.org/abs/2605.18897

Reviewer_KBdsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a way to reconstruct humans' current movements from EEG. It first learns a set of reusable motion patterns from motion data, then trains an EEG model to predict sequences of these patterns and uses a decoder to turn them into continuous movement. A final training stage trains both the EEG model and the motion decoder to improve reconstruction. The method is evaluated on three movement datasets using dataset specfic models and held-out recording sessions, and achieves better average reconstruction scores than the compared baselines.

Critical strengths
  1. The model has a reasonable design; it first learns the motion pattern, then uses it to guide the EEG reconstruction.
  2. Under the paper's evaluation setup, the proposed model achieves improvement over the baseline models, with component ablation studies.
Critical weaknesses
  1. The problem formulation is mismatched: The paper motivates its work with EEG-based control of external effectors such as robot arms (~ L32), but the actual task is reconstructing motion from EEG ( ~ L150). These two tasks don't really match. First, decoding movement intent is not the same as reconstructing actual motion, and the two cannot be mixed together. Second, during reconstruction, the model can see the entire EEG segment, creating a cause-and-effect issue where future EEG signals are used to reconstruct current motion. Acknowledging that online control is "future work" doesn't really address this, because motion reconstruction and effector control are two different tasks, not just offline and online versions of the same thing.

  2. The evaluation does not assess the contribution of movement-related artifacts: In the paper's setup, the EEG is recorded while the subject is moving, so the recording may contain both real brain activity and signal that caused by muscle activities or sensor movement. Even though the author acknowledges that a prior study (~L1111 and Figure 7) shows that " Delta-band and all-band EEG show higher mean PCC and lower mean RMSE than the higher-frequency bands" and "This trend is consistent with prior EEG-based gait decoding studies" but such correlations do not establish which information this model actually uses in the current recordings. Additionally studies [1] show main walking artifact peaks in the delta band, which is also consistent with what is shown in Figure 7. i.e. the performance only using delta band is very similar as the performance of using full band.

[1]Castermans T, Duvinage M, Cheron G, Dutoit T. About the cortical origin of the low-delta and high-gamma rhythms observed in EEG signals during treadmill walking. Neurosci Lett. 2014 Feb 21;561:166-70. doi: 10.1016/j.neulet.2013.12.059. Epub 2014 Jan 8. PMID: 24412128.

  1. The contribution of EEG beyond the motion prior is unclear: The model may use EEG only to determine motion type or gait phase; the decoder then fills in the details learned during training, especially when movements within a dataset are repetitive and similar, and the model is trained per subject (~L834). Thus, the contribution of EEG beyond the motion prior is unclear. Action-conditioned motion generators such as ACTOR[2] and ODMO[3] show that action labels (embeddings) alone can synthesize realistic and diverse motion sequence. A comparison with simpler EEG-based movement-type or phase predictions and phase-to-motion generation can clarify what the learned token interface adds.

  2. The improvements do not establish reliable motion reconstruction across the evaluated subjects: The paper is motivated as a brain control application; despite the application mismatch, the R-square in the FBM dataset is only 0.11, with negative values for 4/9 subjects (Table 1) indicating performance below the mean-prediction reference under this metric. Although the method improves over the compared baselines, it remains unclear whether this method is useful for the intended application.

[2] Petrovich, Mathis, Michael J. Black, and Gül Varol. "Action-conditioned 3d human motion synthesis with transformer vae." 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021. [3] Lu, Qiujing, et al. "Action-conditioned on-demand motion generation." Proceedings of the 30th ACM International Conference on Multimedia. 2022.

  1. Missing citations: STTransformer (L298), EEGMotorContrastive (Figure 3)
Questions for authors
  1. What is the actual use case of reconstructing movements that the participant is already performing?

  2. What distinguishes neural information from movement-related artifacts in the EEG? The zero-input, reversal, and shuffling controls do not isolate these sources, and delta-only band appear to perform similarly to full-band inputs in Figure 7.

  3. In Figure 7, since using only the delta band appears to achieve similar performance to using full-band EEG, could the authors clarify why they use full-band EEG and whether the additional frequency bands improve performance or consistency across subjects?

  4. Table 3 includes the two highest-performing subjects in each dataset, while Figure 7 does not include the FBM subjects with negative R-square. Do the conclusions draw from these selected patients also hold for lower-performing subjects? especially the training is patient-specific.

  5. Could the authors compare their proposed model with a simpler EEG-based type or phase predictor connected to a motion decoder?

  6. There are 4/9 subjects having negative R-square in subjects specific training, What distinguishes these subjects from those with better performance? Are there particular subject characteristics or recording conditions that explain this difference?

8 comments
AuthorsOpenReview ↗

Response to Reviewer KBds(1/6)

We thank you for the thoughtful assessment and constructive feedback. We address the comments and questions below.

W1. Task definition, whole-window reconstruction, and the relationship to external-effector control

This study investigates whole-window reconstruction of continuous motion trajectories from EEG recorded during actual movement, rather than movement-intention recognition or closed-loop external-effector control.

First, external-effector control provides the application context, while continuous motion reconstruction is the task actually studied. Using synchronously recorded kinematic sequences as supervision, we investigate how discrete motion primitives and EEG–motion alignment can improve trajectory reconstruction. The experiments accordingly evaluate motion reconstruction performance. Section 2.1 defines this task as reconstructing the motion sequence paired with an EEG segment.

Second, noncausal, whole-window reconstruction is a valid decoding setting when delayed output is permitted. Our protocol reconstructs concurrent movement from a complete two-second EEG window, allowing each estimate to use information throughout the window. Related segment-input, sequence-output settings appear in MTRT (Wang et al., 2023) [1], which reconstructs four-second motion sequences, and E2T (Kwon et al., 2026) [2], which predicts framewise velocities from three-second EEG windows. We excluded E2T as a baseline because we could not find an official public implementation. NeuroDyGait (Fu et al., 2025) [3] also uses full two-second motion-sequence reconstruction to supervise pretraining. More explicitly, Lv et al. (2010) [4] use an offline Kalman smoother that combines past, present, and future observations to reconstruct movement velocity. These studies provide precedents for sequence-level supervision and reconstruction beyond strictly causal, moment-by-moment prediction.

Third, window-level trajectory reconstruction can support delayed effector execution, providing a concrete link to control applications. Control does not necessarily require identifying a discrete intention before converting it into an action: for example, π₀ directly generates continuous action sequences for robot control (Black et al., 2024) [5]. A potential use case is to collect an EEG window, reconstruct its movement trajectory, and execute it on a robotic arm through motion mapping and trajectory tracking. Because execution follows window acquisition, reconstruction can use the complete EEG segment; our two-second windows entail two seconds of buffering plus processing overhead before replay begins. A concrete engineering demonstration using a physical robotic arm was reported by Kim et al. (2015) [6]. They reconstructed three-dimensional trajectories from EEG recorded during actual reaching and used them to drive the robotic arm in offline reaching-and-grasping experiments. This provides a hardware-validated example of reconstructed trajectories being used for subsequent effector execution. Such applications require sufficient accuracy and acceptable timing and movement constraints. Our study validates the EEG-to-trajectory decoding module within this broader pipeline; the latency–accuracy trade-off with shorter windows and performance in an integrated control system remain to be evaluated.

W2. Movement and muscle artifacts, and interpretation of the low-frequency advantage

First, scalp EEG recorded during movement may contain neural activity, muscle activity, and electrode-motion artifacts, and our decoding results cannot directly distinguish their respective contributions. This is a shared challenge in EEG decoding during actual movement. We do not assume that all predictive information in the recordings originates from neural activity.

Second, we restrict the conclusions of the frequency-band analysis to predictive performance rather than signal origin. Appendix D.2 explicitly states that this analysis cannot separate neural activity from other low-frequency signal components. Figure 7 therefore supports the predictive value of low-frequency signals for movement under the current setting; comparisons with previous studies likewise concern only patterns of frequency-band performance.

Third, as you note, walking artifacts can affect the delta band (Castermans et al., 2014) [7], so similar performance between delta-band and full-band inputs is compatible with low-frequency movement artifacts contributing to prediction. The existing frequency-band experiments cannot distinguish this explanation from the contribution of low-frequency neural information or determine their respective contributions. We therefore treat Figure 7 as a supplementary analysis of frequency-band predictive performance and do not interpret the low-frequency advantage as evidence of neural origin.

AuthorsOpenReview ↗

Response to Reviewer KBds(2/6)

W3. The contribution of EEG relative to the motion prior and the need for the primitive interface

First, EEG predicts local primitive sequences in the motion prior, not whole-action categories or predefined gait phases. Primitives differ from these coarse labels. Each spans about 0.2–0.4 seconds of local motion (dataset-dependent) and is learned by motion encoding and quantization, not assigned semantic labels like walking or stair ascent. The model must predict primitives and their order at all window positions; reconstruction also depends on neighboring primitives. Thus, different combinations can represent local variations within one action category. The framework also covers FR upper-limb reaching and grasping without relying on unified definitions of periodic gait phases. Primitives may encode phase information, but prediction is not limited to phase classification.

Second, predicting primitive sequences from EEG and recovering trajectory details through the motion prior is the core of our hierarchical framework. During inference, the EEG encoder and E2M predict a primitive sequence; the motion decoder reconstructs its trajectory without further EEG input. Exploiting similar, recurring movements in each dataset, this replaces direct high-dimensional continuous trajectory prediction with prediction of structured local motion representations. This constrains learning from limited, low-signal-to-noise EEG. The key is hierarchical cooperation between EEG-based prediction of structured motion-prior codes and prior-based trajectory reconstruction, rather than an extra EEG pathway guiding trajectory details beyond primitive selection.

Third, plausible action generation and accurate reconstruction of the current trajectory require different evaluation criteria. ACTOR [8] and ODMO [9] show that action labels can support realistic, diverse motion generation but do not ensure temporal and numerical correspondence with a particular recorded trajectory. Our task is continuous trajectory regression: PCC, R², and RMSE assess synchronized trajectories' correlation, goodness of fit, and reconstruction error. FID and Diversity are supplementary; distributional consistency and diversity are central to motion generation. With limited movements and two-second windows, many segments in our datasets can only be labeled walking or raising an arm. Their differences mainly concern trajectory details that whole-action labels cannot distinguish. We must recover specific movement-state changes over time—for example, how the left wrist moves from (x1,y1,z1)(x_1,y_1,z_1) to (x2,y2,z2)(x_2,y_2,z_2). Labels such as “raising the right leg” or “the second gait phase” do not determine this process. We therefore predict local primitive sequences and refine continuous trajectories to preserve and improve these local variations.

Moreover, action-conditioned generators depend on their training motion representations and decoder spaces and cannot directly handle our heterogeneous targets. FBM, TM, and FR use dataset-specific formats: full-body motion representations, lower-limb joint angles, and wrist acceleration, respectively. These generators therefore require adapting motion representations and retraining, rather than simply replacing action labels with EEG conditioning.

Fourth, this requires a new task-specific method rather than an existing simplified baseline for this task. Action-conditioned generators' original motion spaces do not directly cover our full-body, joint-angle, and wrist-acceleration targets. Implementing “EEG → category/phase → motion” also requires appropriate labels for each task, trained condition predictors, and a mechanism mapping categories or phases to continuous trajectories. This is a worthwhile independent research direction rather than an ablation obtained by removing NeuroAction components. We therefore consider it future work, rather than an existing baseline that must be added to this study.

AuthorsOpenReview ↗

Response to Reviewer KBds(3/6)

W4. Absolute reconstruction performance, subjects with negative R², and practical value

First, continuous trajectory decoding has a direct methodological connection to external-effector control, and this study evaluates the decoding module within that pipeline. As explained in our response on task definition, decoded trajectories can provide inputs for motion mapping and trajectory tracking, making control applications a reasonable research motivation.

Second, absolute reconstruction performance on FBM remains limited, but methodological progress should be assessed relative to baselines evaluated under the same conditions. A mean R² of 0.11 indeed indicates that full-body motion reconstruction remains challenging. Under the same inputs, session splits, and prediction targets, NeuroAction achieves better mean performance than the compared methods, while component ablations support the contributions of the proposed design. These results demonstrate improved decoding under the available data conditions.

Third, mean performance needs subject-level context: a low overall value does not imply ineffective reconstruction for every subject. Appendix D.6, Table 12 reports PCC/R²/RMSE of 0.878/0.749/0.451 for FBM S10 and 0.725/0.474/0.536 for S03; other subjects remain challenging. The added linear-regression control also performs poorly on difficult subjects, suggesting differences in movement decodability from EEG. Recording quality and individual differences may also contribute, precluding attribution solely to the model. Against the strong overall baseline NeuroDyGait, NeuroAction improves PCC for all nine FBM subjects and both R² and RMSE for 7/9. Benefits thus extend across subjects.

Finally, this study is a methodological investigation of continuous EEG-to-motion decoding, and the current accuracy does not establish reliable application across all subjects. Its value lies in demonstrating that hierarchical motion representations, primitive alignment, and trajectory refinement can improve existing decoding results and apply to different motion targets. We have clarified in the Limitations that the low overall R² and negative values for some subjects indicate that reconstruction accuracy and consistency remain limited.

W5. References for STTransformer and EEGMotorContrastive

We have added the source citation for STTransformer. EEGMotorContrastive was a naming error and has been corrected to NeuroDyGait, whose reference was already included in the baseline description.

Q1. What is the practical purpose of reconstructing movements that subjects have already performed?

As discussed in W1, a concrete potential use case is delayed execution of reconstructed movement trajectories by an external effector. After acquiring a complete EEG window, the reconstructed motion segment can be mapped to a robotic arm and executed, allowing noncausal reconstruction with a window-length buffering delay plus processing overhead. Kim et al. (2015) [6] provide a concrete hardware demonstration: trajectories reconstructed from EEG recorded during actual reaching were used to drive a physical robotic arm in offline reaching-and-grasping experiments. Using EEG and kinematics synchronously recorded during actual movement, our study validates the EEG-to-trajectory decoding module by improving reconstruction under the same data conditions. External-effector applications additionally require meeting the relevant accuracy, timing, and movement constraints.

Q2. Can neural information be distinguished from movement artifacts using the current input controls and delta-band results?

As discussed in W2, the recordings contain neural activity together with possible movement, muscle, and electrode artifacts, and similar performance between delta-band and full-band inputs cannot establish the physiological origin of predictive information. Appendix D.2 explicitly states that frequency-band analysis cannot separate neural activity from other low-frequency signal components.

Table 3 examines the overall input information and its temporal structure rather than isolating artifacts. Zeroing tests the effect of removing all input information, while reversal and shuffling test the model’s dependence on temporal structure. This information includes both neural components and possible artifacts, so these controls do not separate or quantify their respective contributions.

Reviewer_Gb76OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NeuroAction, an EEG-to-motion decoding framework that uses VQ-based discrete motion primitives, contrastive EEG–motion alignment, Transformer-based token prediction, and geometric refinement. Experiments on three motion datasets demonstrate improvements over several baselines, supported by ablations, EEG input controls, and representation analyses.

Critical strengths

S1. Clear motivation and methodology. The paper addresses an important problem in EEG-based motion decoding. The motivation is convincing, and the proposed framework is clearly and concisely presented.

S2. Informative experimental analyses. I particularly appreciate the EEG input controls, representational similarity analysis (RSA), and component ablations, which provide insights beyond simply proposing a new method.

Critical weaknesses

W1. Large between-subject variability. Many results exhibit standard deviations comparable to or larger than their means. For example, STTransformer reports R² = -0.066 ± 0.533 on FR. Such substantial variability raises concerns about the robustness of the reported improvements, especially given the limited statistical significance on TM and FR.

W2. Figure quality. Several figures have noticeable formatting issues. Figure 3 contains an unexpected blue line near the EEGConformer legend, Figure 5 has overlapping legends and curves, and some annotations in Figures 4 and 6 are too small to read clearly.

W3. Insufficient discussion of related work. I found a highly relevant paper, EEG2MOTION: Towards Open-Vocabulary Human Motion Synthesis from Non-invasive Brain Signals (Peng et al., 2026). Although it uses action-observation EEG, its overall framework appears quite similar, including VQ-based motion tokenization, EEG–motion alignment, and token-based motion generation. The methodological novelty of NeuroAction relative to this work needs further clarification.

Questions for authors

Q1. What causes the substantial between-subject variability? How consistently do the reported improvements hold across subjects?

Q2. Could the authors clarify the statistical robustness of the improvements, particularly on TM and FR?

Q3. What are the key methodological differences between NeuroAction and EEG2MOTION? Why would directly adapting EEG2MOTION to your tasks be insufficient, and what specific limitations does NeuroAction address?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Gb76(1/3)

We thank you for the thoughtful assessment and constructive feedback. We address the comments and questions below.

W1. Between-subject variability and consistency of improvements

We acknowledge substantial between-subject variability, but NeuroAction's advantage is reflected not only in cohort-mean performance but also in the individual results of most subjects. The standard deviations in Table 1 describe differences in absolute performance across subjects, whereas the consistency of improvements should be assessed through paired comparisons on the same subjects.

First, between-subject variability is a common challenge in EEG decoding and is not specific to NeuroAction. Individual differences in neural activity and movement execution, together with acquisition-related factors such as electrode contact and signal quality, may affect decoding performance. Such variability is also evident among the compared baselines. Nevertheless, NeuroAction achieves the best mean reconstruction performance across all three datasets (Table 1), and the subject-level results support this advantage (Appendix D.6, Tables 12, 14, and 15). Compared with NeuroDyGait, the strongest original deep learning baseline overall, NeuroAction improves PCC for 9/9 FBM subjects and both R² and RMSE for 7/9; on FR, it improves PCC and R² for 5/5 subjects and RMSE for 4/5; on TM, it improves all three metrics for 4/6 subjects. Thus, the mean performance advantage is accompanied by improvements for most subjects.

Second, the limited numbers of subjects in TM and FR constrain statistical power, and we explicitly acknowledge this limitation in the manuscript. TM and FR include six and five subjects, respectively. With substantial between-subject variability, these sample sizes limit the evidence for statistical significance when improvements are modest. Section 3.2 and Appendix D.7 report and discuss this issue: the evidence for improvement is clearest on FBM, while statistical support on TM and FR is more limited. We therefore restrict our conclusions to the best cohort-mean performance among the compared methods and improvements for most subjects under the datasets and evaluation conditions studied.

W2. Figure quality

In the revised manuscript, we have removed the extraneous blue line near the EEGConformer legend in Figure 3, repositioned the legend in Figure 5 to avoid obscuring the curves, and preprocessed the annotations in Figures 4 and 6 to improve readability.

AuthorsOpenReview ↗

Response to Reviewer Gb76(2/3)

W3. Connection to EEG2MOTION and methodological differences

We thank the reviewer for noting EEG2MOTION [1], a concurrent work. It first appeared on arXiv on August 14, 2026, about one month before our submission; our main research and manuscript preparation were completed earlier. We were unaware of this work during submission preparation, so the original manuscript omitted its discussion. We have added its citation and clarified their connections and differences in the revised manuscript's Related Work. We acknowledge that both works draw inspiration from discrete motion generation in text-to-motion research. However, NeuroAction's main contributions concern the task formulation for decoding actual movement trajectories, primitive-level EEG–motion alignment, and continuous trajectory refinement. Differences follow.

First, the key methodological difference is whether explicit alignment supervises global whole-action representations or discrete primitives for local movement segments within a window. EEG2MOTION pools EEG tokens into a global representation and contrastively aligns it with global features of the entire motion sequence. Its explicit alignment objective thus operates at the whole-window level. NeuroAction instead uses PCL and SupCon with local movement segments' codebook assignments to directly constrain the relationship between corresponding EEG representations and motion primitives. The alignment prototypes, sequence-prediction labels, and motion-decoding codebook belong to the same representation system. This design makes representation learning support local primitive identification and subsequent trajectory reconstruction beyond global action semantics. For the short-window continuous decoding studied here, broad action semantics alone cannot ensure accurate recovery of trajectory phase, amplitude, and local variations, motivating this finer-grained supervision.

Motion representations and signal conditions also differ. HumanML3D uses a unified 263-D motion representation; EEG2MOTION uses motion quantizers and generators pretrained in this space. NeuroAction handles heterogeneous EEG–motion data: FBM full-body representations, TM lower-limb joint angles, and FR wrist acceleration. We learn each dataset's motion prior and link EEG to its targets via shared alignment and sequence decoding. Actual movement introduces electrode-motion and muscle interference, yielding different signal conditions from seated action observation.

Second, tasks and objectives differ. EEG2MOTION generates observed motion sequences from EEG recorded as seated subjects watch action videos. Its one-second EEG segments correspond to roughly five-second motion clips played at 5× speed. It primarily assesses semantic matching and generation quality with R-Precision, MM-Dist, FID, and Diversity. NeuroAction reconstructs continuous trajectories within the corresponding window from EEG synchronously recorded during actual movement, using PCC, R², and RMSE to assess temporal correspondence and numerical accuracy. Thus, generating motion from action-observation EEG and decoding continuous trajectories during actual movement are fundamentally different tasks. Its authors note in their discussion that visual cortical activity primarily drives retrieval performance, while transfer to active-control paradigms still requires validation.

Third, different data conditions lead to different decoding priorities. EEG2MOTION uses HumanML3D motions spanning diverse action types. About five-second clips can convey relatively complete action semantics for EEG-based matching and motion generation. Recording paradigms instead limit our EEG–motion datasets to fewer movement types. Under coarse semantic labels, many segments share just one or two categories, such as walking or reaching. The key challenge is recovering each semantically simple segment's actual trajectory—its phase, amplitude, and local variations. Generating semantically correct walking still requires accurately reconstructing the current subject's specific movement within the corresponding window. Thus, EEG2MOTION prioritizes matching and generation across action semantics; NeuroAction, fine-grained trajectory reconstruction within similar actions, synchronized with current EEG.

Finally, continuous motion-space refinement further improves NeuroAction's trajectory accuracy. After primitive-sequence prediction, Stage 3 uses continuous trajectory MSE to jointly update the EEG encoder, E2M, and motion decoder while freezing the motion encoder and codebook. This directly optimizes continuous motion reconstruction error. These results and Table 2's performance decline without alignment support NeuroAction's contribution beyond a similar discrete generation pipeline: combining primitive-level alignment, sequence prediction, and trajectory refinement to improve continuous trajectory decoding from EEG recorded during actual movement.

AuthorsOpenReview ↗

Response to Reviewer Gb76(3/3)

Q1. Sources of between-subject variability and consistency of improvements

Regarding consistency, the subject-level comparisons summarized in W1 show improvements for most subjects. Compared with NeuroDyGait, NeuroAction improves PCC for 9/9 FBM subjects and both R² and RMSE for 7/9; on FR, it improves PCC and R² for 5/5 subjects and RMSE for 4/5; on TM, it improves all three metrics for 4/6 subjects. Thus, NeuroAction’s mean performance advantage is accompanied by improvements for most subjects.

Regarding the sources of variability, individual neural activity, movement execution, electrode contact, and recording quality may all affect the decodable relationship between EEG and movement. The available datasets do not provide sufficient subject-specific and acquisition information to determine the individual causes, so these factors remain possible explanations. We assess the consistency of improvements through comparisons between methods evaluated on the same subjects under the same recording conditions.

Q2. Statistical robustness of improvements on TM and FR

Appendix D.7 reports two-sided paired t-tests comparing NeuroAction Stage 3 with NeuroDyGait, using subjects as the paired units. The results are summarized below.

ItemFBMTMFR
Number of subjects965
PCC p-value0.000270.2910.0407
R² p-value0.02290.2020.0519
RMSE p-value0.02460.2220.125

At the uncorrected p < 0.05 threshold, all three FBM metrics and FR PCC reach statistical significance, whereas the three metrics on TM and R² and RMSE on FR do not. It is attributed to the fact that TM and FR include only six and five subjects, respectively, and substantial between-subject variability further limits statistical power. Section 3.2 and Appendix D.7 explicitly acknowledge these results: the evidence for improvement is clearest on FBM, while statistical support on TM and FR is more limited. Their performance advantages are also reflected in the cohort means and paired improvements for most subjects.

Q3. Challenges in directly adapting EEG2MOTION and the specific limitations addressed by NeuroAction

Please see W3 above for the key methodological differences and differences in task objectives.

Direct adaptation first requires addressing differences in the motion space and motion prior. EEG2MOTION uses a motion quantizer, codebook, and decoder pretrained on HumanML3D, which cannot be used unchanged for TM joint angles or FR wrist acceleration. Although FBM also uses a 263-dimensional representation, its skeletal joint configuration and data domain differ. NeuroAction addresses the limitation that semantically consistent motion generation does not ensure accurate trajectory details. As discussed in W3, NeuroAction targets local representation matching and accurate trajectory reconstruction, whereas EEG2MOTION emphasizes semantic consistency of the generated actions. Whole-window global semantic alignment does not directly constrain the correspondence between local EEG representations and motion primitives, and semantically consistent motion generation does not guarantee accurate phase, amplitude, or local variations in the corresponding trajectory. NeuroAction introduces primitive-level alignment through PCL and SupCon, then optimizes reconstruction accuracy through continuous trajectory MSE in Stage 3, explicitly addressing local representation matching and continuous trajectory error, respectively.

References

[1] Peng, Y., Pan, Y., Yang, Y., Zheng, N., Chen, W., Hu, X., & Zhang, S. (2026). EEG2MOTION: Towards Open-Vocabulary Human Motion Synthesis from Non-invasive Brain Signals. arXiv preprint arXiv:2608.14754. https://arxiv.org/abs/2608.14754

Reviewer_3jFNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to map EEG to discrete motion primitives for continuous motion decoding. The EEG token embeddings are aligned to the codebook vectors with prototypical and supervised contrastive losses, and are piped into a VQ-VAE that that learns dataset-specific motion codebooks. EEG token embeddings are aligned to the codebook vectors and a transformer with learnable target slots "stiches" the decoded sequence together.

Critical strengths

In the paper, the core idea is that treating EEG as coarse evidence for selecting motion primitives, rather than regressing joint trajectories directly, is fair, as in most cases the EEG signal (as the authors say) is indeed too noisy for anything approaching regression.

Critical weaknesses

A common challenge when benchmarking oneself with others is how much tuning and capacity does one allocate to "bechnmark" models. The benchmarking supports the claim that NeuroAction outperforms these baselines as adapted. It does however not establish state of the art, here is why: NeuroAction has 80M parameters on FBM and gets three-stage training with dataset-specific widths, codebooks and token counts. The baselines get a fixed learning rat4, a dropout sweep and early stopping (their parameter counts are not reported). The authors acknowledge that capacities are not identical, but the result is that this mixes method and tuning effort. Obvious baselines are missing, no linear/ridge regression on low-frequency EEG. Luu et al. (2016, JNeuralEng)decoded (I think) the same treadmill dataset with a UKF, and this paper cites it but neither compares against it nor reports numbers on its protocol.

Questions for authors

How important is the impact of motion and muscle artefacts, do they carry a substantial part of the signal? Decoding relies mainly on delta-band and all-band EEG. This is overground walking and stair climbing, and this may be artefact based performance. Can you show it is not that? E.g. an evaluation after artefact removal (say ICA) and a baseline decoding from non-neural channels such as EMG, if available.

Is the FBM innovation really the driver? NeuroAction is essentially a classifier over a number of primitives. A baseline that classifies locomotion mode or primitive from EEG and outputs the class-mean would show how much of the FBM gain comes from discretisation rather than from alignment.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 3jFN(1/4)

We thank you for the thoughtful assessment and constructive feedback. We address the comments and questions below.

W. Baseline comparisons, model capacity, training effort, and classical decoders

First, we agree that the performance claims should be stated more precisely. Existing EEG-to-motion decoding studies involve different movement tasks, target representations, and evaluation protocols, and many methods are designed for specific datasets. Including them in a unified comparison therefore requires appropriate input and output adaptations. Our experiments support NeuroAction's superior performance over the compared methods under the unified last-session test protocol used in this study, but do not establish an unqualified field-wide state-of-the-art claim. We have revised the corresponding performance claims in the Abstract, Introduction, Section 3.2, and Conclusion.

Second, we have taken explicit measures to ensure a fair baseline comparison. As described in Appendix B.4, all methods use the same EEG preprocessing, session splits, prediction targets, normalization, and evaluation metrics. We retain most of the backbone structures from the public implementations and make the input and output adaptations required for each dataset. For each deep learning baseline, dropout is selected from {0.0, 0.1, …, 0.9}, and training uses an initial learning rate of 10⁻⁴, cosine learning-rate scheduling, a 200-epoch budget, and early stopping with patience 10. We therefore preserve the original implementations as closely as possible while conducting an explicit hyperparameter search.

Regarding model capacity. The full default models have 89.804M, 22.458M, and 24.359M parameters for FBM, TM, and FR, respectively; approximately 80M refers specifically to the FBM inference model. Table 1 now lists parameter counts beneath method names in FBM/FR/TM order, including adapted regression heads, allowing joint assessment of capacity and performance. Table 5 also lists model-size ablation parameter counts in FBM/TM/FR order, with reconstruction results in Table 6. Settings for the added LR, RR, and UKF baselines are detailed separately in Appendix B.4. The capacity ablations show that even the smallest Extra Small configuration achieves PCC/R²/RMSE of 0.856/0.714/0.469 on FBM S10, 0.791/0.283/0.311 on TM S07, and 0.766/0.545/0.187 on FR S05 (Appendix D.1, Tables 5–6), outperforming the compared baselines on these subjects while remaining close to the largest configuration. These representative-subject results show that the framework's effectiveness is not limited to its largest default configuration.

Regarding the three-stage training procedure. Multistage training is common in discrete representation learning and model distillation. As described in Section 2.7, NeuroAction's three stages respectively learn the motion prior, predict and align motion primitives from EEG, and refine continuous trajectories. Together, they form a training process that progresses from discrete representation learning to continuous reconstruction optimization.

Appendix B.2 reports the actual training costs. All training uses a single NVIDIA L4 GPU. Stage 1 is performed once per dataset and takes approximately six hours for FBM and two hours for TM or FR; the resulting motion prior is reused across subject-specific training within the same dataset. Subject-specific Stages 2 and 3 take approximately 10–60 minutes and 3–12 minutes, respectively, for a combined 13–72 minutes. Reusing the dataset-level motion prior therefore allows subsequent subject-specific training to be completed on roughly an hour scale on a single GPU, keeping the training cost of the three-stage design reasonable.

Finally, we have added classical decoder comparisons. As suggested by the reviewer, we include linear regression (LR), ridge regression (RR), and an unscented Kalman filter (UKF), adapted from Nakagome et al.'s (2020) public EEG-neural-decoding implementation [1]. All methods use the same session splits, prediction targets, and evaluation metrics. Results are reported in Section 3.2 and revised Table 1, with adaptation and implementation details in Appendix B.4 and subject-level reconstruction results in Appendix D.6, Tables 12, 14, and 15 for FBM, FR, and TM, respectively. Each cell reports the cohort-mean PCC/R²/RMSE.

MethodFBMFRTM
LR (added)0.218 / −1.374 / 1.2150.298 / −0.611 / 0.3430.183 / −1.720 / 0.719
RR (added)0.219 / −1.359 / 1.2110.285 / −0.678 / 0.3140.166 / −1.759 / 0.678
UKF (added)0.182 / −0.613 / 1.0410.193 / −2.614 / 0.4930.183 / −2.213 / 0.731
NeuroAction Stage 30.522 / 0.110 / 0.7180.657 / 0.250 / 0.2300.485 / 0.184 / 0.404

NeuroAction still outperforms the added classical decoders on all three cohort-mean reconstruction metrics across all three datasets.

AuthorsOpenReview ↗

Response to Reviewer 3jFN(2/4)

Q1. The influence of movement and muscle artifacts

We agree that scalp EEG during actual movement may contain non-neural components from eye movements, muscle activity, and electrode motion. Strong delta-band or full-band prediction alone does not establish that all predictive information is neural.

This issue requires distinguishing the signals' physiological origin from decoding methods' relative effectiveness. It also requires considering artifact-removal assumptions and limitations and processing differences across datasets and studies.

First, uniform artifact removal across datasets cannot guarantee preservation of all neural information. For example, ICA assumes statistically independent sources and relatively stable linear mixing. These components still require identification and cannot automatically be labeled purely neural or artifactual. Artifacts during movement may span multiple components: removing only a few may leave residual contamination, whereas broader removal may also discard neural information. Studies of walking and running have likewise reported that gait-related movement artifacts can remain in many independent components, making them difficult to remove by rejecting only a few components (Gwin et al., 2010) [2].

Second, related decoding studies differ in preprocessing scope. NeuroDyGait (Fu et al., 2025) [3] reports bandpass filtering, common-average referencing and resampling, but no additional ICA-component rejection. MTRT (Wang et al., 2023) [4] reports EOG removal, abnormal-channel and trial rejection, and experimental limits on unrelated body movements. These reflect differing acquisition conditions and tasks, not a uniform comprehensive artifact-removal standard. Our original comparison thus uses identical inputs across methods.

We added before-and-after comparisons to test our methodological conclusions' robustness to existing artifact-processing strategies. Although absolute decoding performance decreases after processing, NeuroAction retains the best mean performance among compared methods using the same inputs.

Additional processing uses each dataset's auxiliary recordings and released options. TM/FR provide EOG but no synchronized EMG or publisher-released version with movement artifacts removed. Identifying further muscle/electrode-motion components requires separately defined, validated criteria. Ocular correction follows the dataset publications: EOG-reference adaptive denoising for TM (He et al., 2018) [5] and EOG-assisted ICA removal of ocular components for FR (Thapa et al., 2025) [7]. For FBM, we use publisher-released EEG with bad-channel handling, re-referencing and artifact subspace reconstruction (ASR) (Brantley et al., 2018) [6].

Per dataset, we selected three representative subjects spanning high, intermediate, and low performance in the original main experiments—FBM S02/S07/S10, FR S05/S07/S22, and TM S01/S02/S05—and retrained and evaluated NeuroAction, linear regression, and NeuroDyGait, the strongest baseline overall. All methods shared processed EEG and original session splits, prediction targets and evaluation metrics. Before/after results are in Appendix D.1, Table 11 and below. Cells list PCC / R² / RMSE.

EEG inputMethodFBMFRTM
OriginalNeuroAction0.568 / 0.232 / 0.7340.698 / 0.272 / 0.2110.363 / 0.104 / 0.428
OriginalLinear regression0.295 / −1.203 / 1.3390.391 / −0.185 / 0.2700.111 / −1.473 / 0.753
OriginalNeuroDyGait0.065 / −0.140 / 0.9950.633 / 0.158 / 0.2230.327 / 0.030 / 0.444
ProcessedNeuroAction0.494 / 0.181 / 0.7610.567 / 0.035 / 0.2440.199 / 0.010 / 0.452
ProcessedLinear regression0.215 / −0.914 / 1.3040.216 / −1.392 / 0.3750.041 / −6.243 / 1.115
ProcessedNeuroDyGait0.044 / −0.149 / 0.9980.485 / −0.223 / 0.2590.075 / −0.030 / 0.461

Although absolute decoding performance decreased after processing, NeuroAction retained the best mean performance among the compared methods across all three datasets. This suggests that altered or removed components contained movement-predictive information. However, the decline cannot be attributed entirely to artifacts, since processing may also remove neural information mixed with these components. NeuroAction nevertheless retained its relative advantage over the baselines under these specified artifact-reduction procedures.

For an EMG control, available recordings cannot support a uniform auxiliary-channel comparison across all three datasets and all movement targets. TM and FR lack EMG, and FBM records only lower-limb muscles. Moreover, EMG-to-motion prediction cannot directly quantify EMG contamination's contribution within EEG. Our supplementary experiments thus examine whether decoding performance and each method's relative advantage are preserved after processing the original EEG inputs.

AuthorsOpenReview ↗

Response to Reviewer 3jFN(3/4)

Q2. Do the performance gains on FBM come from discretization or alignment?

Our key point is that hierarchical motion representation and EEG–motion alignment jointly contribute to decoding, and the existing ablations show that discretization alone cannot explain the performance gains on FBM. These are the two main contributions of our method: introducing a hierarchical motion representation into EEG-based motion decoding—as the reviewer describes, decomposing the task into the classification of discrete motion primitives—and aligning EEG representations with those primitives. Moreover, reconstructing continuous trajectories from primitive sequences differs substantially from identifying a movement category and outputting its mean trajectory.

First, alignment and discrete primitive classification are closely connected: alignment helps EEG representations support primitive classification. Our hierarchical framework uses motion primitives as an intermediate representation, and alignment helps EEG representations adapt to this space. Specifically, PCL and SupCon constrain EEG representations using the motion codebook and its assignments, guiding representation learning through the structure of motion prototypes rather than relying solely on the final prediction error. This representation-level regularization is intended to improve primitive prediction from low-signal-to-noise EEG and thereby support subsequent trajectory generation.

Second, our hierarchical decoding is not equivalent to classifying a movement and outputting the corresponding mean trajectory. Codebook entries are learned local latent representations, rather than predefined labels such as level-ground walking, stair ascent, or stair descent. The model predicts a sequence of motion primitives, from which a motion decoder with a temporal receptive field reconstructs the continuous trajectory. The same primitive can produce different local outputs depending on its neighbors. For example, the interval associated with primitive “2” may be decoded differently in the sequences “1–2–3” and “3–2–1.” In addition, E2M predicts the motion-token sequence through attention between the EEG-token sequence and target positions, allowing primitive prediction to use sequence context rather than independently classifying each position and concatenating fixed templates.

Third, the existing ablation experiments support the additional contribution of alignment.

The w/o Alignment configuration in Table 2 retains motion discretization, E2M, and motion decoding, removing only PCL and SupCon. It therefore evaluates hierarchical decoding without explicit alignment constraints. Results across four FBM subjects show that removing alignment substantially worsens mean PCC, R², and RMSE. Even with the same discrete representation and sequence-decoding architecture, the configuration without alignment does not achieve the full model's performance, indicating that discretization alone cannot explain the entire improvement.

The w/o E2M ablation further shows that relatively independent per-position primitive classification remains insufficient to match the full method's performance. This configuration uses the alignment-trained EEG encoder to select codes directly through cosine similarity with codebook vectors, followed by reconstruction with the motion decoder. It removes both the E2M Transformer and Stage 3, thereby removing E2M's mechanism for exploiting temporal context during primitive prediction, and performs markedly worse than the full model.

We additionally introduce a more direct classification control, Primitive classifier, by further removing alignment from w/o E2M and adding a classification head at each EEG-token position to classify motion primitives individually, followed by reconstruction with the same motion decoder. This additional control is described in Section 3.3 and evaluated on the same four-subject subsets as the other component ablations. Its results are reported in Table 2 and summarized below. Each cell lists PCC / R² / RMSE.

ConfigurationFBMFRTM
Primitive classifier0.206 / −0.019 / 0.9720.545 / 0.061 / 0.2670.151 / −0.097 / 0.483

This configuration performs worse than w/o Alignment and the full model, and is weaker overall than w/o E2M. Together, these experiments support the combined contribution of hierarchical decoding that incorporates temporal context and alignment constraints.

Comments on the paper

1 comment

Dear Reviewers,

Please take a moment to review the authors’ rebuttal, assess whether your concerns have been adequately addressed, and actively participate in the discussion. Please update your assessment if needed. Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.