acceptodds
Under review as a conference paper at ICLR 2027

Anchoring Motion, Refining Trajectories: Generalizable Continuous Neural Decoding

Abstract

Brain activity encodes motor information across multiple temporal scales, enabling continuous neural control beyond discrete action recognition. However, existing methods are typically tailored to specific subjects and tasks, with direct neural-to-kinematic mappings that can entangle subject-specific neural variability with task-dependent motion patterns, limiting generalization. To address these limitations, we introduce AMRT, a shared neural decoding framework for heterogeneous electrode configurations and movement paradigms. Motivated by cortical encoding of movement direction and speed, AMRT adopts a two-stage design that anchors local motion through velocity estimation and refines the resulting trajectories over longer horizons, encouraging reuse of local motion tendencies across subjects and tasks. Across three datasets, AMRT generalizes to unseen trials and subjects and demonstrates transfer potential across neural recordings and movement paradigms, highlighting the potential of shared neural representations and anchor-and-refine decoding for generalizable continuous neural control. Our code is available at https://anonymous.4open.science/r/AMRT.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Continuous Neural Decoding, Electroencephalography (EEG), Motor Trajectory Decoding, Cross-Subject Generalization, Brain-Computer Interface

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mt8UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AMRT, which decodes continuous hand or cursor trajectories from scalp EEG and intracranial EEG with one framework meant to work across subjects, electrode layouts, and datasets.

The paper’s main contribution is a two-stage decoder that first predicts velocity and integrates it into a rough trajectory, then uses the neural signals together with that trajectory to correct the drift, and it pairs this with channel adapters that let one model handle both fixed-cap EEG and variable intracranial electrode layouts. Its second contribution is a step-by-step generalization study across three datasets, covering unseen trials, unseen subjects, few-shot calibration, and what the authors call the first systematic cross-dataset transfer study for continuous decoding between scalp and intracranial EEG.

The proposed method is tested on WAY-EEG-GAL, U-Limb, and Joystick-Track for unseen trials, unseen subjects, few-shot calibration, and transfer between datasets, using PCC. AMRT beats three recent EEG decoders. Pretraining on another dataset lets it reach 0.454 on U-Limb with 10% of the target data, against 0.192 when trained from scratch on all of it. However, transfer between EEG and iEEG fails in both directions, and predicting position directly does as well as AMRT or better when training and testing on the same subject.

Critical strengths
  1. The target problem is very relevant in the field. Continuous kinematic decoding that generalizes across subjects, electrode layouts, and datasets is an important and under-explored problem for practical BCIs.
  2. The proposed method is intuitive. Splitting decoding into velocity-based motion anchoring and trajectory refinement is easy to follow, and it clearly helps on U-Limb, the dataset with the most diverse movements.
  3. Evaluation experiments are comprehensive. The experiments go step by step from unseen trials to unseen subjects to transfer between EEG and iEEG datasets.
  4. The authors provided source code for reproduction of results.
Critical weaknesses
  1. The manuscript has many grammatical and spelling errors, including in the main figures. For example, Figure 1 has “Ressidual” for “Residual”. The whole manuscript and its figures need careful proofreading.
  2. The caption of Figure 1 is a single sentence. It should briefly explain each module and the data flow from the channel adapters, through the encoder, to Stage-0 and Stage-1.
  3. All datasets involve executed movements, and the preprocessing uses only a notch filter and common average reference, with no ICA or removal of motion, eye or muscle artifacts. Such artifacts are time-locked to the task, especially in the low-frequency band used for WAY-EEG-GAL, so the model could learn them as a shortcut instead of decoding neural activity. The authors should report results after artifact removal and add a shuffled-EEG or template-trajectory baseline to show that performance depends on genuine neural information.
  4. The PCC metric ignores trajectory scale and offset, yet correcting integration drift in amplitude and position is the main role of Stage1. A prediction with the right shape but the wrong amplitude can still score a high PCC. The authors should report R² and RMSE alongside PCC, especially when comparing AMRT with the velocity-integration baseline.
  5. Figure 2 is low quality, and most of its text is too small to read.
  6. Table 8 shows that cross-dataset pretraining helps substantially only when U-Limb is the target. Transfer from EEG to iEEG never beats scratch, and with Stage-1-only adaptation it is far worse (W→J at 10%: −0.047 vs. 0.730). Pretraining on two source datasets is also worse than the best single source for U-Limb (Table 10). Yet the abstract and conclusion describe transfer potential in general terms. The authors could state these limits explicitly in a limitations section and discuss why transfer to iEEG fails.
Questions for authors
  1. Why is Li et al.(2026a), “Learning Residual Kinematic Corrections for Continuous Neural Decoding via Reinforcement Learning”, only mentioned in Appendix A.3 and not compared as a baseline? It seems closely related to the proposed two-stage design and could be a good baseline comparison.
  2. The manuscript reports results over 10 random seeds, but the released code uses 11, 5 and 4 seeds for WAY-EEG-GAL, U-Limb and Joystick-Track. Can the authors clarify the seed setting? In addition, could you please clarify whether the 8:1:1 split is fixed for Tables 1 and 3 across seeds, or were trials shuffled?
  3. The code adds a grasp-classification loss for WAY-EEG-GAL (weight 0.5), a trajectory-consistency loss for U-Limb, per-axis loss reweighting, and a gate-sparsity term in Stage-1, none of which are discussed in the paper. Were these used or dropped in the reported experiments?
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer mt8U

We thank the reviewer for the constructive comments. We have addressed the presentation, evaluation, and reproducibility concerns as follows.

W1/W2/W5: Improved presentation and figure clarity. We have proofread the manuscript, corrected spelling and grammatical errors, improved the readability of Figures 1–2, and expanded the Figure 1 caption to explain the roles and data flow.

W3: Robustness to artifacts and trajectory-template controls. We adopt consistent preprocessing across EEG/iEEG datasets (Sec. 3.1). To assess artifact contamination, we additionally applied ICA to remove candidate ocular and muscle components on WAY-EEG-GAL under the ctri protocol. PCC remained comparable to the original result (0.90 ± 0.03 vs. 0.91 ± 0.03), suggesting limited sensitivity to the removed components. We also added an EEG-free trajectory-template baseline (Appendix E.3.4). Cross-dataset template transfer yields generally low PCC. Together with successful W→U and J→U transfer, these findings suggest that AMRT's cross-dataset gains cannot be fully explained by stereotyped trajectory patterns.

Cross-dataset trajectory-template transfer:

SourceTargetPCC ↑RMSE ↓
WU0.35 ±\pm 0.08108.61 ±\pm 9.37
JU−0.16 ±\pm 0.02128.97 ±\pm 6.43
WJU0.18 ±\pm 0.08103.37 ±\pm 10.00
UW0.28 ±\pm 0.048.64 ±\pm 0.51
JW−0.08 ±\pm 0.1211.87 ±\pm 0.88
UJW0.06 ±\pm 0.189.65 ±\pm 0.58
UJ−0.41 ±\pm 0.1014466.14 ±\pm 451.11
WJ−0.18 ±\pm 0.1216155.78 ±\pm 412.53
UWJ−0.26 ±\pm 0.1114874.93 ±\pm 183.37

W4: Additional positional error metrics. We add trajectory RMSE results in Appendix E.1.1 (Tables 7–9). Compared with Vel-Int, AMRT reduces RMSE across all evaluated settings, demonstrating improved positional reconstruction beyond trajectory PCC.

DatasetSettingVel-IntAMRT
Wctri16.764 ±\pm 13.3664.047 ±\pm 0.616
ctri-mix12.260 ±\pm 4.5025.237 ±\pm 1.159
csub20.814 ±\pm 15.9115.809 ±\pm 0.565
Uctri-mix85.979 ±\pm 4.33180.926 ±\pm 3.760
csub87.462 ±\pm 16.05779.106 ±\pm 13.723
Jctri9028.251 ±\pm 1842.1467422.698 ±\pm 604.229
ctri-mix8591.401 ±\pm 1070.7377692.757 ±\pm 616.446
csub13465.264 ±\pm 1075.4138494.897 ±\pm 1492.847

W6: Directional cross-dataset transfer. The observed asymmetry may reflect differences in neural sampling and task compatibility. Scalp EEG captures spatially mixed activity from distributed cortical sources, whereas ECoG provides more localized measurements with subject-specific cortical coverage [1]. Thus, local motion-related features learned from ECoG may remain accessible through EEG, while distributed spatial patterns learned from EEG may not be preserved in locally sampled ECoG recordings. Task differences may also contribute: our additional trajectory-template transfer analysis yields PCCs of −0.161 ± 0.021 for J→U and −0.406 ± 0.095 for U→J, suggesting greater difficulty in the reverse direction.

Q1: Comparison with Li et al. (2026a). We discuss this closely related work in Appendix A.3. As its code is not publicly available, a direct comparison is currently unavailable. We will reproduce the method based on the paper's description and evaluate it later.

Q2: Random seeds and data partitioning. The reported results follow the configurations described in the paper; the released scripts contain example configurations and do not fully specify every reported run. Under ctri-mix, complete trials are randomly shuffled and repartitioned at an 8:1:1 ratio for each seed, rather than using a fixed split across seeds (Sec. 3.2.1). This avoids window-level leakage and evaluates robustness across different trial partitions.

Q3: Clarification of training losses. The grasp-classification loss (weight 0.5) is associated with the supplementary grasp-state decoding experiment (Appendix E.3.2). The U-Limb trajectory-consistency loss and Stage-1 gate-sparsity regularization were exploratory variants and were not used in the final reported experiments. Per-axis loss reweighting was used but omitted from the manuscript; we have added its description in D.2.

Reference [1] Dubey A, Ray S. Cortical electrocorticogram (ECoG) is a local signal[J]. The Journal of Neuroscience, 2019, 39(22): 4299-4311.

Thanks for the author's response and additional results. However, it is still unclear to me. Does the ICA check cover only WAY-EEG-GAL under ctri? What about U-Limb, where AMRT’s gains are largest? Could you please report the baseline results under the within-dataset setting and artifact removal on U-Limb and under csub?

AuthorsOpenReview ↗

Author Response to Reviewer mt8U 2

We thank the reviewer for the thoughtful follow-up and for highlighting the need for a more comprehensive artifact analysis.

We confirm that our previous rebuttal only reported the ICA analysis on WAY-EEG-GAL under ctri. We have now extended the evaluation to U-Limb under both ctri-mix and csub-LOSO using ICA-based artifact removal.

ProtocolInputPCC ↑ΔPCCRMSE ↓
csub-LOSOBefore ICA0.430 ± 0.073—79.106 ± 13.723
csub-LOSOAfter ICA0.424 ± 0.114−0.00681.500 ± 15.687
ctri-mixBefore ICA0.530 ± 0.029—80.926 ± 3.760
ctri-mixAfter ICA0.522 ± 0.029−0.00881.658 ± 5.136

Across both protocols, ICA-based artifact removal results in negligible PCC reductions (≤ 0.008), with modest RMSE changes. These findings demonstrate that AMRT's decoding performance is largely unaffected by ICA-based artifact removal, including under cross-subject evaluation, suggesting that the observed performance is unlikely to be primarily driven by the removed artifacts.

Reviewer_DtEGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AMRT, a framework for continuous movement decoding across subjects and datasets. Its main design elements are a channel adapter for heterogeneous electrode configurations and a two-stage velocity-anchoring and trajectory-refinement scheme. For recordings with consistent channel identities, the channel adapter uses label-based mapping with an optional learnable linear residual. For variable electrode layouts, it jointly encodes neural waveforms and electrode coordinates and uses attention-based aggregation to produce a shared, fixed-dimensional representation. The two-stage decoder first predicts velocity from neural signals and integrates it over time to construct a coarse trajectory (Stage-0). Stage-0 is subsequently frozen, and Stage-1 combines its neural features with the coarse trajectory to predict gated residual increments that accumulate into trajectory corrections. The authors evaluate the framework on WAY-EEG-GAL, U-Limb, and Joystick-Track, covering EEG and ECoG recordings, through cross-trial, cross-subject, and cross-dataset transfer experiments. Trajectory Pearson correlation coefficient is the primary evaluation metric, and reconstruction-strategy comparisons and component ablations are used to assess the contributions of the proposed design. The reported results show improvements in several settings, while cross-dataset transfer benefits depend substantially on the source–target pairing.

Critical strengths
  1. The research problem has clear practical relevance
    The paper addresses continuous movement decoding and its generalization across subjects and recording configurations, which are relevant to the continuous and fine-grained control required by brain-controlled assistive devices in everyday settings. Compared with recognizing discrete action categories alone, continuous trajectory decoding more directly addresses the control requirements of such applications. Improving its applicability across users and recording conditions is therefore a practically meaningful research direction.

  2. The two-stage design has a reasonable rationale and receives partial experimental support
    Separating velocity estimation from trajectory refinement provides a clear modeling strategy for addressing errors accumulated through velocity integration. Stage-1 conditions its corrections on both the coarse trajectory and neural features, rather than relying exclusively on postprocessing the first-stage predictions. Table 3 provides partial support for this design: for example, under cross-subject evaluation on U-Limb, adding Stage-1 improves trajectory PCC from 0.369 for Vel-Int to 0.430. This demonstrates a benefit of trajectory refinement in at least some generalization settings.

Critical weaknesses
  1. Limited methodological novelty, with an unclear substantive advance over existing approaches
    The paper presents channel adaptation and two-stage decoding as its main contributions, but their methodological advances over existing approaches remain insufficiently clear. For EEG recordings with consistent channel labels, the fixed mapping primarily performs channel reordering and zero-padding for missing channels, while the learnable residual in Eq. (2) introduces an additional linear mixing of channels. This appears consistent with a combination of conventional channel organization and a learnable spatial transformation. Although Table 2 includes an ablation of the overall channel adapter, it does not clearly isolate the effect of removing only the learnable residual while retaining the same label-based mapping. The independent contribution of this residual therefore remains unclear. For variable electrode layouts, the geometry-aware branch does go beyond simple zero-padding, but its core consists of coordinate encoding and the cited set-based attention aggregation. Its substantive improvement over existing set-encoding approaches also requires further clarification.

Regarding the two-stage decoder, prior work has established not only continuous velocity decoding but also trajectory reconstruction by estimating movement velocity at successive time steps and integrating it over time. For example, Korik et al. (2019) [1] estimated continuous three-dimensional velocity from EEG and integrated the normalized predicted velocities into relative position trajectories. Thus, the basic velocity-estimation-and-integration principle used in Stage-0 is not itself a new methodological contribution. Likewise, the two-stage approach of initial decoding followed by residual correction is not unique to this paper, as acknowledged in its discussion of related work in Appendix A.3. What the paper needs to establish is the substantive advance offered by its specific combination of chunk-based prediction, neural-conditioned trajectory refinement, and cross-domain adaptation over existing approaches. At present, the design differences and supporting experiments do not sufficiently convince me that the methodological contribution extends beyond the combination and application of existing techniques.

  1. The necessity of the relatively complex design and the benefits of its specific mechanisms are not sufficiently demonstrated
    AMRT combines channel adaptation, multi-horizon encoding, action-chunk decoding, and gated second-stage refinement, but the existing ablations do not sufficiently establish which aspects of this complex design are necessary. In particular, Table 3 shows that the additional benefit of Stage-1 is limited in some settings. On Joystick-Track, adding Stage-1 increases PCC only from 0.770 to 0.778 under the cross-trial mixed-subject protocol and from 0.689 to 0.713 under cross-subject evaluation. Although the improvement is larger on U-Limb—for example, cross-subject PCC increases from 0.369 to 0.430—these results also indicate that the additional benefit of the second stage depends substantially on the evaluation setting.

The reconstruction-strategy comparisons show that adding trajectory refinement can outperform velocity integration alone in some settings. However, they do not sufficiently distinguish whether these gains arise from the proposed gating and accumulated-increment mechanisms or simply from an additional trainable module and trajectory-level supervision. Comparisons against direct position-residual regression, ungated correction, and simpler parameter-matched refinement modules using the same frozen front end are missing. Consequently, the necessity and independent methodological contribution of the specific Stage-1 design remain insufficiently established.

Furthermore, the primary evaluation metric is trajectory PCC, whereas a central claim is the correction of accumulated drift caused by velocity integration. Improved correlation does not necessarily imply reduced positional error: a predicted trajectory can retain a high PCC despite amplitude scaling or a constant offset. Without RMSE, MAE, or a quantitative analysis of positional error over time, the results do not adequately establish whether the proposed refinement mechanism actually mitigates accumulated positional error. Overall, the current evidence does not sufficiently demonstrate that the benefits of the additional stage and its particular architecture substantiate the paper’s central methodological claims.

Questions for authors
  1. What is the independent contribution of channel adaptation beyond conventional channel handling? For EEG recordings with consistent channel labels, Eq. (2) can be interpreted as fixed channel reordering and zero-padding supplemented by learnable linear channel mixing. Could the authors clarify the substantive methodological novelty of this design relative to conventional channel organization combined with a learnable spatial transformation? In addition, which operations are removed in the “AMRT w/o CA” ablation in Table 2? Is there a comparison that retains exactly the same label-based mapping and zero-padding while removing only the learnable residual ΔMs\Delta M_s, to establish its independent contribution to cross-subject generalization?

  2. What is the complete computation performed by the geometry-aware channel adapter? Appendix D.1 states that the four spatial-query outputs are concatenated and projected to 128 dimensions. However, FθspF_{\theta}^{\mathrm{sp}} in Eq. (3) does not explicitly specify how waveform, relative-coordinate, and absolute-coordinate embeddings are fused. Could the authors provide equations or pseudocode covering the complete computation from individual-electrode encoding through spatial-query aggregation to the fixed-dimensional shared representation? Please include the tensor dimensions, the sources of QQ, KK, and VV in the attention operation, and the masking of invalid channels. These details are needed to establish how recordings with different electrode counts are processed by the same model and to reproduce the module.

  3. How are self-attention and cross-attention organized within ChunkDec? Section 2.2.3 describes the action-chunk decoder through the abstract function ChunkDec, and the appendix specifies its hidden dimension and number of attention heads, but the detailed computational path remains unclear. Could the authors specify the number of decoder layers, the order of self-attention and cross-attention, the input to each sublayer, the residual and normalization connections, and the temporal attention masks? In particular, does query self-attention occur before reading the current sample’s neural memory, after doing so, or in alternating layers? This distinction affects how the decoder jointly predicts different output time steps based on the current neural input.

  4. The latency introduced by block-based decoding is not explicitly quantified Section 2.1 describes an online-compatible, block-causal decoding protocol that allows joint access to neural observations within the current output block; although the appendix specifies a 1-second output block and a 0.5-second stride, it does not explicitly state the actual output latency associated with this configuration.

  5. Does Stage-1 actually reduce positional error and integration drift? The paper primarily reports trajectory PCC, which does not fully characterize amplitude errors or offsets in predicted positions. Could the authors report positional RMSE or MAE for Vel-Int and the full AMRT on the same test sets, together with positional error as a function of elapsed trial time? Do these comparisons establish that Stage-1 not only improves correlation with the ground-truth trajectory but also reduces the accumulated positional error emphasized in the paper?

  6. Can more direct comparisons establish the substantive contribution of the two-stage design over existing approaches? Given prior work on reconstructing trajectories by integrating decoded velocities [1], as well as on residual correction following initial decoding, discussed in Appendix A.3, what do the authors consider to be AMRT’s principal methodological innovation? In view of the limited additional gains from Stage-1 in some settings in Table 3, are there comparisons against direct position-residual regression, ungated correction, or simpler parameter-matched refinement modules using the same frozen front end and comparable training budgets? Such results would help determine whether the improvements specifically require the proposed gating and accumulated-increment mechanisms, rather than primarily reflecting additional trainable capacity and trajectory-level supervision.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer DtEG (1/2)

We thank the reviewer for the detailed and constructive comments. We clarify the methodological contributions and provide additional experiments below.

W1/Q1: Generalization-oriented channel adaptation. AMRT's central contribution is not a novel channel-mapping operator, but a unified framework addressing two generalization challenges: heterogeneous neural recordings and task-dependent motion patterns. For neural heterogeneity, the Channel Adapter enables EEG/iEEG recordings with different electrode identities and layouts to share the same encoder and decoder.

We clarify that the original "AMRT w/o CA" removes the entire adapter. For J, channels are zero-padded to 64; for JT-common, the first 48 channels are retained in their original order. Both bypass geometry encoding and set aggregation. We further isolate the learnable residual (EEG) and spatial information (iEEG), retaining the remaining adapter operations.

Cross-subject PCC (↑):

MethodWAY-EEG-GALU-LimbJoystick-TrackJT-common
AMRT w/o CA0.789 ±\pm 0.1030.360 ±\pm 0.1050.646 ±\pm 0.1380.673 ±\pm 0.107
AMRT w/o CA-res/si0.824 ±\pm 0.0680.372 ±\pm 0.1140.680 ±\pm 0.1160.679 ±\pm 0.092
AMRT0.841 ±\pm 0.0550.430 ±\pm 0.0730.713 ±\pm 0.1200.687 ±\pm 0.125

The ablations outperform removing the entire adapter but underperform full AMRT, supporting the contributions of both structural alignment and learnable spatial information.

W2/Q5: Stage-1 effectiveness beyond trajectory correlation. We agree that Stage-1 gains vary across settings and that PCC alone cannot establish improved positional accuracy. We therefore added RMSE evaluations across all reconstruction settings (Appendix E.1.1, Tables 7–9).

Trajectory RMSE (↓):

DatasetMethodctrictri-mixcsub
WAY-EEG-GALVel-Int16.764 ±\pm 13.36612.260 ±\pm 4.50220.814 ±\pm 15.911
AMRT4.047 ±\pm 0.6165.237 ±\pm 1.1595.809 ±\pm 0.565
U-LimbVel-Int—85.979 ±\pm 4.33187.462 ±\pm 16.057
AMRT—80.926 ±\pm 3.76079.106 ±\pm 13.723
Joystick-TrackVel-Int9028.251 ±\pm 1842.1468591.401 ±\pm 1070.73713465.264 ±\pm 1075.413
AMRT7422.698 ±\pm 604.2297692.757 ±\pm 616.4468494.897 ±\pm 1492.847

Stage-1 consistently reduces RMSE relative to velocity integration across all evaluated settings, supporting improved positional reconstruction beyond PCC.

Time-resolved Positional Error Analysis. To further examine whether Stage-1 mitigates accumulated positional errors rather than merely reducing overall RMSE, we analyze positional errors over elapsed trial time. On WAY-EEG-GAL, the relative error reduction increases from 4.25% at 1 s to 57.02% at 5 s. On U-Limb, improvements are limited during the first 5 s (mean trial duration: approximately 12.6 s), but reach 18.31% over the final 20% of the trajectory and 26.09% at the endpoint. On Joystick-Track, improvements are more uniform, with only a 2.40-percentage-point greater reduction in the later segment. These findings provide additional evidence that Stage-1 mitigates accumulated positional errors, particularly on WAY-EEG-GAL and U-Limb.

W2/Q6: Contribution of Gated Accumulated-Increment Refinement. To distinguish architectural benefits from additional parameters and trajectory supervision, we further introduce three controlled refinements using the frozen Stage-0: (i) direct position-residual regression without gating (w/o Gate-Increment), (ii) ungated accumulated-increment correction (w/o Gate), and (iii) a parameter-matched pointwise MLP (Matched-MLP).

ModelU-LimbWAY-EEG-GALJoystick-Track
AMRT0.530 ± 0.0290.837 ± 0.0450.778 ± 0.076
AMRT-Stage00.471 ± 0.0200.785 ± 0.0380.770 ± 0.076
w/o Gate0.502 ± 0.0220.802 ± 0.0410.727 ± 0.036
w/o Gate-Increment0.497 ± 0.0210.793 ± 0.0420.705 ± 0.022
Matched-MLP0.497 ± 0.0310.812 ± 0.0220.697 ± 0.021

The complete additional ablations (Appendix E.3.2, Table 15) show that removing the gate while retaining accumulation reduces PCC across all three datasets. The matched MLP also fails to reproduce the full model's PCC performance. These results support the contribution of structured refinement beyond additional capacity alone.

AuthorsOpenReview ↗

Author Response to Reviewer DtEG (2/2)

Q2/Q3: Implementation details and reproducibility. We have expanded Appendix D.1 with the complete geometry-aware adapter computation, including waveform/coordinate fusion, tensor dimensions, spatial-query aggregation, attention Q/K/V construction, and invalid-channel masking. We also clarify the ChunkDec computation, including decoder depth, attention ordering, residual and normalization connections, and temporal masking.

Q4: Block-based decoding latency. AMRT outputs predictions after each complete 1-s block, with a 0.5-s stride. The estimated buffering delay is 0.5–1.0 s per output time step (0.75 s on average). Batch-size-one replay on an NVIDIA RTX 3090 yields mean inference times of 13.73 ms for WAY-EEG-GAL and 12.97 ms for Joystick-Track, giving estimated average output delays of 0.764 s and 0.763 s, respectively. These estimates exclude acquisition, preprocessing, and actuator/display delays; actual closed-loop latency remains to be evaluated.

W1/Q6: Neuroscience-motivated two-stage decoding for generalization. As discussed in Sec.1, motor cortical encoding of movement direction and speed [1], together with longer-timescale population dynamics [2,3], motivates our proposed Anchor-and-Refine framework, which separates local velocity estimation from trajectory refinement. To clarify why we adopt Velocity-to-Trajectory rather than Trajectory-to-Trajectory or Velocity-to-Velocity refinement, we added two parameter-matched baselines. Trajectory-to-Trajectory shows weaker generalization on cross-subject and cross-dataset paradigms, while Velocity-to-Velocity suffers from substantially larger positional RMSE, reflecting errors accumulated through velocity integration.

Cross-subject generalization (PCC↑):

DatasetAMRTTraj-Traj
WAY-EEG-GAL0.841 ±\pm 0.0550.849 ±\pm 0.054
U-Limb0.430 ±\pm 0.0730.301 ±\pm 0.092
Joystick-Track0.713 ±\pm 0.1200.597 ±\pm 0.114

Cross-dataset generalization (PCC↑):

TransferAdaptationAMRTTraj-Traj
J →\rightarrow UFull-FT0.337 ±\pm 0.1350.123 ±\pm 0.094
J →\rightarrow UStage-1-only0.454 ±\pm 0.0560.195 ±\pm 0.100
W →\rightarrow UFull-FT0.264 ±\pm 0.1360.100 ±\pm 0.053
W →\rightarrow UStage-1-only0.371 ±\pm 0.0720.296 ±\pm 0.048

ctri-mix

DatasetMethodPCC ↑RMSE ↓
Vel-Refine0.845 ± 0.02138.808 ± 3.625
AMRT0.837 ± 0.0455.764 ± 1.35
Vel-Refine0.480 ± 0.035202.908 ± 5.480
AMRT0.530 ± 0.02980.926 ± 3.760
Vel-Refine0.774 ± 0.07615966.974 ± 6047.528
AMRT0.778 ± 0.0767692.757 ± 616.446

Reference

[1] Moran D W, Schwartz A B. Motor cortical representation of speed and direction during reaching[J]. Journal of neurophysiology, 1999, 82(5): 2676-2692.

[2] Churchland M M, Cunningham J P, Kaufman M T, et al. Cortical preparatory activity: representation of movement or first cog in a dynamical machine?[J]. Neuron, 2010, 68(3): 387-400.

[3] Kaufman M T, Churchland M M, Ryu S I, et al. Cortical activity in the null space: permitting preparation without movement[J]. Nature neuroscience, 2014, 17(3): 440-448.

Thank you for the response and additional experiments. The implementation details, channel-adapter ablations, RMSE results, and Stage-1 comparisons address several of my previous questions. After reading the revised manuscript, my main concern remains the limited methodological novelty: the contribution primarily consists of combining and applying existing techniques. I have three specific concerns.

  1. The new contribution of the channel adapter remains unclear.

    The EEG branch uses fixed channel mapping with a learnable linear residual, while the variable-layout branch uses coordinate encoding and set-based attention aggregation. The additional ablations show that these components improve performance, but my concern is their specific distinction from existing spatial transformations and set encoders. The authors also explicitly state that the contribution is not a new channel-mapping operator.

    What new technical design does the channel adapter introduce? If this component primarily integrates existing methods, what previously unresolved technical problem does that integration solve?

  2. The relationship between the two-stage design and existing residual-correction methods remains unclear.

    Decoding velocity and integrating it into trajectories, as well as applying residual corrections to initial predictions, have been explored in prior work. AMRT’s specific design choices center on neural-feature conditioning, gating, and the accumulation of correction increments. Table 15 provides internal comparisons supporting these choices within the framework, but the central distinction from the closest existing methods remains vague.

    In another response, the authors acknowledge that they have not compared AMRT with Li et al. (2026a). I would like the authors to clarify the architectural and training differences between these methods, beyond explaining why they chose velocity-to-trajectory prediction.

    Please compare AMRT with this related work in terms of conditioning inputs, the quantity being corrected, the correction mechanism, and the training objectives, and identify the principal innovation. Which additional experiment directly demonstrates the benefit of that innovation? Also, how are residual supervision, smoothness loss, and gate-utility loss handled in each variant in Table 15?

  3. The scope of the empirical contribution needs to be clearer, particularly the gains over simpler alternatives.

    The additional results show that the advantage of the two-stage design depends substantially on the evaluation setting. In Table 9, direct position prediction achieves lower RMSE across all three WAY protocols. AMRT’s advantages are more evident on U-Limb and in cross-subject evaluation on Joystick-Track.

    Table 17 also shows that a template using only trial onset and the average training trajectory already achieves PCC close to AMRT on WAY and Joystick-Track. For example, the cross-subject results are 0.82 versus 0.84 on WAY and 0.70 versus 0.71 on Joystick-Track. These findings raise a concrete question about how much the more complex model adds to the prediction of trial-specific movement variations.

    How do the authors explain the differences in the two-stage design’s performance across settings? Does its advantage primarily come from handling more complex movement patterns, improving cross-subject transfer, or correcting positional errors? In settings where the template already performs strongly, can the model predict individual trials’ deviations from the average trajectory?

I still view this work primarily as a combination and targeted refinement of existing methods. The additional experiments clarify the rationale for the design, but my assessment of its novelty and contribution remains unchanged. I therefore maintain my original recommendation.

Reviewer_UtxgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes AMRT, a framework for continuous neural decoding across subjects and heterogeneous recording configurations. It combines a channel adapter with a two-stage decoding approach: the first stage predicts movement velocity and integrates it into a coarse trajectory, while the second stage uses neural representations and the coarse trajectory to predict residual corrections. The framework is evaluated on cross-trial, cross-subject, and cross-dataset generalization across EEG and ECoG recordings.

Critical strengths
  • Addresses an important generalization problem. The framework accommodates differences in electrode counts and layouts, making it applicable beyond a single subject or fixed recording configuration.
  • Provides a structured decoding approach. Separating velocity estimation from trajectory refinement is well motivated, and the ablations show that refinement improves over velocity integration alone, particularly under heterogeneous decoding settings.
  • Includes extensive evaluation. The experiments cover multiple datasets, recording modalities, adaptation budgets, and architectural variants. Reporting negative transfer and cases where fine-tuning hurts performance provides a useful picture of the method’s limitations.
Critical weaknesses

Evaluation relies primarily on trajectory correlation. PCC does not adequately characterize errors in trajectory amplitude or spatial accuracy. Position-error metrics and online control evaluation would strengthen the practical claims.

Cross-dataset transfer is strongly directional. While Joystick-Track→U-Limb transfer is promising, transfer from the EEG datasets to Joystick-Track does not yield consistent improvements.

Questions for authors
  1. why the domain transfer is directional, it is easier to transfer from ECoG to EEG, but not the ofhter case.
  2. How does AMRT compare with an end-to-end residual trajectory decoder of comparable parameter count, and with a two-stage model that corrects velocity before integration rather than correcting the trajectory directly?
  3. Can authors report other spatial metrics such as RMSE?
1 comment
AuthorsOpenReview ↗

Author Response to Reviewer Utxg

We thank the reviewer for the constructive comments.

W1/Q3: Additional spatial metrics. We agree that PCC alone cannot fully characterize trajectory amplitude and positional accuracy. We therefore added RMSE results in the appendix E.1.1 to complement PCC. AMRT adopts a block-causal decoding protocol (Sec. 2.1), using only neural observations available up to the current block endpoint, which support simulated online decoding, while closed-loop control remains future work.

W2/Q1: Directional cross-dataset transfer. The observed asymmetry may reflect differences in neural sampling and task compatibility. Scalp EEG captures spatially mixed activity from distributed cortical sources, whereas ECoG provides more localized measurements with subject-specific cortical coverage [1]. Thus, local motion-related features learned from ECoG may remain accessible through EEG, while distributed spatial patterns learned from EEG may not be preserved in locally sampled ECoG recordings. Task differences may also contribute: our additional trajectory-template transfer analysis yields PCCs of −0.161 ± 0.021 for J→U and −0.406 ± 0.095 for U→J, suggesting greater difficulty in the reverse direction.

Q2: Why staged optimization and trajectory-space refinement? AMRT separates local velocity estimation from trajectory refinement, motivated by the relationship between motor cortical activity and movement direction and speed (Sec. 1). Freezing Stage-0 preserves its learned neural representations and local motion predictions during refinement, while enabling restricted Stage-1-only adaptation with limited target data (Sec. 3.2.2). Unlike velocity refinement, Stage-1 directly optimizes trajectory-level corrections to address errors accumulated through velocity integration.

We added two parameter-comparable baselines under the ctri-mix protocol: an end-to-end residual trajectory decoder (End-to-End) and a two-stage velocity-correction model (Vel-Refine).

DatasetMethodPCC ↑RMSE ↓
WAY-EEG-GALEnd-to-End0.798 ± 0.0416.951 ± 0.458
Vel-Refine0.845 ± 0.02138.808 ± 3.625
AMRT0.837 ± 0.0455.237 ± 1.59
U-LimbEnd-to-End0.369 ± 0.01892.958 ± 3.726
Vel-Refine0.480 ± 0.035202.908 ± 5.480
AMRT0.530 ± 0.02980.926 ± 3.760
Joystick-TrackEnd-to-End0.727 ± 0.0107920.285 ± 302.610
Vel-Refine0.774 ± 0.07615966.974 ± 6047.528
AMRT0.778 ± 0.0767692.757 ± 616.446

AMRT outperforms End-to-End across all three datasets, supporting staged optimization. Compared with Vel-Refine, AMRT achieves substantially lower RMSE.

Reference

[1] Dubey, A., & Ray, S. (2019). Cortical electrocorticogram (ECoG) is a local signal. Journal of Neuroscience, 39(22), 4299–4311.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.