acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Latent Factorization for Non-invasive Sequential Motor Decoding

Abstract

Non-invasive neural decoding is challenged by context-dependent representation drift: during sequential movements, the same finger movement can produce different neural patterns across participants, sequences, and serial positions. These sources of variation operate over different temporal scales, and mix nonlinearly, hindering stable decoding. We formulate this setting as primary-task prediction from mixed, context-dependent observations and propose HiLa, a hierarchically latent factorization framework for stable decoding during sequential movements. HiLa uses hierarchical labels as supervisory signals and conditional branch splitting to learn context-robust representations that preserve action identity across contexts. HiLa achieves 62.02% movement identity accuracy during sequential finger tapping and 50.44% keystroke accuracy during typing. HiLa's backbone outperforms benchmarks, while hierarchical supervision improves accuracy, with gains remaining stable in cross-participant and cross-context tests. Latent factorization yields more decontextualized and biologically plausible representations without sacrificing decoding accuracy. Together, these results show that hierarchically supervised latent factorization can improve primary-task decoding under structured contextual variation in non-invasive neural recordings.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Hierarchical latent factorization, Non-invasive neural decoding, Context-robust representation learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_BJTFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces HiLa to leverage multiple levels of behavior labels to guide representation learning in sequential decoding tasks with MEG, with improved performance over baselines on two sequential decoding dataset. HiLa showed generalization performance to held-out data while finetuning only the participant specific parameters, and analysis on gradient showed biologically plausible pattern.

Critical strengths
  • Propose to learn on multiple level of behavior labels in the sequential tasks
  • Show improved performance on two datasets
  • Detailed experiments on ablation study and representation analysis
Critical weaknesses
  • The method’s usability is limited: for sequential task, online setting is not evaluated and long context is not leveraged; the labels used is constrained to sequential tasks hence unlikely to be generic
  • I find the model architecture very complicated and seems arbitrary (multiple mixer and multi-scale convolution with 5 specific k values)
  • Latent disentanglement is manually forced rather than emerged
  • Code is not provided
Questions for authors
  1. Regarding the tasks, can the author elaborate on why the position would help the performance especially when window size is small? Is there only one or multiple words or finger press in each window?
  2. Are there statistical test results in Table 2? What are the results of [y][c] and [y][e]?
  3. Please separate methods and related works, right now it is highly mixed
  4. Can the authors describe roughly the roles of each of those multiple mixer and multi-scale convolution. Are the choices of the 5 K empirically or theoretically?
  5. Why frame the motivation as generic non-invasive recording (as shown in title, abstract and introduction) while actually running on MEG? Is the method adoptable to EEG / FMRI?
  6. For cross-subject / cross-sentence experiment, could the author provide curve when trained from scratch and finetuning with competitive baselines
  7. In ablation study (Figure 3) what are the values for each variant, the first 3 or 4 variants looks similar.
  8. Why there are no recent EEG foundation models baselines in Table 1 (Labram or REVE) which can be more competitive?
Reviewer_x1emOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on MEG-based sequential typing by formulating sequential motor decoding as a primary-task prediction problem from context-dependent mixed observations. To this end, the authors propose HiLa, a hierarchical latent factorization framework that learns representations from MEG signals and decodes individual movements. They evaluate HiLa on sequential finger-tapping and keystroke-typing tasks, showing improved finger and character prediction accuracy over baselines in cross-participant and cross-context settings. They also show that HiLa learns biologically plausible representations from MEG signals that are consistent with domain knowledge of sequential movements.

Critical strengths

Extensive experimental evaluation: The evaluation includes strong performance on target movement decoding, generalization to held-out participants, an ablation study of the model components, and an interpretability analysis showing that HiLa captures biologically plausible sequential patterns. Together, these results give solid evidence that the proposed framework is effective, generalizes across individuals, and learns meaningful neural representations.

Critical weaknesses

Clarity of the proposed work: Although the motivation and high-level explanation are strong, it is not easy to understand how HiLa works due to missing definitions and explanations in the methods section, such as the label definitions and the function and variable definitions in lines 158–160. The section lists the layers and blocks used, but the incomplete definitions make it hard to follow, for example, how participant behavior is injected into the model. Details of the loss function are also missing. If these are application-specific, this should be highlighted. Without these details, the method is difficult to understand, verify.

Questions for authors
  • How is participant information used in the participant adapter module?

  • What is the difference between [y+c+e] and [y], [c], [e] in Table 2? If a label is included in supervision, is its corresponding loss term used? If so, is this effectively an ablation of the loss terms on the validation set?

As detailed in the weaknesses section, clearer variable and function definitions would make the model details, design choices, and motivations easier to understand. The methods section could also be further supported by adding the input and output details of each block to Figure 2.

Reviewer_EsgVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the HiLa model, which utilizes hierarchical supervision and latent variable branch decomposition to enhance the decoding capability of non-invasive magnetoencephalography (MEG) for continuous finger movements and keyboard inputs.

Critical strengths
  1. The research problem is clearly defined and has practical application value.

  2. The model's overall decoding performance is competitive.

  3. The experimental comparison is comprehensive.

Critical weaknesses
  1. The methodological innovation is relatively limited, as the model is largely composed of existing techniques. Furthermore, the paper does not sufficiently explain why this branch structure can effectively resolve the non-linear mixing problem in complex neural signals.

  2. The performance improvement brought by the core method is minimal. Although the full model achieves competitive results, the actual gains attributable to hierarchical supervision and branch decomposition are limited. The paper's most notable accuracy advantage appears to come primarily from the encoder, rather than from the latent variable decomposition mechanism emphasized in the title.

  3. The authors employ t-SNE visualization and mutual information analysis to demonstrate that the different branches exhibit a certain degree of specialization. However, since each branch is directly supervised by its corresponding label, the emergence of label-related clustering is expected.

  4. The paper contains substantial experimental content, resulting in a relatively dense main text. Some figures and surrounding text appear crowded, and additional spacing or layout adjustments would improve readability.

Questions for authors
  1. Currently, the results show that three branches decomposition does not significantly improve decoding accuracy. It needs to be demonstrated whether this structure indeed improves the consistency of action representation, reduces contextual interference, or provides benefits under more challenging generalization settings.

  2. The introduction mentions that "the input–target relationship is intrinsically non-stationary." Could this aspect be discussed in greater depth? Specifically, how stable are repeated actions from the same subject?

Reviewer_eNJ7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The manuscript proposes HiLa, a hierarchical latent factorization framework for non-invasive sequential motor decoding from MEG recordings. It combines a participant-specific adapter, a multiscale convolutional and axial-attention encoder, and task-specific latent branches supervised by movement identity, serial position, and behavioral context. Experiments on sequential finger tapping and sentence typing demonstrate improvements over several adapted EEG/MEG decoding baselines. Additional analyses examine hierarchical supervision, cross-participant adaptation, latent specialization, and sensor-level contributions. While the encoder achieves encouraging results, the independent contribution of hierarchical factorization remains less clear.

Critical strengths

see Questions For Authors

Critical weaknesses

see Questions For Authors

Questions for authors

The paper proposes HiLa for context-dependent MEG motor decoding, combining a convolution/axial-attention encoder, participant adapters, and latent branches supervised by movement, serial position, and context. The two-dataset evaluation, component ablations, held-out tests, and sensor analyses are useful. The encoder's benchmark improvements are encouraging. However, several questions remain about the central factorization claim.

  1. Novelty of Latent Factorization

Section 3.2 partitions the posterior mean into three blocks trained with separate prediction losses. It is unclear what makes this an identifiable or conditional factorization rather than conventional multi-task learning with task-specific branches. What mechanism explicitly separates the underlying factors? A parameter-matched multi-task baseline with shared and separate heads would help isolate the contribution beyond auxiliary supervision.

  1. Limited Gains from Hierarchical Supervision

In Table 1, target-only HiLa achieves 60.42%/49.60%, versus 62.02%/50.44% for the full model. Table 2 suggests most of the improvement comes from serial-position supervision. On SpanishBCBL, the three-branch model (50.44%) is below the one-branch joint-supervision model (50.60%) and target-plus-position model (50.59%). Thus, branch splitting does not improve target accuracy there. Please quantify the significance and uncertainty of each incremental gain and explain the added value of context supervision, especially for typing.

  1. Fairness of Baseline Comparisons

The encoder substantially outperforms adapted EEGNet, ATCNet, TCFormer, and Brain2Qwerty, which is promising. Yet Appendix B.2 uses published default hyperparameters for Brain2Qwerty while tuning other baselines with Optuna, and removes Brain2Qwerty's original sentence-level decoder. Were comparable tuning budgets, participant adapters, and parameter-matched alternatives considered? How much improvement comes from the backbone and adaptation rather than hierarchical supervision?

  1. Strength of the Disentanglement Evidence

Figure 5 shows suggestive latent organization, but t-SNE is qualitative. Appendix B.7 averages per-dimension MI estimates rather than estimating joint multivariate MI; comparing a finger branch to all concatenated branches also changes dimensionality. Appendix B.1 further states that sequence identity is coupled to participant identity. Please provide dimension-matched probing and within-movement, cross-context analyses to distinguish branch specialization from genuine context robustness or source separation.

  1. Event-Aligned Windows Versus Continuous Decoding

Neuron2019 uses EMG-defined press intervals, while SpanishBCBL extracts windows around known keystrokes (-200 to +300 ms). This establishes onset-aligned event classification, not detection of unknown events from continuous MEG; the typing window also includes post-onset information. How are event boundaries obtained at inference without behavioral annotations? Please evaluate a causal/continuous-stream setting or qualify the practical decoding claims.

  1. Class Imbalance and Statistical Validity

Appendix B.1 reports 333,139 non-press windows among 409,551 Neuron2019 windows, with class-balanced sampling during training; Appendix B.5 also balances evaluation windows for adaptation. Is the main accuracy balanced or measured at natural prevalence? Event-level recall and false positives would aid interpretation. Given the 500-ms windows with 40-ms stride and reuse of participants across folds/seeds, please provide uncertainty estimates that treat trials or participants as independent units, rather than only fold/seed-level tests.

  1. Generalization and Biological Claims

Figure 4 tests unseen participants after adaptation using labeled trials per sequence, not zero-shot decoding. Please add unadapted results and calibration-matched baselines. Figure 6 provides useful sensor sensitivity diagnostics, but gradients and Gaussian-noise ablations alone do not identify cortical sources or exclude correlated non-neural signals. The biological interpretation should be appropriately bounded.

Overall, HiLa offers encouraging decoding performance, but the independent methodological contribution of latent factorization, beyond the stronger encoder and auxiliary serial-position supervision, is not yet fully established.

Comments on the paper

1 comment

Dear Reviewers,

Please take a moment to review the authors’ rebuttal, assess whether your concerns have been adequately addressed, and actively participate in the discussion. Please update your assessment if needed. Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.