acceptodds
Under review as a conference paper at ICLR 2027

Empowering Brain-Computer Interface Decoding with Large Language Models

Abstract

Decoding brain signals supports a broad range of brain–computer interface (BCI) applications spanning cognitive, motor, and clinical domains. These tasks vary substantially in their targeted brain states, neural patterns, and output spaces. Despite recent advances in brain foundation models, the field still relies on disjointed prediction heads and task-specific adaptation pipelines, limiting unified knowledge sharing and robust generalization across tasks. To overcome these limitations, we introduce UniMind, a unified multi-task brain decoding framework driven by large language models (LLMs). UniMind reformulates heterogeneous task outputs into a unified question–answer paradigm, casting diverse neural signals into a universal semantic manifold optimized via a single shared autoregressive objective. To reconcile task heterogeneity within a single model, we design a task-aware query selector to dynamically extract task-relevant representations. Furthermore, a neuro-language connector is proposed to distill high-dimensional spatiotemporal brain patterns into compact neural-semantic representations directly interpretable by LLMs. Across ten scalp EEG datasets spanning five task domains, UniMind achieves an average improvement of 8.94 percentage points in balanced accuracy. It shows strong cross-subject performance and generalizes to unseen datasets. Beyond scalp EEG, we show that UniMind also benefits intracortical spike decoding in non-human primates, raising average and producing more separable spike representations. These results establish UniMind as a scalable framework for unified neural decoding across diverse subjects, tasks, and recording modalities.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain-computer Interfaces, Brain Foundation Model, Large Language Model, Multi-task Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4PmgOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents UniMind, a unified multi-task brain decoding framework that reformulates heterogeneous EEG decoding tasks as multimodal-language question-answering problems. The proposed framework integrates a pretrained EEG encoder with a Task-aware Query Selector (TQS), a Neuro-Language Connector (NLC), and an LLM-based autoregressive decoder. The TQS dynamically selects task-relevant spatial and temporal queries, while the NLC extracts and projects neural representations into the LLM embedding space.

The authors evaluate UniMind on ten EEG datasets spanning five task domains, reporting an average improvement of 8.94 percentage points in balanced accuracy over NeuroLM. Additional experiments demonstrate cross-dataset generalization on an unseen sleep dataset and explore the transfer of EEG-adapted language representations to intracortical spike decoding.

Critical strengths
  1. The paper addresses an important limitation of existing EEG foundation models: the reliance on separately optimized task-specific prediction heads. The proposed framework provides a coherent approach to heterogeneous neural decoding, supported by experiments across ten datasets and five task domains.
  2. Convincing empirical performance. UniMind consistently outperforms NeuroLM across all ten datasets and achieves competitive performance against task-specific methods. The comparison between individual and joint multi-task training further supports the effectiveness of shared representations across different EEG tasks.
  3. Reasonably comprehensive ablation studies. The authors investigate the contributions of TQS and NLC, the effects of different query configurations, and the comparison with contrastive EEG-text alignment. These experiments provide useful evidence supporting the architectural design choices.
  4. Encouraging generalization results. The evaluation on the unseen SHHS dataset provides additional evidence for cross-dataset transfer. The extension to intracortical decoding is also an interesting attempt to investigate whether the learned representation space can benefit other neural recording modalities.
Critical weaknesses
  1. The evidence for cross-modality transfer is still preliminary. The intracortical experiments are conducted using recordings from a single non-human primate, and the improvements are not consistent across the two tasks. I know that non-human data is hard to access. In particular, performance decreases on center-out reaching. Furthermore, the auxiliary language modeling loss predicts the reaching task category, so it would be useful to compare against a conventional auxiliary task-classification objective to better isolate the contribution of the EEG-adapted LLM representation space.
  2. Missing Visual Language Model (VLM) Baselines. The paper evaluates UniMind on several clinically relevant EEG benchmarks, including TUEV, TUSL, and TUAB, where discriminative information is often directly observable from conventional EEG waveform visualizations. However, the experimental comparison does not include VLM-based approaches that interpret EEG signals through rendered waveform images. Given the recent progress of VLMs in visual pattern recognition, it would be valuable to compare UniMind against a strong pretrained VLM (e.g., Qwen3-VL) using standardized multichannel EEG visualizations under the same evaluation protocols. Such a comparison would help determine whether the proposed neural-language alignment framework offers advantages over directly leveraging the visual understanding capabilities of existing multimodal LLMs [1].

[1] Pan, J., Wang, Y., Zhang, E., Lu, B., Zheng, W., & Li, D. (2026). Visualizing the Invisible: Generative Visual Grounding Empowers Universal EEG Understanding in MLLMs. ArXiv, abs/2605.18172.

Questions for authors

See weaknesses. Problem2 is not a neccessity but I am glad to see a comparsion on at least TUEV.

Reviewer_Qrq3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes UniMind, a unified multi-task EEG decoding framework that reformulates heterogeneous scalp EEG tasks as natural-language question-answer problems and decodes them with a shared autoregressive LLM objective. The method introduces a Task-aware Query Selector (TQS) that dynamically selects spatial and temporal queries, and a Neuro-Language Connector (NLC) that maps task-relevant spatiotemporal EEG patterns into the LLM input space.

UniMind is evaluated on ten EEG datasets across five task domains, reporting an average balanced-accuracy gain of 8.94 points over NeuroLM, plus cross-dataset generalization on SHHS and an extension to intracortical spike decoding. Main contributions are the unified generative formulation, the TQS/NLC architecture, and empirical evidence of multi-task knowledge sharing and cross-dataset transfer.

Critical strengths
  1. The unified multi-task formulation is practically meaningful: it removes task-specific prediction heads and achieves consistent average gains across ten EEG datasets. This directly supports the central claim of unified decoding.

  2. The validation is reasonably comprehensive, including multi-task and task-specific baselines, cross-dataset SHHS evaluation, component ablations, and an intracortical decoding extension. These together strengthen confidence in the framework.

Critical weaknesses
  1. Fairness of comparison and generality remain insufficiently established. Task-specific baselines are tuned per dataset while UniMind is a single unified model; although the authors acknowledge this, the claimed generalization advantage may still be affected by training-set size, task mixing, and hyperparameter selection.

  2. Some key design choices lack sufficient explanation. TQS routing is mostly supported by qualitative visualization, the static query is disabled in the SHHS experiment, and the intracortical results show a clear CO-task R² drop but a large RT-task gain. These issues affect assessment of method stability and interpretability.

Questions for authors
  1. In the SHHS zero-shot evaluation, was any target-domain data used for model selection or hyperparameter tuning? Were the TQS query pools and routers trained only on the source domains?

  2. How do the authors explain the CO-task R² decrease and the large RT-task improvement in Table 4? Is this related to task difficulty, trial count, or the λ setting?

  3. The “Baseline GRU + LM Loss” variant diverged to NaN. Does this suggest that generic LLM regularization is unstable for intracortical decoding? What specifically makes UniMind Loss stable?

  4. The 4B model does not outperform the 1.7B model on some datasets such as SHU and TUEV. Do the authors view this as task-dependent scaling behavior or possible negative transfer?

Reviewer_HjgmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents UniMind, an LLM-based framework that formulates heterogeneous EEG classification tasks as natural-language question answering. It uses a pretrained LaBraM EEG encoder, a Task-aware Query Selector to select input-dependent spatial and temporal queries, and a Neuro-Language Connector to map neural representations into the input space of Qwen3. The framework is trained jointly across tasks with LoRA adaptation of the LLM. Evaluation covers ten EEG datasets spanning five task domains and transfer to an unseen sleep-staging dataset. The paper also explores intracortical decoding by using the frozen, EEG-adapted LLM to provide an auxiliary language-modeling objective for a GRU that predicts movement velocity from macaque spike recordings.

Critical strengths

Broad multi-task evaluation with clear improvements over an existing multi-task approach. Across ten EEG datasets, UniMind-1.7B improves average balanced accuracy over NeuroLM by 8.94 percentage points. Handling heterogeneous tasks with one shared model is practically meaningful, even though it does not outperform separately optimized task-specific models on every dataset.

Critical weaknesses

1.The technical advance and necessity of the proposed components are insufficiently established. The method builds on an existing LaBraM encoder, query routing, cross-attention connectors, and LLM instruction tuning. The paper does not sufficiently establish why the proposed spatial/temporal query design offers a distinctive advantage over simpler connectors, conventional routing mechanisms, or multi-task models without an LLM.

In Table 3, NLC increases average balanced accuracy from 64.36% to 66.14%, and TQS further increases it to 67.06%. These gains are real in the reported results but unevenly distributed: the full model improves over the baseline by only 0.30 and 0.11 percentage points on SEEDIV and TUAB, respectively, and slightly decreases performance on SEEDV. Adding TQS reduces TUEV performance from 64.98% to 62.44%. The table does not report uncertainty for these ablation differences. Meanwhile, Figure 3 shows relatively similar overall performance between UniMind and CBraMod. Although a unified multi-task model and separately finetuned models offer different benefits, the current evidence does not adequately disentangle the contributions of joint training, model scale, language pretraining, and the proposed modules. Consequently, the technical novelty and justification of the design remain insufficiently supported.

2.The intracortical experiments do not adequately establish that the gains arise from the learned neural–language space. In Table 4, UniMind Loss improves RT (R^2) from 0.4759 to 0.6607 but reduces CO (R^2) from 0.7505 to 0.7029. Thus, the results demonstrate task-dependent benefits rather than consistent improvement. The evaluation involves only two tasks from one macaque, and the table does not report uncertainty across repeated runs.

More importantly, Appendix F.2 specifies that the auxiliary language-modeling objective predicts the CO/RT task label. The improvement could therefore arise from additional task-label supervision or generic multi-task regularization, rather than the EEG-adapted language space specifically. A direct comparison against a conventional auxiliary classifier trained with the same CO/RT labels is needed. Furthermore, the generic pretrained-LLM baseline diverges to NaN. This demonstrates an optimization failure under the tested configuration, but does not by itself establish that the model lacks useful representations or regularization capacity. These limitations weaken the claimed transfer across recording modalities.

Questions for authors

1.Architecture and optimization clarification: How are gradients propagated to the router through the Top-K selection in Equation (1)? Are static queries learned separately for each task or dataset, or shared globally? Please specify the broadcasting, reshaping, and concatenation operations in Equations (2)–(4).

2.Evidence for the component and LLM contributions: Under the same LaBraM initialization, training data, and optimization settings, can the authors provide repeated-run results for Table 3 and comparisons against both an LLM model with a simple shared connector and a non-LLM multi-task model with conventional task heads? These controls would directly affect my assessment of the necessity and technical contribution of the proposed modules.

3.Source of the intracortical gains: How does a conventional auxiliary CO/RT classification head perform when added to the same GRU and velocity-regression objective? Are the RT improvement and CO degradation in Table 4 consistent across random seeds and held-out sessions?

4.Clarification of the failed baseline: What causes the generic pretrained-LLM baseline to produce NaNs? Were loss weighting, learning rate, numerical precision, and gradient stability examined to ensure that this baseline was reasonably optimized?

Comments on the paper

1 comment

Thank you for your additional experiments. I'm not sure why the other reviewers didn't follow up, but I'd like to express my concerns. Essentially, I am quite skeptical as to why EEG-LLM uses auxiliary loss rather than model weights (distillation learning or fine-tuning) to assist in spike decoding? Just using an auxiliary loss method is too indirect. Could it lack scientificity? I still believe that the conclusion of this transfer decoding task is not very reliable. The amount of data in the downstream dataset is too small, which makes it prone to overfitting. Moreover, I think "our claim that the EEG-adapted LLM provides auxiliary supervision for spike decoding does not depend on a specific data format," if this supervision does not depend on a specific data format, then it actually requires evidence to support it, rather than simply drawing this conclusion based on a downstream dataset. I suggest that the authors do not consider this transfer decoding as the main contribution, unless it has been fully verified.

In addition, the multi-task performance is good. It is necessary to check whether the parameter quantities are matched. If it is only because the number of trainable parameters is larger during multi-task fine-tuning, then the performance comparison of multi-task is actually unfair.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.