acceptodds
Under review as a conference paper at ICLR 2027

Visualizing the Invisible: Generative Visual Grounding Empowers Universal EEG Understanding in MLLMs

Abstract

Leveraging the universal representations of pre-trained Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) has emerged as a promising paradigm for enhancing the capability of brain foundation models. Constrained by the severe scarcity of visually-evoked electroencephalography (EEG) datasets, existing foundation models predominantly focus on LLMs and compromise by aligning neural signals solely with abstract text—a lossy translation that inevitably discards the fine-grained, perceptual details encoded in brain activity. In this work, we propose Generative Visual Grounding (GVG), a framework that visualizes the invisible. Rather than forcing neural signals into text, we employ an EEG-to-Image generative model as a "visual translator" to hallucinate instance-specific proxy images for non-visual EEG. This strategy provides structured visual contexts that allow MLLMs to apply their visual priors to interpret clinical states. To validate this core hypothesis, we first establish image-only alignment with GVG-X-Omni and GVG-Janus. GVG-X-Omni approaches the average performance of the 1.7B-parameter NeuroLM-XL while tuning 170M parameters over a frozen 7B backbone. Image-only GVG-Janus improves over text-only alignment on all six primary tasks, raising average balanced accuracy from 45.89% to 53.46% and surpassing NeuroLM-XL on four tasks. Building on this result, trimodal EEG–image–text alignment further raises GVG-Janus's average to 60.05% under full adaptation, the highest among the compared multi-task models. Both low-rank adaptation (LoRA) and full adaptation exceed NeuroLM-XL on all six tasks. Predicted native visual tokens also support EEG-based reconstruction through frozen decoders, while clinical analyses identify physiological variation retained in aligned EEG features. These results support visual proxy grounding as an effective complement to textual alignment for universal EEG understanding.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG, EEG Foundation, MLLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_3rmyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Generative Visual Grounding (GVG), which connects EEG signals to pretrained multimodal language models through visual representations. For recordings without visual stimuli, an adapted EEG-to-image teacher generates training-only proxy images. The framework aligns EEG with image/text features, predicts native image tokens, and combines these tokens with continuous EEG features for task prediction. Evaluation covers six jointly trained EEG tasks and two additional tasks with supervised adaptation. Full GVG-Janus achieves 60.05% average balanced accuracy on the six primary tasks, compared with 51.13% for NeuroLM-XL. Ablations, proxy-image controls, reconstruction experiments, and physiological probes provide additional validation.

Critical strengths
  1. Training-only proxy images make it possible to reuse a pretrained visual interface for clinical recordings, while inference requires only EEG and a task prompt. This is a meaningful methodological contribution beyond the usual EEG-to-text alignment.

  2. Full and LoRA GVG-Janus exceed NeuroLM-XL on all six primary tasks. Comparisons against continuous EEG inputs, class-prototype images, shuffled proxies, and a length-matched prefix help establish that the visual pathway contributes to performance. These controls make the central empirical finding more convincing than an improvement over external baselines alone.

Critical weaknesses
  1. The physiological meaning of instance-specific clinical proxies remains uncertain. In Table 15, class-prototype images slightly outperform EEG-conditioned proxies on HMC and TUAB; the clinical advantage of instance-specific proxies is concentrated in TUEV. Moreover, the physiological probes evaluate Full against a stage-only baseline, rather than against the original LaBraM features or class-prototype alignment. The results support useful training supervision, but do not establish that the generated images preserve additional physiological information. This limits the stronger explanation offered for why clinical visual grounding works.

  2. The evidence for broad generalization and comparative advantage is narrower than the framing suggests. Emotion datasets use subject-dependent splits; TUSL and Workload require supervised adaptation, and TUSL includes subject overlap. Most external baseline cells are inherited from published reports rather than reproduced under a common protocol. These choices do not invalidate the results, but leave uncertainty about performance on unseen users and the size of the advantage under fully matched evaluation.

Questions for authors
  1. Given Table 15, should the claim about the benefit of instance-specific clinical proxies be limited primarily to TUEV? What explains their lack of improvement over class prototypes on HMC and TUAB?

  2. Do the same physiological probes show an improvement over frozen LaBraM features and class-prototype alignment, using identical subjects and probe settings? This would help distinguish preservation of existing EEG information from a benefit specifically introduced by instance-level visual grounding.

5 comments
AuthorsOpenReview ↗

Response to Q1/W1

Thank you for recognizing the contribution of training-only visual proxies and the evidence from the internal controls. Our analyses connect clinical category organization, correspondence between individual EEG recordings and their visual targets, and physiological variation retained in the aligned EEG features. The additional probes clarify how these properties coexist in the learned interface.

Q1/W1: Instance-specific proxies versus class prototypes

We agree that the additional clinical classification gain over class prototypes is concentrated in TUEV. Full and class prototypes perform similarly on HMC and TUAB, with Full lower by 0.17 and 0.81 percentage points, respectively, whereas EEG-conditioned proxies improve TUEV by 14.72 points. Table 15 gives the clinical results (BAcc, %):

Clinical datasetFullClass prototypeFull − prototype (pp)Within-class shuffleShuffle − Full (pp)
HMC71.5171.68−0.1751.43−20.08
TUAB80.0880.89−0.8176.71−3.37
TUEV60.2045.48+14.7219.63−40.57

We interpret this pattern through the recording-level information needed for each task. A fixed prototype makes the visual supervision target constant within a class, while the input EEG features remain specific to each recording. Richer targets therefore add useful supervision when their within-class differences help distinguish the task's categories.

HMC sleep stages are characterized by typical rhythms and events, while physiological variation within a stage need not change its label. A stable class-level target can therefore support stage discrimination without representing every within-stage difference. Correct pairing still matters when recording-dependent targets are used: within-class shuffling preserves category identity and the proxy image pool but changes EEG assignments, lowering HMC from 71.51% to 51.43% (−20.08 points). Unlike a fixed class anchor, this introduces recording-dependent targets associated with a different EEG sample. Full also improves class-centered EEG–proxy residual correspondence relative to w/o Stage 1 (Δ = 0.0226, significant after Holm correction; Section 5.4). These results support the presence and pairing relevance of within-class variation, even though it does not yield an additional staging gain over the stable prototype target.

TUAB asks the broader normal-versus-abnormal question, with different abnormal manifestations sharing the same label. Consistent category anchors may already provide effective supervision for this decision, while finer target variation provides no additional classification gain in this comparison. Scores are closely grouped across Text-only (80.11%), image-only (80.27%), class prototypes (80.89%), Full (80.08%), and specialist LaBraM (81.40%). Within-class shuffling lowers Full to 76.71% (−3.37 points), a smaller drop than on HMC or TUEV. TUAB likewise shows no significant residual-correspondence improvement relative to w/o Stage 1 (Δ = −0.0037, p = .232).

TUEV distinguishes spike/sharp waves, generalized and lateralized periodic discharges, eye movements, artifacts, and background activity. These distinctions involve event morphology and spatial distribution, making recording-dependent target differences potentially more useful for classification. One fixed image per class does not provide those target differences. Here, EEG-conditioned proxies improve classification from 45.48% to 60.20%, while within-class shuffling reduces it to 19.63% (−40.57 points). Full also improves class-centered residual correspondence relative to w/o Stage 1 (Δ = 0.0335, significant after Holm correction). The larger prototype gap and stronger shuffling effect support the greater practical value of recording-specific targets for TUEV's event discrimination.

AuthorsOpenReview ↗

Response to Q2/W1/W2

Q2/W1: Physiological information and the requested LaBraM/prototype probes

Following your suggestion, we added matched physiological probes for frozen LaBraM and class-prototype alignment. The comparison shows stronger readouts of relative delta power and normalized spectral entropy from these representations than from Full. These results do not support attributing GVG's downstream classification gains to improved readout of these two spectral attributes.

All representations follow the HMC Physiology-Probe Protocol in Appendix D.2, using identical samples, subjects, cross-validation splits, and probe settings. Scores report the subject-weighted MSE reduction (%) relative to the sleep-stage mean baseline.

RepresentationRelative delta power: MSE reduction (%)Spectral entropy: MSE reduction (%)
Stage-only mean predictor0.000.00
Frozen pretrained LaBraM, mean-patch pooling77.9384.98
Frozen pretrained LaBraM, CLS token72.4674.41
Class-prototype52.4050.33
GVG Full50.3738.82

Relative delta power measures the proportion of low-frequency activity, while normalized spectral entropy describes how power is distributed across frequencies. Both vary among recordings assigned to the same sleep stage. Removing sleep-stage means tests whether aligned EEG features retain these continuous differences beyond the categorical labels.

Full reduces MSE by 50.37% for relative delta power (95% subject-bootstrap CI: 43.0–56.5%) and 38.82% for spectral entropy (31.8–45.0%). Both readouts have Holm-adjusted p = .004 against the cyclic-shift null. These predictions show that low-frequency activity and spectral-distribution differences remain recoverable from the aligned EEG features. LaBraM and class-prototype features make the same two attributes more accessible to the matched probe.

The new comparisons refine the interpretation of Section 5.4. Category clustering describes task organization, and residual EEG–proxy correspondence describes how the two representations vary together beyond class means. The physiological probes further identify measurable EEG attributes retained in the resulting features. The alignment, pairing, and input ablations provide evidence for contributions to downstream performance; the physiological analysis characterizes the signal information present in the learned representation.

W2: Generalization and external baseline comparisons

Thank you for raising the questions of evaluation comparability and generalization scope. We evaluate GVG using established EEG benchmarks and their customary protocols to strengthen the credibility of the evaluation and enable meaningful comparison with prior work. We follow the published data-processing and evaluation procedures as closely as possible and document the dataset partitions and result sources in Appendix A. For the additional controls, we use common preprocessing and metric definitions.

On SEED-VII, all seven specialist baselines and three NeuroLM variants are reproduced using the same partition and data manifest as GVG. Other external benchmark entries retain the published reference values identified in the appendix. The new LaBraM controls add a direct multi-task reference: a shared backbone with task-specific heads is jointly trained, and the same model is evaluated on all six tasks. The direct multi-task comparison gives:

Joint modelMacro₆ BAcc (%)
LaBraM + task-specific two-layer MLP heads56.24
GVG-Janus Full60.05

These comparisons, together with the internal alignment, input-route, and proxy controls, provide complementary evidence for the reported benefits.

We acknowledge that the benchmarks use several evaluation settings, including subject-dependent splits, cross-subject testing, and supervised adaptation to additional datasets. These settings follow the conventions of the respective benchmarks and assess different capabilities. SEED, SEED-IV, and SEED-VII use subject-dependent trial splits; HMC uses held-out subjects, and TUAB/TUEV use their official subject-independent partitions. TUSL and Workload are excluded from AVDE training and GVG Stage-1/2 pretraining and subsequently evaluated through adaptation with target-task labels, assessing transfer to tasks and electrode configurations unseen during pretraining. TUSL's subject overlap is documented as part of this task-transfer setting.

Using these established protocols allows us to evaluate the proposed visual-grounding approach within the existing EEG literature. We interpret the results as evidence of multi-task understanding and transfer under the corresponding benchmark settings, with each generalization claim tied to its actual data partition and adaptation procedure.

AuthorsOpenReview ↗

Manuscript revisions

Manuscript revisions. Following your comments on instance-specific clinical proxies, we will revise Section 5.2 and the discussion of Table 15 to explain the comparable HMC/TUAB results and the larger TUEV benefit, using the prototype and within-class shuffling comparisons to clarify task-dependent contributions. We will add the matched frozen-LaBraM and class-prototype physiological probes to Appendix D.2 and revise Section 5.4 to distinguish physiological information retained beyond sleep-stage labels from evidence explaining classification gains.

We will also clarify in Section 4.1 and Appendix A the sources of external baseline results, the subject-dependent and subject-independent evaluation settings, and the supervised adaptation used for TUSL and Workload, including TUSL's subject overlap. The generalization claims will be aligned with these evaluation settings.

Reviewer_3JAvOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Generative Visual Grounding (GVG) framework, which connects EEG signals with multimodal large language models (MLLMs) through the models' native visual-token vocabularies. For clinical EEG without paired visual stimuli, an EEG-to-image model (AVDE) generates proxy images as training-only visual targets, thereby enabling both stimulus-paired EEG and clinical EEG to be aligned with image and text features. The aligned EEG features are then mapped to native visual tokens. These tokens, combined with continuous EEG features, support prompt-based task understanding and can also be decoded into images by a frozen visual decoder. Experiments on six EEG benchmarks as well as TUSL and Workload show consistent improvements.

Critical strengths

This paper proposes a useful hypothesis: alignment with dense visual representations can provide additional supervision for EEG understanding. The study implements a framework for both visual-stimulus EEG and clinical EEG, and the task results demonstrate its effectiveness

Critical weaknesses
  1. Clinical EEG participates in Stage 1 alignment through proxy images, but Stage 2 token prediction is trained only on real EEG–stimulus pairs. The paper does not directly evaluate whether this token mapping transfers to clinical EEG, nor whether it can generate decoded images consistent with the corresponding proxy images.

  2. The continuous-only ablation shows that adding predicted visual tokens improves average task performance. However, two comparisons are not reported. First, a visual-token-only configuration that removes the continuous EEG prefix would assess task performance using the visual tokens alone. Second, replacing the visual-token prefix with an EEG-derived text prefix under comparable training settings would help assess whether the benefit is specific to visual representations.

Questions for authors
  1. For clinical EEG, how consistent are the decoded images with the corresponding proxy images?
  2. Please clarify how the “w/o Stage 2” ablation is implemented.
  3. With the same language-model adaptation settings, how does a visual-token-only input compare with the full model? How does replacing the visual prefix with an EEG-derived text prefix affect performance?
  4. It would be helpful to include GPT-6 scores for the comparison methods under the same evaluation protocol.
5 comments
AuthorsOpenReview ↗

Response to Q1/W1/Q2

Thank you for recognizing the value of visual supervision for EEG understanding. We address your questions with direct clinical reconstruction measurements, visual-token-only and text-token comparisons, and matched-sample ratings of the reconstruction methods.

Q1/W1: Clinical token transfer and decoded-image–proxy consistency

We have added a direct consistency evaluation for clinical EEG, comparing the decoded images and predicted token sequences with the corresponding AVDE proxy images and their tokens:

DatasetToken Top-1 (%)Token Top-10 (%)PSNR vs. proxy (dB)SSIMLPIPS
HMC0.150.988.670.12350.7330
TUEV0.090.807.490.09280.7685
TUAB0.050.506.700.07250.7859

These results show limited exact token agreement and detailed proxy reconstruction, with the strongest image consistency on HMC. The reference images are generated by AVDE rather than observed visual ground truth: clinical recordings have no paired visual stimulus. HMC, TUAB, and TUEV are also excluded from Stage-2 training, which uses only real EEG–stimulus pairs and is not trained to reproduce these clinical proxies. The metrics therefore quantify agreement with the teacher's synthetic references. Low image similarity to those references alone is insufficient to judge the generated clinical outputs as invalid.

For clinical EEG, the purpose of transferring the Stage-2 mapping is to provide a structured, discrete representation that the MLLM can use through its pretrained visual interface. Real EEG–stimulus pairs supervise the mapping from frozen aligned EEG features to native visual tokens. The w/o Stage 2 comparison tests the clinical benefit of this dedicated pretraining: both it and Full retain continuous EEG and predicted visual tokens, but the control randomly initializes the predictor and trains it during Stage 3, as detailed in Q2.

Clinical datasetw/o Stage 2 BAcc (%)Full BAcc (%)Difference (points)
HMC59.5871.51+11.93
TUEV50.7660.20+9.44
TUAB79.8780.08+0.21

Dedicated Stage-2 pretraining improves HMC and TUEV by 11.93 and 9.44 points, respectively, with a smaller 0.21-point gain on TUAB. These results support the transfer of the mapping learned from visually evoked EEG to subsequent clinical understanding.

The continuous-only control retains Stage-1/2 pretraining but omits the visual-token input. Its comparison with Full therefore evaluates the contribution of using the learned token representation alongside continuous EEG during Stage 3 (Table 12):

DatasetContinuous EEG onlyFull: continuous EEG + visual tokensDifference (points)
HMC68.7571.51+2.76
TUEV49.6160.20+10.59
TUAB80.7880.08−0.70
Six-task Macro₆55.4560.05+4.60

Adding the token branch improves HMC and TUEV and raises the six-task average by 4.60 points. TUAB shows closely grouped results across Text-only, image-only, w/o Stage 2, continuous-only, visual-token-only, and Full (79.87–80.96% BAcc). The 0.70-point decrease from continuous-only to Full is a small difference within this otherwise similar performance range. The stage-removal and input controls provide complementary evidence that the mapping learned from visually evoked EEG supports subsequent clinical understanding.

Q2: Implementation of “w/o Stage 2”

In w/o Stage 2, the token predictor is randomly initialized without token-prediction pretraining, while Stage 3 retains both continuous EEG features and predicted visual tokens. Stage 3 trains the predictor alongside the EEG-to-LLM adapter and language model, keeping the aligned EEG path frozen. This configuration reaches 51.76% Macro₆, compared with 60.05% for Full (Table 8).

AuthorsOpenReview ↗

Response to Q3/W2/Q4

Q3/W2: Visual-token-only and EEG-derived text-prefix inputs

We thank the reviewer for highlighting these two ablations, which are important to the empirical support for our work. First, visual-token-only removes the continuous EEG input throughout Stage-3 training and inference, retaining only the task prompt and EEG-predicted visual tokens. Comparing this configuration with continuous-EEG-only and Full establishes whether the tokens independently support task understanding and how the two representations work together. All scores below are BAcc (%).

Input routeSEEDSEED-IVSEED-VIITUEVTUABHMCMacro₆
Visual tokens only68.8641.8526.2729.7480.9667.9352.60
Continuous EEG only66.7632.3434.4849.6180.7868.7555.45
Full68.7045.3534.4560.2080.0871.5160.05

Visual-token-only reaches 52.60% Macro₆, showing that the predicted tokens carry task information that the MLLM can decode without the continuous EEG input. The two representations have different strengths: visual-token-only scores higher on SEED, SEED-IV, and TUAB, whereas continuous-only scores higher on SEED-VII, TUEV, and HMC. Combining them in Full raises Macro₆ to 60.05%, exceeding visual-token-only by 7.45 points and continuous-only by 4.60 points. This aggregate improvement supports their complementarity for shared multi-task understanding.

Second, the Text-matched control changes the Stage-2 training target to the text description associated with each EEG recording and replaces visual tokens with predicted text tokens during Stage-3 training.

ConfigurationSEEDSEED-IVSEED-VIITUEVTUABHMCMacro₆
Text-only54.8534.0619.0344.4180.1142.8945.89
Text-matched64.0631.5423.4239.9080.9866.3951.05
Image-only57.2942.0226.6448.3880.2766.1853.46
w/o Stage 263.4125.5631.3650.7679.8759.5851.76
Full68.7045.3534.4560.2080.0871.5160.05

Extending Text-only to Text-matched raises Macro₆ from 45.89% to 51.05% (+5.16 points), supporting the usefulness of learning discrete EEG-derived tokens and supplying them to the MLLM. This extension adds both token-prediction training and predicted-token inputs. The visual Stage-2 ablation addresses the role of dedicated pretraining more directly: w/o Stage 2 and Full both retain continuous EEG plus predicted visual tokens, but w/o Stage 2 starts with a randomly initialized predictor trained during Stage 3, as described in Q2. Full reaches 60.05% versus 51.76% (+8.29 points), supporting learning the EEG-to-token mapping before task adaptation. The two comparisons thus support both the learned discrete-token route and the value of its dedicated pretraining stage.

Across grounding configurations, Macro₆ increases from Text-matched (51.05%) to image-only (53.46%) and then trimodal Full (60.05%). Text-matched and image-only each lead on three tasks, while combining textual and visual grounding gives the strongest aggregate result: Full exceeds Text-matched by 9.00 points and image-only by 6.59 points. Full also exceeds Text-matched on five tasks; TUAB favors the text route (80.98% versus 80.08%). These experiments support our central claim: visual grounding complements text alignment, and discrete representation learning helps MLLMs decode and use EEG information more effectively for downstream understanding.

Q4: GPT-6 scores for comparison methods

Thank you for suggesting this comparison. We evaluate AVDE and GVG-Janus using GPT-6-Astra on 300 matched test samples—100 each from SEED, SEED-IV, and SEED-VII—with the same prompt and scoring rubric as Appendix E.1. The three dimensions are semantic category, color, and spatial layout, each scored on a 1–5 scale. The last column is their arithmetic mean.

ModelSemantic categoryColorSpatial layoutThree-dimension mean
AVDE2.392.191.862.15
GVG-Janus2.122.582.592.43

GVG-Janus scores higher on color and spatial layout, by 0.39 and 0.73 points, respectively, whereas AVDE scores 0.27 points higher on semantic category. GVG-Janus has a higher three-dimension mean (2.43 versus 2.15), with its advantage concentrated in overall appearance and spatial composition. This pattern is consistent with Section 5.3: EEG-predicted native visual tokens retain aspects of the stimulus's colors and coarse layout that can be decoded through the frozen native decoder, while semantic-category recovery remains limited.

AuthorsOpenReview ↗

Manuscript revisions

Manuscript revisions. We will add the clinical token and decoded-image–proxy consistency measurements to Appendix E and explain in Section 5.1 how the Stage-2 and input controls assess transfer to clinical EEG. Appendix B.1 will specify the random initialization, retained mixed inputs, and Stage-3 trainable modules of w/o Stage 2.

We will include the two input comparisons you requested—visual-token-only versus continuous-only and Full, and the EEG-derived text-token route—in Sections 4.2 and 5.1, with configuration definitions in Appendix A.1 and taskwise results in Appendix B. We will add the AVDE comparison alongside the GVG-Janus GPT-6-Astra ratings in Appendix E.1 and discuss the differences in color, spatial layout, and semantic-category correspondence in Section 5.3 under the same evaluation protocol.

Comments on the paper

9 comments
AuthorsOpenReview ↗

Response to Q1/W1

Thank you for recognizing the clinical proxy formulation, the pairing controls, and the shared six-task model. We address your questions with the new direct-classification, text-token, and visual-token-only results, alongside the existing controls.

Q1/W1: Value beyond LaBraM and computational cost

We thank the reviewer for pointing out this missing comparison. We agree that the task-specialist LaBraM results in Table 1 (62.18% Macro₆) do not establish whether GVG adds value over a direct classifier in a shared multi-task setting. We have therefore added three LaBraM multi-task baselines, covering a frozen backbone with linear heads and an unfrozen backbone with either linear or two-layer MLP heads.

Following LaBraM-Base's classification-based fine-tuning approach, we add simple task-specific classification heads to a shared LaBraM backbone and train jointly on the six datasets.

ConfigurationSEEDSEED-IVSEED-VIITUEVTUABHMCMacro₆
Frozen shared LaBraM + task-specific linear heads43.2029.1719.6238.6174.0853.9243.10
Unfrozen shared LaBraM + task-specific linear heads66.0339.0232.6051.9179.5767.5156.11
Unfrozen shared LaBraM + task-specific two-layer MLP heads64.8242.1130.8551.3779.0569.2856.24
GVG-Janus Full68.7045.3534.4560.2080.0871.5160.05

Joint training requires a shared representation to accommodate heterogeneous task objectives, which can introduce interference between tasks. GVG-Janus Full reaches 60.05%, exceeding the strongest LaBraM multi-task baseline by 3.81 points and scoring higher on all six tasks than each baseline; its gains over the strongest MLP baseline include 8.83 points on TUEV and 3.88 on SEED. These results support the value of the complete visual-grounding and MLLM pipeline for shared multi-task EEG understanding.

GVG also produces task answers through a unified language-generation interface, while the same EEG-predicted visual tokens can be decoded into images by the frozen native decoder. Combining language-based task understanding with visual reconstruction provides a foundation for a more general framework for EEG understanding and generation. As MLLMs, EEG foundation models, and available EEG data improve, this shared interface could support broader task understanding and visual generation with higher fidelity.

These capabilities come with higher inference cost: Table 18 reports 3.31× and 10.09× NeuroLM-XL's GFLOPs for Janus and X-Omni in the single-sample HMC setting. X-Omni's 170M figure measures trainable adaptation parameters; inference also computes through its frozen components. We consider the additional computational cost an acceptable trade-off for the stronger shared multi-task performance and the combined capabilities of language-based task understanding and visual reconstruction.

AuthorsOpenReview ↗

Response to Q2/W2

Q2/W2: Text-only versus image-only and the symmetric text-token control

Our Text-only baseline was designed to represent the conventional EEG–language alignment paradigm for connecting EEG to LLMs/MLLMs, with NeuroLM as an established external reference. Text-only Janus reaches 45.89% Macro₆, below NeuroLM-XL's 51.13%. These systems differ in EEG pretraining resources and language-model backbones, which can affect their baseline performance. Introducing our visual-grounding and native-token pipeline raises Janus to 53.46% with image-only grounding and 60.05% with trimodal alignment, surpassing NeuroLM-XL on four and six tasks, respectively. This reversal supports the effectiveness of the proposed pipeline for extending conventional EEG–language alignment.

We acknowledge that this comparison changes both the supervision modality and the token-prediction pathway and is not a fully rigorous controlled comparison. We therefore added an EEG-derived text-token route: Stage 2 trains the predictor using the text description associated with each EEG recording as the ground-truth target, and Stage 3 adds the predicted text tokens as an input prefix.

The comparison below reports BAcc (%).

ConfigurationSEEDSEED-IVSEED-VIITUEVTUABHMCMacro₆
Text-only54.8534.0619.0344.4180.1142.8945.89
Text-matched64.0631.5423.4239.9080.9866.3951.05
Image-only57.2942.0226.6448.3880.2766.1853.46
w/o Stage 263.4125.5631.3650.7679.8759.5851.76
Full68.7045.3534.4560.2080.0871.5160.05

The text-token route improves Macro₆ by 5.16 points over Text-only, supporting the value of token learning for text alignment. The paper's w/o Stage 2 versus Full ablation provides complementary evidence: both retain continuous EEG and predicted visual-token inputs, but dedicated Stage-2 token-prediction pretraining improves Macro₆ by 8.29 points. Together, these comparisons support learning a discrete EEG representation before task adaptation to help MLLMs understand EEG.

After adding discrete representation learning to the text route, the visual route retains a 2.41-point Macro₆ advantage. Image-only maintains similar performance on TUAB and HMC, while improving SEED-IV, SEED-VII, and TUEV by 10.48, 3.22, and 8.48 points, respectively. The text route performs better on SEED by 6.77 points, indicating different strengths across tasks.

Combining textual and visual grounding yields the strongest result: trimodal Full reaches 60.05% Macro₆, exceeding the text and image-only routes by 9.00 and 6.59 points, respectively. Full remains competitive on TUAB while outperforming both routes on all five remaining tasks. Relative to the stronger individual route on each task, Full gains 4.64 points on SEED, 3.33 on SEED-IV, 7.81 on SEED-VII, 11.82 on TUEV, and 5.12 on HMC. These results show the benefit of combining the two grounding signals and support visual grounding as a complement to text alignment.

AuthorsOpenReview ↗

Response to Q3

Q3: Contribution of the visual-token branch

We agree that the existing controls do not provide a strict quantitative decomposition of the 4.60-point gain into pipeline effects and visual-semantic content. Our additional visual-token-only experiment removes continuous EEG throughout Stage-3 training and inference, so the MLLM receives only the task prompt and EEG-predicted visual tokens.

The input and interface controls are summarized below (BAcc, %).

Input route or controlSEEDSEED-IVSEED-VIITUEVTUABHMCMacro₆
Visual tokens only68.8641.8526.2729.7480.9667.9352.60
Continuous EEG only66.7632.3434.4849.6180.7868.7555.45
Full: continuous EEG + visual tokens68.7045.3534.4560.2080.0871.5160.05
Length-matched continuous prefix56.3232.3523.0445.3681.1866.3750.77
Within-sample token-position shuffle56.6731.7428.1221.8675.6345.5043.25

Visual tokens alone carry task information the MLLM can use, although their average performance remains below continuous-EEG-only. This 2.85-point aggregate shortfall is mainly driven by TUEV (29.74% versus 49.61%): its 19.87-point deficit contributes 3.31 points to the six-task average, while the other five tasks collectively offset part of that deficit.

Combining the two inputs in Full raises Macro₆ to 60.05%, improving on continuous-only by 4.60 points after common Stage-1/2 pretraining. Full improves four of the six tasks: SEED by 1.94 points, SEED-IV by 13.01, TUEV by 10.59, and HMC by 2.76. The token branch can therefore supplement continuous EEG even though it is less effective as a standalone input.

The length-matched control tests whether extra input positions alone account for this improvement. It replaces the visual-token prefix with 256 adaptively pooled continuous EEG positions, while retaining the original continuous EEG prefix, task prompt, and upstream initialization. Performance is 4.68 points below continuous-only, rather than reproducing Full's gain. Supplying additional EEG-derived inputs is therefore insufficient: the content and form of the learned discrete representation matter.

Token organization provides a further part of the explanation. Keeping each sample's predicted token multiset unchanged but permuting its positions reduces Macro₆ by 16.80 points; training and evaluation use the same fixed permutation. Thus, retaining token content while disrupting its organization substantially impairs downstream decoding and can make the extra inputs interfere with understanding. Together, these experiments support two useful properties of the visual-token branch: EEG-derived content expressed in the MLLM's native visual vocabulary, and a positional organization that helps the MLLM use that content. Their combination complements continuous EEG; the benefit cannot be explained by additional input positions alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.