- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Generative Visual Grounding (GVG), which connects EEG signals to pretrained multimodal language models through visual representations. For recordings without visual stimuli, an adapted EEG-to-image teacher generates training-only proxy images. The framework aligns EEG with image/text features, predicts native image tokens, and combines these tokens with continuous EEG features for task prediction. Evaluation covers six jointly trained EEG tasks and two additional tasks with supervised adaptation. Full GVG-Janus achieves 60.05% average balanced accuracy on the six primary tasks, compared with 51.13% for NeuroLM-XL. Ablations, proxy-image controls, reconstruction experiments, and physiological probes provide additional validation.
Critical strengths
-
Training-only proxy images make it possible to reuse a pretrained visual interface for clinical recordings, while inference requires only EEG and a task prompt. This is a meaningful methodological contribution beyond the usual EEG-to-text alignment.
-
Full and LoRA GVG-Janus exceed NeuroLM-XL on all six primary tasks. Comparisons against continuous EEG inputs, class-prototype images, shuffled proxies, and a length-matched prefix help establish that the visual pathway contributes to performance. These controls make the central empirical finding more convincing than an improvement over external baselines alone.
Critical weaknesses
-
The physiological meaning of instance-specific clinical proxies remains uncertain. In Table 15, class-prototype images slightly outperform EEG-conditioned proxies on HMC and TUAB; the clinical advantage of instance-specific proxies is concentrated in TUEV. Moreover, the physiological probes evaluate Full against a stage-only baseline, rather than against the original LaBraM features or class-prototype alignment. The results support useful training supervision, but do not establish that the generated images preserve additional physiological information. This limits the stronger explanation offered for why clinical visual grounding works.
-
The evidence for broad generalization and comparative advantage is narrower than the framing suggests. Emotion datasets use subject-dependent splits; TUSL and Workload require supervised adaptation, and TUSL includes subject overlap. Most external baseline cells are inherited from published reports rather than reproduced under a common protocol. These choices do not invalidate the results, but leave uncertainty about performance on unseen users and the size of the advantage under fully matched evaluation.
Questions for authors
-
Given Table 15, should the claim about the benefit of instance-specific clinical proxies be limited primarily to TUEV? What explains their lack of improvement over class prototypes on HMC and TUAB?
-
Do the same physiological probes show an improvement over frozen LaBraM features and class-prototype alignment, using identical subjects and probe settings? This would help distinguish preservation of existing EEG information from a benefit specifically introduced by instance-level visual grounding.
5 comments
Response to Q2/W1/W2
Q2/W1: Physiological information and the requested LaBraM/prototype probes
Following your suggestion, we added matched physiological probes for frozen LaBraM and class-prototype alignment. The comparison shows stronger readouts of relative delta power and normalized spectral entropy from these representations than from Full. These results do not support attributing GVG's downstream classification gains to improved readout of these two spectral attributes.
All representations follow the HMC Physiology-Probe Protocol in Appendix D.2, using identical samples, subjects, cross-validation splits, and probe settings. Scores report the subject-weighted MSE reduction (%) relative to the sleep-stage mean baseline.
| Representation | Relative delta power: MSE reduction (%) | Spectral entropy: MSE reduction (%) |
|---|---|---|
| Stage-only mean predictor | 0.00 | 0.00 |
| Frozen pretrained LaBraM, mean-patch pooling | 77.93 | 84.98 |
| Frozen pretrained LaBraM, CLS token | 72.46 | 74.41 |
| Class-prototype | 52.40 | 50.33 |
| GVG Full | 50.37 | 38.82 |
Relative delta power measures the proportion of low-frequency activity, while normalized spectral entropy describes how power is distributed across frequencies. Both vary among recordings assigned to the same sleep stage. Removing sleep-stage means tests whether aligned EEG features retain these continuous differences beyond the categorical labels.
Full reduces MSE by 50.37% for relative delta power (95% subject-bootstrap CI: 43.0–56.5%) and 38.82% for spectral entropy (31.8–45.0%). Both readouts have Holm-adjusted p = .004 against the cyclic-shift null. These predictions show that low-frequency activity and spectral-distribution differences remain recoverable from the aligned EEG features. LaBraM and class-prototype features make the same two attributes more accessible to the matched probe.
The new comparisons refine the interpretation of Section 5.4. Category clustering describes task organization, and residual EEG–proxy correspondence describes how the two representations vary together beyond class means. The physiological probes further identify measurable EEG attributes retained in the resulting features. The alignment, pairing, and input ablations provide evidence for contributions to downstream performance; the physiological analysis characterizes the signal information present in the learned representation.
W2: Generalization and external baseline comparisons
Thank you for raising the questions of evaluation comparability and generalization scope. We evaluate GVG using established EEG benchmarks and their customary protocols to strengthen the credibility of the evaluation and enable meaningful comparison with prior work. We follow the published data-processing and evaluation procedures as closely as possible and document the dataset partitions and result sources in Appendix A. For the additional controls, we use common preprocessing and metric definitions.
On SEED-VII, all seven specialist baselines and three NeuroLM variants are reproduced using the same partition and data manifest as GVG. Other external benchmark entries retain the published reference values identified in the appendix. The new LaBraM controls add a direct multi-task reference: a shared backbone with task-specific heads is jointly trained, and the same model is evaluated on all six tasks. The direct multi-task comparison gives:
| Joint model | Macro₆ BAcc (%) |
|---|---|
| LaBraM + task-specific two-layer MLP heads | 56.24 |
| GVG-Janus Full | 60.05 |
These comparisons, together with the internal alignment, input-route, and proxy controls, provide complementary evidence for the reported benefits.
We acknowledge that the benchmarks use several evaluation settings, including subject-dependent splits, cross-subject testing, and supervised adaptation to additional datasets. These settings follow the conventions of the respective benchmarks and assess different capabilities. SEED, SEED-IV, and SEED-VII use subject-dependent trial splits; HMC uses held-out subjects, and TUAB/TUEV use their official subject-independent partitions. TUSL and Workload are excluded from AVDE training and GVG Stage-1/2 pretraining and subsequently evaluated through adaptation with target-task labels, assessing transfer to tasks and electrode configurations unseen during pretraining. TUSL's subject overlap is documented as part of this task-transfer setting.
Using these established protocols allows us to evaluate the proposed visual-grounding approach within the existing EEG literature. We interpret the results as evidence of multi-task understanding and transfer under the corresponding benchmark settings, with each generalization claim tied to its actual data partition and adaptation procedure.
Manuscript revisions
Manuscript revisions. Following your comments on instance-specific clinical proxies, we will revise Section 5.2 and the discussion of Table 15 to explain the comparable HMC/TUAB results and the larger TUEV benefit, using the prototype and within-class shuffling comparisons to clarify task-dependent contributions. We will add the matched frozen-LaBraM and class-prototype physiological probes to Appendix D.2 and revise Section 5.4 to distinguish physiological information retained beyond sleep-stage labels from evidence explaining classification gains.
We will also clarify in Section 4.1 and Appendix A the sources of external baseline results, the subject-dependent and subject-independent evaluation settings, and the supervised adaptation used for TUSL and Workload, including TUSL's subject overlap. The generalization claims will be aligned with these evaluation settings.
Response to Q1/W1
Thank you for recognizing the contribution of training-only visual proxies and the evidence from the internal controls. Our analyses connect clinical category organization, correspondence between individual EEG recordings and their visual targets, and physiological variation retained in the aligned EEG features. The additional probes clarify how these properties coexist in the learned interface.
Q1/W1: Instance-specific proxies versus class prototypes
We agree that the additional clinical classification gain over class prototypes is concentrated in TUEV. Full and class prototypes perform similarly on HMC and TUAB, with Full lower by 0.17 and 0.81 percentage points, respectively, whereas EEG-conditioned proxies improve TUEV by 14.72 points. Table 15 gives the clinical results (BAcc, %):
We interpret this pattern through the recording-level information needed for each task. A fixed prototype makes the visual supervision target constant within a class, while the input EEG features remain specific to each recording. Richer targets therefore add useful supervision when their within-class differences help distinguish the task's categories.
HMC sleep stages are characterized by typical rhythms and events, while physiological variation within a stage need not change its label. A stable class-level target can therefore support stage discrimination without representing every within-stage difference. Correct pairing still matters when recording-dependent targets are used: within-class shuffling preserves category identity and the proxy image pool but changes EEG assignments, lowering HMC from 71.51% to 51.43% (−20.08 points). Unlike a fixed class anchor, this introduces recording-dependent targets associated with a different EEG sample. Full also improves class-centered EEG–proxy residual correspondence relative to w/o Stage 1 (Δ = 0.0226, significant after Holm correction; Section 5.4). These results support the presence and pairing relevance of within-class variation, even though it does not yield an additional staging gain over the stable prototype target.
TUAB asks the broader normal-versus-abnormal question, with different abnormal manifestations sharing the same label. Consistent category anchors may already provide effective supervision for this decision, while finer target variation provides no additional classification gain in this comparison. Scores are closely grouped across Text-only (80.11%), image-only (80.27%), class prototypes (80.89%), Full (80.08%), and specialist LaBraM (81.40%). Within-class shuffling lowers Full to 76.71% (−3.37 points), a smaller drop than on HMC or TUEV. TUAB likewise shows no significant residual-correspondence improvement relative to w/o Stage 1 (Δ = −0.0037, p = .232).
TUEV distinguishes spike/sharp waves, generalized and lateralized periodic discharges, eye movements, artifacts, and background activity. These distinctions involve event morphology and spatial distribution, making recording-dependent target differences potentially more useful for classification. One fixed image per class does not provide those target differences. Here, EEG-conditioned proxies improve classification from 45.48% to 60.20%, while within-class shuffling reduces it to 19.63% (−40.57 points). Full also improves class-centered residual correspondence relative to w/o Stage 1 (Δ = 0.0335, significant after Holm correction). The larger prototype gap and stronger shuffling effect support the greater practical value of recording-specific targets for TUEV's event discrimination.