acceptodds
Under review as a conference paper at ICLR 2027

Teaching Neural Decoders with Language

Abstract

Brain–computer interfaces decode neural activity, yet class-label supervision (CLS) leaves relationships among classes implicit. We propose Language as Supervision (LaS), which aligns neural features with natural-language class descriptions through learned projections and predicts classes using candidate text prototypes. We evaluate seven encoder architectures with independently trained models for visual, motor, and speech decoding from scalp and intracranial recordings. LaS improves speech-decoding accuracy by up to 17.15 pp over matched CLS baselines and supports subject and session adaptation. Frequency-structured neural–text correspondence further enables conditional decoding of unseen stimulus frequencies in steady-state visual evoked potential (SSVEP) responses. Dual-sided attribution visualization shows contributions from neural signal components and prompt words to the same class-specific matching score. These findings suggest that text descriptions can serve as general-purpose references for learning and interpretation in neural decoding.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain-computer interfaces, neural decoding, language supervision, multimodal representation learning, prototype-based learning, zero-shot learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_g8g8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the possible shortcomings of traditional category-supervised CLS and proposes a LaS approach that uses frozen text encoders, learnable neural/text projections, and bidirectional InfoNCE to align neural trials with category descriptions and classify by candidate text prototypes. The paper evaluates seven encoder architectures across motor imagery, SSVEP, and overt speech decoding, and explores candidate-subset reuse, adaptation, and conditional unseen-frequency decoding. It also analyzes repeated class descriptions in contrastive training and visualizes neural and word contributions to the same matching score.

Critical strengths
  1. I appreciated seeing the same class-description interface tested on motor imagery, SSVEP, and speech decoding, with both EEG and intracranial recordings and several encoder families. I also found the adaptation and candidate-set experiments informative when considering its possible uses beyond the main classification tasks.
  2. I think the task-dependent results and cases without improvement were informative, and I appreciated that the authors discuss them alongside the positive findings. This gives the work value as an investigation of language-guided decoding and helps readers understand its scope.
Critical weaknesses
  1. I think the methodological advance and the contribution of text semantics need clearer support. NICE++ [1] already uses a similar language-guided neural alignment based on infoNCE loss on image classification tasks, where I think the novelty is not prominent enough due to seemingly high method overlap. In addition, EmotionCLIP and EEG-PRIME mentioned in related work also use class-text matching. A comparison with an applicable text-prototype approach would help establish LaS's contribution. More critically, the paper lacks random or learned class prototypes under the same readout and loss. The nonsignificant closed-set P0/P1 differences make these controls particularly relevant.
  2. I cannot yet separate the contribution of language supervision from the accompanying decoder changes. For example, MI EEGNet changes its readout from 48 to 400 dimensions, CNN-BiGRU changes how it reads the bidirectional sequence, and other models change pooling, dropout, or trial aggregation. And parameter counts also change substantially in either direction. These choices can affect information retention and regularization. I would find the language-supervision claim more convincing with matched readouts, training settings, and effective parameter budgets. [1] Song, Yonghao, et al. "Recognizing natural images from eeg with language-guided contrastive learning." IEEE Transactions on Neural Networks and Learning Systems 36.9 (2025): 15896-15910.
Questions for authors
  1. Have authors also evaluated LaS on a Du-IN MAE-pretrained encoder, alongside the corresponding pretrained CLS baseline? This would help establish whether the benefit carries over to the strongest available Du-IN setting.
  2. Could authors provide experimental results on multiple random seed pairs for CLS/LaS? It would help to see the variation in gains across runs.
  3. In Figure 3, could authors also restrict the predictions of an existing four-class CLS checkpoint to the same candidate subsets? The current comparison uses subset-trained CLS models. This additional result would clarify how much of the reuse benefit comes from the text interface versus training on the larger class set.
8 comments
AuthorsOpenReview ↗

Response to g8g8 (1/6): W1: Methodological contribution and prior methods

Thank you for taking the time to review our paper and for recognizing the value of evaluating the same class-description interface across paradigms, recording modalities, and encoders.

The additional results show that LaS achieves MI four-class accuracy close to LEAF-FT with approximately 1/55 as many trainable parameters, and that all four Speech encoders retain positive mean gains across five runs against the updated CLS controls. The random-vector and frequency-adjacency experiments further address how class representations support closed-set fitting and extension to unseen frequencies. Below, we address your concerns and questions point by point using these additional experiments and analyses. The MI class labels are: 1=thumb, 2=index finger, 3=middle finger, and 4=little finger.

Weakness 1. Methodological contribution, existing methods, and the role of class representations

I think the methodological advance and the contribution of text semantics need clearer support.

Text provides general-purpose semantic representations for classes across neural decoding paradigms, allowing class attributes and relationships to inform supervision and prediction. LaS generates class representations with a pretrained text encoder and uses the same descriptive approach in independently trained models for MI, SSVEP, and Speech.

Relationship to existing methods and direct comparison

Thank you for stressing the importance of a direct comparison with existing text-prototype methods; we have added fine-tuning results for LEAF (Jiang et al., 2026) using the same target training set and validation and evaluation splits.

LaS replaces the classification head with neural projection and text-prototype matching, using the same form of language supervision with classical encoders across multiple paradigms. NICE++ (Song et al., 2025) uses language to improve image decoding, EmotionCLIP (Yan et al., 2026) targets EEG emotion recognition, and EEG-PRIME (Zhang et al., 2026) uses pretraining and multi-level conditioning. We examine this approach through paired evaluations of CLS and LaS and further analyze how class descriptions affect learning and prediction.

We distinguish the class-discrimination role of N2T from the within-class matching role of T2N along text-prototype directions and analyze the loss terms arising from repeated descriptions. With the input on the other side held fixed, dual-sided attribution relates the contributions of neural channels, spatial locations, time intervals, and prompt words to the same specified matching score, allowing researchers to inspect the basis of matching from both input sides.

The new direct comparison shows that LaS achieves MI four-class accuracy close to LEAF-FT with approximately 1/55 as many trainable parameters. LEAF-FT starts from the official instruction-trained weights and fine-tunes the EEG encoder and Q-Former on the same target training set as LaS, with fixed text prototypes and the same validation and evaluation splits. Both use the five-fold protocol with held-out validation and trial-level evaluation adopted here, and LEAF-FT follows the official EEG encoder + Q-Former fine-tuning procedure.

MethodTrainable parameters (M){1,2,3,4}{2,3}{1,4}{1,2,3}{1,3,4}
LEAF-FT26.4241.99 ± 7.2857.55 ± 6.7974.21 ± 8.1645.05 ± 6.7753.92 ± 8.61
LaS, EEG-Conformer0.4841.94 ± 8.8959.53 ± 9.7874.60 ± 8.4045.51 ± 8.0954.30 ± 10.12

Accuracy is reported in %, as the mean across 21 subjects and the between-subject sample SD. Parameter counts include the parameters being fine-tuned or trained and exclude the frozen text encoder.

The neural encoder in LaS is trained from scratch, with its four-class mean differing from LEAF-FT by only 0.05 pp and its means on the four subtasks higher by 0.38–1.98 pp.

When each class has m identical text entries in a balanced batch, N2T decomposes into class cross-entropy and the constant log⁡m\log m; ablations of direction, loss objective, and batch further examine how these training effects relate to decoding performance.

AuthorsOpenReview ↗

Response to g8g8 (2/6): W1: Fixed-random and unseen-frequency results

Contribution of text semantics and fixed random-vector controls

More critically, the paper lacks random or learned class prototypes under the same readout and loss.

We understand your concern: the role of class representations needs to be tested directly under the same architecture and training objective.

The differences in closed-set mean accuracy between fixed random vectors and text are below 1 pp in all three paradigms.

For each class, we independently generate a standard Gaussian vector with the same dimensionality as the language features, apply L2 normalization, and then fix it: gc∼N(0,Id),rc=gc/∣gc∣2g_c\sim\mathcal N(0,I_d),\\ r_c=g_c/\\|g_c\\|_2. We replace only the precomputed language features and retain the trainable projection WtW_t; the rest of the neural model, bidirectional InfoNCE loss, temperature, data splits, and training settings remain the same as in the language control. The vectors before projection are fixed, while the projected prototypes are updated during training.

Task and encoderConditionReal text (%)Fixed random vectors (%)Text−random (pp)
MI, EEG-ConformerFour classes, 1.0 s41.9441.39+0.55
SSVEP, MTSNet40 classes, 1.0 s, RW→IW66.7967.36−0.57
Speech, Du-IN61 classes, 3.0 s60.5859.79+0.79

For each paradigm, the fixed-random control reports the mean accuracy of experiments with three independent sets of random class vectors. MI uses the newly established five-fold protocol with held-out validation; 1.0 s is the model input window length, and trial predictions are obtained by averaging the window scores within each trial. SSVEP uses the corresponding conditions in Table 1 and aggregates results with equal weight for each of the 35 subjects.

Both text vectors and random vectors can provide reference coordinates for class alignment. The pairwise inner products of independently normalized Gaussian vectors have zero mean and an expected squared value of 1/d1/d, so these vectors are usually nearly orthogonal in high dimensions. This gives different classes distinct fixed references.

The trainable projection further allows the model to adjust these references. Let R∈RC×dR\in\mathbb R^{C\times d} contain the random vectors; when C≤dC\leq d, RR has full row rank almost surely. For any target prototype matrix Q∈RC×pQ\in\mathbb R^{C\times p} whose rows are unit vectors, an unconstrained linear projection can be chosen as

\quad\Longrightarrow\quad RW_t^{(r)}=Q.$$ Under the full-row-rank condition above, the trainable projection can map fixed random vectors to given seen-class prototypes, providing a geometric explanation for the similar closed-set results. The compensation for seen-class permutations analyzed in Appendix C.1 also uses this expressive capacity of the projection. **With the model fixed, introducing frequency adjacency only for unseen classes increases ZSL accuracy from 17.92% to 94.52%.** We compare independent random class vectors, real text, and a frequency-adjacency construction using MTSNet with 1-second EEG inputs. The three conditions use the same eight class splits and evaluation trials, with predictions among five unseen frequency classes in each split and a chance level of 20%. The 40 frequencies range from 8.0 to 15.8 Hz in increments of 0.2 Hz; the held-out frequencies in split j are $8.0+0.2(j-1)+1.6k$ Hz, where $j=1,\ldots,8$ and $k=0,\ldots,4$. The unseen frequencies in each split have adjacent seen frequencies available for the construction. Within each class split, the two random-representation conditions reuse the same trained checkpoint, keeping both the neural branch and the projection fixed and changing only the input vectors for unseen classes at inference. The adjacency-based construction is performed before projection: for an interior unseen frequency, we average the initial unit random vectors of its neighboring seen classes on either side and normalize the result again. For example, 8.2 Hz uses the vectors for 8.0 and 8.4 Hz. | Class representation | 1.0 s ZSL accuracy (%) | | -------------------------- | -------------------: | | Independent random class vectors | 17.92 ± 8.66 | | Real text | 90.89 ± 1.43 | | Random vectors constructed using frequency adjacency | 94.52 ± 0.83 | The results are the mean accuracy and sample SD across the eight class splits, covering 35 subjects. The endpoints 8.0 and 15.8 Hz use the vectors of the nearest seen frequencies, 8.2 and 15.6 Hz, respectively.
AuthorsOpenReview ↗

Response to g8g8 (3/6): W1: Frequency relationships and stimulus control

Let ν(v)=v/∣v∣2\nu(v)=v/\\|v\\|_2, and let r−r_- and r+r_+ be the initial unit random vectors of adjacent seen classes. For an interior unseen class uu, under a linear projection and provided that the vectors being normalized are nonzero,

t_u=\nu(r_uW_t) =\nu(r_-W_t+r_+W_t).$$ The adjacency construction connects the projected direction of an unseen class to the trained directions of adjacent classes. Independent random vectors provide distinct class references; the adjacency construction further encodes local frequency relationships in the class representations. When neural representations vary smoothly with frequency locally and are compatible with this interpolation relationship, the model can use the matching relationships learned from seen classes to recognize unseen frequencies. Section 1 notes that one-hot targets identify classes without explicitly encoding class attributes or semantic relationships. Independently sampled fixed random inputs likewise do not encode these relationships in advance. The frequency-adjacency control adds relationships to the random inputs by construction, improving unseen-frequency accuracy by 76.61 pp with the same random-vector model. Text achieves 90.89% through frequency descriptions; together with the P0/P1 permutation results, these findings support the role of class relationships in this prediction task. Frequencies have an explicit numerical order, making adjacency rules straightforward to specify; more abstract class relationships may lack an equally direct vector construction. Natural language can describe class attributes and relationships directly, and a pretrained text encoder converts these descriptions into representations, allowing classes in MI, SSVEP, and Speech to use the same descriptive approach. With the neural input held fixed, word attribution allows researchers to inspect each word’s contribution to the matching score for a specified class. **Comparison of text and stimulus representations** We also compared supervision that directly encodes stimulus frequency and phase. The stimulus-encoding and text conditions use the same MTSNet neural encoder, 1-second EEG inputs, eight class splits, and evaluation trials. The stimulus branch uses ModernTCN ([Luo and Wang, 2024](https://openreview.net/forum?id=vpJMJerXHU)) to encode a 1-second sinusoidal reference and is trained jointly with MTSNet. | Method | 1.0 s accuracy (%) | Relative to FBCCA (pp) | | ------------------------- | ---------------: | --------------: | | FBCCA | 88.83 ± 1.22 | — | | MTSNet + ModernTCN stimulus encoding | 82.57 ± 3.52 | −6.26 | | MTSNet + text representations (LaS) | 90.89 ± 1.43 | +2.06 | The results are the mean accuracy and sample SD across the eight class splits. Text achieves higher unseen-frequency accuracy in this comparison. Text features can be precomputed with the frozen text encoder, and training updates only the linear projection on the text side; the stimulus approach requires additional trainable parameters for ModernTCN and joint training of the stimulus encoder.
Reviewer_nWjaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Language as Supervision (LaS), a framework that trains neural decoders by aligning neural signal representations with textual descriptions of candidate classes via text embeddings. The authors evaluate the approach across seven encoder architectures and three neural recording modalities (motor imagery, SSVEP, and overt speech). The empirical findings show that the largest performance gains occur in speech decoding, where LaS achieves up to a 17.15 percentage point improvement over conventional class-label supervision.

Critical strengths
  • Comprehensive Evaluation: The proposed method is benchmarked across multiple encoder architectures and neural recording modalities. The speech decoding task shows consistent accuracy improvements across all four tested encoders, highlighting the practical potential of the approach.
  • Insightful Analysis: The paper effectively decouples closed-set classification performance from zero-shot generalization to unseen target frequencies. Crucially, the permutation control shows that shuffling class descriptions has minimal impact on closed-set accuracy, whereas it collapses unseen-frequency decoding performance to near-chance levels.
Critical weaknesses
  • Confounding Variables in Baseline Comparisons: The experimental setup leaves ambiguity regarding whether the observed gains stem from language semantics or from architectural and optimization differences.
    • Loss & Readout Discrepancies: Appendix A notes that the Class-Label Supervision (CLS) baselines apply cross-entropy loss over sigmoid outputs, whereas LaS employs temperature-scaled cosine similarity. The readouts and regularization protocols also differ. These design choices directly affect gradient dynamics and optimization, making raw-logit CLS controls essential to isolate the true effect of language supervision.
    • Lack of Random Embedding Baselines: Given that closed-set accuracy remains robust under class description permutations, a control experiment using fixed random embeddings (under the exact same alignment objective) is necessary to determine if text representations provide meaningful semantic structure or merely act as an arbitrary fixed geometric target.
  • Missing Comparisons with Existing Text-Based Decoders: While the paper reviews prior work on language-guided neural decoding, it lacks direct experimental comparisons. In particular, EEG-PRIME is highly relevant as it already incorporates text-defined class prototypes for motor imagery classification. A comparative evaluation on a unified task—properly accounting for differences in pretraining and architecture—is necessary to establish the state-of-the-art positioning of LaS.
Questions for authors
  • Ablation & Baseline Controls:
    • Do the performance gains in speech decoding persist when the CLS baseline uses standard raw logits with cross-entropy loss, while matching the readout architecture and regularization as closely as possible?
    • How does LaS perform when the text embeddings are replaced with fixed random embeddings while keeping the linear projections, alignment loss, and temperature scaling identical?
  • Comparison with Prior Work: Could the authors provide a comparative analysis between LaS and an adapted implementation of EEG-PRIME on the motor imagery task, evaluated under identical data splits and explicitly controlling for any additional pretraining data?
5 comments
AuthorsOpenReview ↗

Response to nWja (1/4): Speech baselines

Thank you for carefully reviewing our paper and recognizing the value of the breadth of the evaluation and the class-correspondence analysis.

Below, we address each weakness and question through supplementary analyses and clarifications, which will be included in the revised manuscript. The MI class indices are: 1=thumb, 2=index finger, 3=middle finger, and 4=little finger.

1. Do the performance gains in speech decoding persist when the CLS baseline uses standard raw logits with cross-entropy loss, while matching the readout architecture and regularization as closely as possible?

We understand your concern about the Speech comparison settings and have added experiments using raw logits and matched the bidirectional readout for CNN-BiGRU.

All four Speech encoders retain positive mean gains against the CLS controls using raw logits, and CNN-BiGRU still improves by 5.52 pp after matching the bidirectional readout.

We retrained all four CLS baselines, using the same five random seeds for CLS and LaS for each encoder. All four Speech CLS baselines compute cross-entropy directly on raw logits. CNN-BiGRU was also changed to concatenate the final hidden states in both directions from the last layer, giving it the same 256-dimensional readout as LaS.

EncoderUpdated CLS accuracy (%)LaS accuracy (%)LaS−CLS (pp)
EEGNet6.1215.46+9.33
CNN-BiGRU47.9553.47+5.52
EEG-Conformer50.6553.56+2.90
Du-IN58.8460.29+1.45

The table reports the mean accuracy of each method over the same set of five random seeds. Across the five paired comparisons, the gains for EEGNet, CNN-BiGRU, and EEG-Conformer are all positive, falling in the ranges 8.74–10.28, 4.97–6.39, and 2.30–4.20 pp, respectively; the gains for Du-IN range from −0.10 to 2.10 pp and are positive in four of the five comparisons.

For Speech models with an MLP classification head, the head combines neural features through nonlinear transformations and outputs class scores, while the bottleneck layer also compresses the features. At the neural feature output, LaS replaces the classification head with projection and text-prototype matching to map neural features to class scores. The attachment point is determined by the encoder's feature interface; we did not select or optimize the readout location.

Apart from the CNN-BiGRU bidirectional readout matched in these experiments, the readout and pooling for each model follow the settings of its respective classification or projection path. Some dropout layers in CLS are within the MLP classification head, which LaS replaces with projection and text-prototype matching. Some dropout rates also differ between CLS and LaS in the Transformer backbones of Speech EEG-Conformer and Du-IN. The optimizers and learning-rate annealing strategies are largely consistent between the two methods, and paired training uses the same weight-decay coefficient. Both methods select optimization configurations based on validation performance, with comparable tuning effort.

The revised manuscript will report the updated Speech comparison and update the relevant performance conclusions accordingly. CNN-BiGRU gains 5.52 pp with raw-logit cross-entropy and the matched bidirectional readout; the original 17.15 pp result will retain a label identifying its original output and readout configuration.

AuthorsOpenReview ↗

Response to nWja (2/4): Fixed-random controls: closed-set results

2. How does LaS perform when the text embeddings are replaced with fixed random embeddings while keeping the linear projections, alignment loss, and temperature scaling identical?

We understand your concern: the role of class representations needs to be tested directly under the same architecture and training objective.

Fixed random vectors and text differ by less than 1 pp in mean closed-set accuracy in each of the three paradigms; text also provides a general way to express class attributes and relationships.

We designed a new experiment in which an independent standard Gaussian vector of the same dimensionality as the language features is generated for each class, L2-normalized, and then fixed. We replace only the precomputed language features and retain the trainable projection WtW_t; the neural model, bidirectional InfoNCE loss, temperature, data splits, and training settings otherwise remain the same as in the text control. The random vectors before projection are fixed, while the class prototypes after projection continue to update during training.

Task and encoderConditionReal text (%)Fixed random vectors (%)Text−random (pp)
MI, EEG-ConformerFour classes, 1.0 s41.9441.39+0.55
SSVEP, MTSNet40 classes, 1.0 s, RW→IW66.7967.36−0.57
Speech, Du-IN61 classes, 3.0 s60.5859.79+0.79

For each paradigm, the fixed-random control reports the mean accuracy of experiments with three independent sets of random class vectors. MI uses the newly established five-fold protocol with held-out validation; 1.0 s is the model input window length, and trial predictions are obtained by averaging the window scores within each trial. SSVEP uses the corresponding conditions in Table 1 and aggregates results with equal weight for each of the 35 subjects.

Independent normalized Gaussian vectors are typically nearly orthogonal in high dimensions: the inner product between different classes has expectation 0 and expected square 1/d1/d. They can provide separate fixed references for different classes. The trainable projection in LaS further determines the positions of these references in the matching space. Let R∈RC×dR\in\mathbb R^{C\times d} be the random-vector matrix, where CC is the number of classes and dd is the input vector dimension. When C≤dC\leq d, RR has full row rank almost surely; given a target unit-prototype matrix Q∈RC×kQ\in\mathbb R^{C\times k}, where kk is the matching-space dimension, we can set

Wt(r)=R†Q,RWt(r)=Q.W_t^{(r)}=R^\dagger Q,\qquad RW_t^{(r)}=Q.

Fixed random references combined with a trainable linear projection can express these seen-class prototypes, consistent with the similar closed-set accuracies observed. The projection compensation for permutations of seen classes in Appendix C.1 uses the same idea.

AuthorsOpenReview ↗

Response to nWja (3/4): Fixed-random controls: unseen-frequency results

Once the random-representation model is fixed, introducing frequency adjacency only for unseen classes increases ZSL accuracy from 17.92% to 94.52%. We designed another new experiment using MTSNet with 1 s EEG input to compare independent random class vectors, real text, and a construction based on frequency adjacency. The three conditions use the same eight class splits and evaluation trials, with predictions among five unseen frequency classes in each split and a chance level of 20%. The 40 frequencies range from 8.0 to 15.8 Hz in steps of 0.2 Hz; the frequencies held out in split j are 8.0+0.2(j−1)+1.6k8.0+0.2(j-1)+1.6k Hz, where j=1,…,8j=1,\ldots,8 and k=0,…,4k=0,\ldots,4. The unseen frequencies in each split have adjacent seen frequencies available for the construction.

Within each class split, the two random-representation conditions reuse the same trained checkpoint, keeping both the neural branch and the projection fixed and changing only the input vectors for unseen classes at inference. The adjacency-based construction is performed before projection: for an interior unseen frequency, we average the initial unit random vectors of its neighboring seen classes on either side and normalize the result again. For example, 8.2 Hz uses the vectors for 8.0 and 8.4 Hz.

Class representation1.0 s ZSL accuracy (%)
Independent random class vectors17.92 ± 8.66
Real text90.89 ± 1.43
Random vectors constructed using frequency adjacency94.52 ± 0.83

The results are the mean accuracy and sample SD across the eight class splits, covering 35 subjects. The endpoints 8.0 and 15.8 Hz use the vectors of the nearest seen frequencies, 8.2 and 15.6 Hz, respectively.

Let ν\nu denote L2 normalization of a nonzero vector, and let r−r_- and r+r_+ be the initial unit random vectors of neighboring seen classes. For an interior unseen class uu, when the relevant vectors are nonzero,

t_u=\nu(r_uW_t)=\nu(r_-W_t+r_+W_t).$$ The adjacency construction thus connects the projected direction of an unseen class to the directions learned for its neighboring classes. Provided that the neural representations vary smoothly with frequency locally and are compatible with this interpolation relationship, the model can use the matching relationships learned for seen classes to recognize unseen frequencies. Independent random vectors provide separate class identifiers, while the adjacency construction further encodes known local frequency relationships. Section 1 notes that one-hot targets identify classes without explicitly encoding class attributes or semantic relationships. Independently sampled fixed random inputs likewise do not encode these relationships in advance. The frequency-adjacency control adds relationships to the random inputs by construction, improving unseen-frequency accuracy by 76.61 pp with the same random-vector model. Text achieves 90.89% through frequency descriptions; together with the P0/P1 permutation results, these findings support the role of class relationships in this prediction task. Frequencies have an explicit numerical order, making adjacency rules straightforward to specify; more abstract class relationships may lack an equally direct vector construction. Natural language can describe class attributes and relationships directly, and a pretrained text encoder converts these descriptions into representations, allowing classes in MI, SSVEP, and Speech to use the same descriptive approach.
Reviewer_zjiuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Language as Supervision (LaS), a framework that aligns neural-signal features with natural-language class descriptions using learned projections and a bidirectional contrastive objective. At inference time, classes are predicted by matching neural features to candidate text prototypes. The method is evaluated with seven encoder architectures across three decoding tasks: motor imagery and steady-state visual evoked potential (SSVEP) decoding from scalp EEG, and speech decoding from intracranial stereo-electroencephalography (sEEG). Each task–encoder combination is trained separately. The largest reported gain is a 17.15-percentage-point increase in speech-decoding accuracy over the corresponding class-label supervision (CLS) baseline.

The experiments also examine candidate-subset reuse, subject and session adaptation, and recognition of held-out SSVEP frequencies under the specified evaluation setting. Further analyses consider how repeated class descriptions affect contrastive training, compare prompt configurations and text encoders, and visualize the contributions of neural inputs and prompt words through dual-sided attribution.

Critical strengths
  1. Broad evaluation with consistent speech-decoding gains. The paper evaluates seven encoder architectures across motor imagery, SSVEP, and intracranial speech decoding. All four speech encoders improve over their corresponding baselines, with gains of 5.22 percentage points for Du-IN and 17.15 percentage points for CNN-BiGRU. The improvements across all four encoders suggest that the speech-decoding results are not limited to a single architecture and provide the strongest empirical support for the paper.
  2. A useful distinction between closed-set fitting and unseen-frequency recognition. In the SSVEP experiments, neural examples from selected frequency classes are withheld, and recognition is evaluated on unseen candidates. Permuting class descriptions has little effect on closed-set performance but reduces unseen-frequency recognition to near chance. This contrast suggests that performance on seen classes does not necessarily reflect the ability to transfer to unseen frequencies, providing a useful perspective on what the two evaluation settings measure.
Critical weaknesses
  1. The main comparisons do not sufficiently isolate the benefit of language supervision. Appendix A.4 states that three Speech CLS baselines apply cross-entropy to sigmoid outputs, whereas LaS uses temperature-scaled similarity scores. CNN-BiGRU also uses different readouts: CLS takes the last sequence position, while LaS concatenates the final bidirectional hidden states. This encoder yields the largest reported gain, making the readout difference particularly relevant. These design differences complicate attribution of the gains to language supervision. Although the paper includes prompt-permutation and template-variation experiments, unchanged closed-set performance under prompt permutation does not establish how much semantic information contributes, especially given the flexibility of the learned text projection. Matched readouts, a standard raw-logit cross-entropy baseline, and nonsemantic prototype controls would help isolate the effect of language supervision.

  2. Some performance estimates are not independent of model selection. In the MI evaluation, the same held-out fold is used both for checkpoint selection and for reporting trial-level accuracy (§4.1; Appendix A.2), potentially biasing the reported results upward. In the SSVEP adaptation experiments, validation-block accuracy also determines the selected training epochs (§4.3), so the reported performance is not independently evaluated. Paired significance tests do not address this reuse of data. An independent outer-test evaluation for MI and test blocks excluded from epoch selection for SSVEP adaptation would help determine whether the findings generalize beyond the data used for model selection.

Questions for authors
  1. Speech baseline implementation and controlled comparison. Appendix A.4 reports that three Speech CLS baselines apply cross-entropy to sigmoid outputs. It also describes differences in CNN-BiGRU readouts and dropout settings between CLS and LaS. Do the reported gains, particularly the 17.15-percentage-point improvement for CNN-BiGRU, persist with raw-logit cross-entropy, matched readouts and dropout, and comparable validation-based tuning budgets? Please also clarify how the original training configurations were selected.

  2. Contribution of language semantics. The closed-set P0–P1 differences are nonsignificant, and Proposition C.3 shows that a learned text projection can compensate for a fixed prompt permutation under the stated assumptions. How does LaS compare with otherwise matched models that use fixed random class vectors or learned class prototypes instead of pretrained text embeddings? This would help clarify whether semantic information contributes beyond the learned classifier parameterization and training objective.

  3. Independence of model selection and evaluation. The MI protocol uses the same held-out fold for checkpoint selection and final reporting, while SSVEP adaptation selects training epochs using the reported validation-block accuracy. How do the results change when checkpoint or epoch selection uses inner validation data and performance is measured on an independent test partition that is not used for model selection? In particular, do the MI improvements and the ranking of adaptation methods remain stable?

  4. Candidate-subset reuse versus classifier restriction. Figure 3 compares a reused four-class LaS model with CLS models trained separately for each subtask. How does LaS compare with the original four-class CLS model when its predictions are restricted to the same candidate subset by masking the other logits, without retraining? This would clarify whether the text-based candidate interface offers an advantage over simply restricting the original classifier to the same candidates.

  5. Cross-session adaptation on a common evaluation set. Figure 4b and Table B.11 compare direct inference on Base runs with adapted-model performance on later Fine runs. Can the frozen source model and adapted models both be evaluated on the same Fine runs? How much improvement remains under this matched evaluation, and does the ranking of adaptation methods change?

  6. What enables unseen-frequency transfer? Please clarify how the eight held-out frequency-class sets are constructed, including their frequency spacing. How does LaS compare with an otherwise matched model that uses explicit numerical frequency features instead of pretrained language embeddings? This comparison would help determine whether pretrained language embeddings provide additional value beyond a matched representation of numerical stimulus frequency.

11 comments
AuthorsOpenReview ↗

Response to zjiu (1/6): Speech baselines

Thank you for taking the time to review our paper and for recognizing the Speech gains across encoders and the distinction between closed-set fitting and unseen-frequency prediction.

In the additional experiments, all four Speech encoders retain positive mean gains over five runs against the CLS controls using raw logits; after separating checkpoint selection from evaluation for each MI fold, DeepConvNet improves by 3.06 pp. After using the common Fine data for the cross-session comparison, the gains from full fine-tuning are 9.69 and 6.95 pp for binary and three-class classification, respectively. Below, we address your concerns and questions in turn with the additional experiments and analyses. The MI classes are numbered consistently as follows: 1=thumb, 2=index finger, 3=middle finger, and 4=little finger.

1. Speech baseline implementation and controlled comparison

We understand that these implementation differences affect the Speech comparison, so we added raw-logit baselines and standardized the bidirectional readout for CNN-BiGRU.

With cross-entropy computed on raw logits, all four Speech encoders still retain positive mean gains. In addition to the official implementation, we retrained the four CLS baselines, using five different random seeds for each model. All four Speech CLS baselines compute cross-entropy directly on raw logits. We also changed CNN-BiGRU to concatenate the final hidden states from both directions of its last layer, using the same 256-dimensional readout as LaS.

EncoderUpdated CLS accuracy (%)LaS accuracy (%)LaS−CLS (pp)
EEGNet6.1215.46+9.33
CNN-BiGRU47.9553.47+5.52
EEG-Conformer50.6553.56+2.90
Du-IN58.8460.29+1.45

Both methods use the same set of five random seeds. The updated CLS accuracy for CNN-BiGRU is 47.95%. The revised manuscript will report the updated Speech comparison and update the related performance conclusions accordingly. The gain for CNN-BiGRU with raw-logit cross-entropy and the standardized bidirectional readout is 5.52 pp; the original 17.15 pp will remain labeled with its original output and readout configuration.

Across the five paired comparisons, the gains for EEGNet, CNN-BiGRU, and EEG-Conformer are all positive, falling in the ranges 8.74–10.28, 4.97–6.39, and 2.30–4.20 pp, respectively; Du-IN ranges from −0.10 to 2.10 pp, with positive gains in four of the five runs.

The original implementations of some Speech CLS baselines followed the output and loss path of the public Du-IN benchmark. For Speech models with an MLP classification head, the head combines neural features through nonlinear transformations to produce class scores, and the bottleneck layer also compresses the representation. LaS replaces the classification head at the neural-feature output with projection and text-prototype matching, mapping neural features to class scores. The attachment point is determined by the encoder's feature interface; we did not select or optimize the readout location.

Some dropout layers in CLS are within the MLP classification head, which LaS replaces with projection and text-prototype matching. Some dropout rates also differ between CLS and LaS in the Transformer backbones of Speech EEG-Conformer and Du-IN. The optimizers and learning-rate annealing strategies are largely consistent between the two methods, and each paired Speech CLS–LaS training setup uses the same weight-decay coefficient. CLS follows the configurations of public implementations where applicable, with adjustments based on the validation set when adaptation to the data is needed or a corresponding implementation is unavailable. LaS also selects its specific hyperparameters based on validation performance; the tuning effort is comparable between the two methods, and the specific configurations will be made public after the paper is accepted.

AuthorsOpenReview ↗

Response to zjiu (2/6): Text semantics: closed-set controls

2. Contribution of language semantics

Fixed random vectors and text differ by less than 1 pp in mean closed-set accuracy in each of the three paradigms; text also provides a general way to express class attributes and relationships.

We designed a new experiment in which an independent standard Gaussian vector of the same dimensionality as the language features is generated for each class, L2-normalized, and then fixed. We replace only the precomputed language features and retain the trainable projection WtW_t; the neural model, bidirectional InfoNCE loss, temperature, data splits, and training settings otherwise remain the same as in the text control. The random vectors before projection are fixed, while the class prototypes after projection continue to update during training.

Task and encoderConditionReal text (%)Fixed random vectors (%)Text−random (pp)
MI, EEG-ConformerFour classes, 1.0 s41.9441.39+0.55
SSVEP, MTSNet40 classes, 1.0 s, RW→IW66.7967.36−0.57
Speech, Du-IN61 classes, 3.0 s60.5859.79+0.79

For each paradigm, the fixed-random control reports the mean accuracy of experiments with three independent sets of random class vectors. MI uses the newly established five-fold protocol with held-out validation; 1.0 s is the model input window length, and trial predictions are obtained by averaging the window scores within each trial. SSVEP uses the corresponding conditions in Table 1 and aggregates results with equal weight for each of the 35 subjects.

Independent normalized Gaussian vectors are typically nearly orthogonal in high dimensions: the expected inner product between different classes is 0, and its expected square is 1/d1/d, so they can provide fixed, separated references for different classes. Let the random-vector matrix be R∈RC×dR\in\mathbb R^{C\times d}, where CC is the number of classes and dd is the input vector dimensionality. When C≤dC\leq d, RR has full row rank almost surely; given a target unit-prototype matrix Q∈RC×kQ\in\mathbb R^{C\times k}, where kk is the matching-space dimensionality, we can set

Wt(r)=R†Q,RWt(r)=Q.W_t^{(r)}=R^\dagger Q,\qquad RW_t^{(r)}=Q.

Under the full-row-rank condition above, the trainable projection can map fixed random vectors to given seen-class prototypes, providing a geometric explanation for the similar closed-set results. The compensation for seen-class permutations analyzed in Appendix C.1 also uses this expressive capacity of the projection.

AuthorsOpenReview ↗

Response to zjiu (3/6): Text semantics: unseen-frequency decoding

With the random-representation model fixed, introducing frequency adjacency only for unseen classes raises ZSL accuracy from 17.92% to 94.52%. Using MTSNet with 1-second EEG input, we compare independent random class vectors, real text, and a construction based on frequency adjacency. The three conditions use the same eight class splits and evaluation trials, with predictions among five unseen frequency classes in each split and a chance level of 20%. The specific frequencies and spacing for the eight splits are given in Response 6.

Within each class split, the two random-representation conditions reuse the same trained checkpoint, keeping both the neural branch and the projection fixed and changing only the input vectors for unseen classes at inference. The adjacency-based construction is performed before projection: for an interior unseen frequency, we average the initial unit random vectors of its neighboring seen classes on either side and normalize the result again. For example, 8.2 Hz uses the vectors for 8.0 and 8.4 Hz.

Class representation1.0 s ZSL accuracy (%)
Independent random class vectors17.92 ± 8.66
Real text90.89 ± 1.43
Random vectors constructed using frequency adjacency94.52 ± 0.83

The results are the mean accuracy and sample SD across the eight class splits, covering 35 subjects. The endpoints 8.0 and 15.8 Hz use the vectors of the nearest seen frequencies, 8.2 and 15.6 Hz, respectively.

Let ν\nu denote L2 normalization of a nonzero vector, and let r−r_- and r+r_+ be the initial unit random vectors of neighboring seen classes. For an interior unseen class uu, when the relevant vectors are nonzero,

t_u=\nu(r_uW_t)=\nu(r_-W_t+r_+W_t).$$ The adjacency-based construction encodes local frequency relationships in the unseen-class vectors, so their projection directions are determined by the trained directions of neighboring classes. When the neural representations vary smoothly with frequency locally and are compatible with this interpolation relationship, the model can use the matching relationships learned from seen classes to recognize unseen frequencies. Section 1 notes that one-hot targets identify classes without explicitly encoding class attributes or semantic relationships. Independently sampled fixed random inputs likewise do not encode these relationships in advance. The frequency-adjacency control adds relationships to the random inputs by construction, improving unseen-frequency accuracy by 76.61 pp with the same random-vector model. Text achieves 90.89% through frequency descriptions; together with the P0/P1 permutation results, these findings support the role of class relationships in this prediction task. Frequencies have an explicit numerical order, making adjacency rules straightforward to specify; more abstract class relationships may lack an equally direct vector construction. Natural language can describe class attributes and relationships directly, and a pretrained text encoder converts these descriptions into representations, allowing classes in MI, SSVEP, and Speech to use the same descriptive approach.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Contributions and Additional Evidence

We thank the chairs and all three reviewers for their assessment and suggestions.

Our core contributions are:

  • We propose LaS, which makes language supervision practical for classical neural decoders across motor, visual, and speech tasks, giving independently trained models a common way to learn and predict from class descriptions.
  • We reveal how text provides a general-purpose representation of class attributes and relationships across paradigms, enabling neural decoders to use these relationships for unseen-frequency prediction in SSVEP.
  • We explain the two alignment directions and their loss structure and show how neural activity and words contribute to the same prediction score, making language-supervised decoding more interpretable.

In their initial reviews, the reviewers highlighted:

  1. All three reviewers valued the broad evaluation across neural decoding tasks and model families.
  2. zjiu and nWja recognized consistent speech-decoding gains and useful insights into generalization to unseen frequencies.
  3. g8g8 valued the study’s insight into the method’s possible uses and scope.

To address the questions raised, we have:

  • Baseline comparisons: Added raw-logit Speech CLS baselines with paired seeds, matched CNN-BiGRU's bidirectional readout, and clarified readout, pooling, dropout, and optimization settings. All four Speech encoders retain positive mean gains.
  • Evaluation protocols: Separated model selection from evaluation for MI and SSVEP adaptation, evaluated frozen and adapted models on the same Fine trials, and restricted existing four-class CLS predictions to candidate subsets.
  • Class representations and prior methods: Added fixed-random controls, SSVEP frequency-adjacency and stimulus controls, same-split LEAF-FT, and MAE-initialized LaS against the published CLS reference.

Full results and protocols are provided in the individual responses and will be incorporated into the revised manuscript. We respectfully ask reviewers to reconsider their ratings in light of these contributions and evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.