acceptodds
Under review as a conference paper at ICLR 2027

From Minutes to Days: Scaling Intracranial Speech Decoding with Supervised Pretraining

Abstract

Decoding speech from brain activity has typically relied on limited neural recordings collected during short and highly controlled experiments. Here, we introduce a framework to leverage week-long intracranial and audio recordings from patients undergoing clinical monitoring, effectively increasing the training dataset size by over two orders of magnitude. With this pretraining, our contrastive learning model substantially outperforms models trained solely on classic experimental data, with gains that scale log-linearly with dataset size. Analysis of the learned representations reveals that, while brain activity represents speech features, its global structure largely drifts across days, highlighting the need for models that explicitly account for cross-day variability. Overall, our approach opens a scalable path toward decoding and modeling brain representations in both real-life and controlled task settings.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Pretraining, representation learning, speech decoding, neuroscience

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LFFgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper uses about a week of clinical iEEG and room audio from epilepsy patients to pretrain a speech decoder before finetuning it on a short audiobook listening task. The model follows Défossez et al. (2023) and uses a CLIP loss to match 3-second brain windows with averaged wav2vec 2.0 features. In three participants, pretraining on 84 to 108 hours of ambient data lowers the retrieval rank on the true audiobook test set compared with training only on task data, and the gain grows roughly log-linearly with the amount of pretraining data. The authors also report that zero-shot transfer is close to chance, that wav2vec targets beat mel spectrogram targets in two of three participants, and that the brain embeddings are strongly organized by recording day.

Critical strengths
  1. Using the week of clinical audio and iEEG is a smart idea that adds no extra burden on patients.

  2. Pretraining raises downstream performance over the task-only baseline in all three participants.

  3. The scaling curve from 10% to 100% of the pretraining data supports the claim that more ambient data helps.

  4. The finding that brain embeddings group strongly by recording day is useful for anyone building long-term iEEG decoders.

  5. The ambient audiobook condition helps separate audio domain shift from task shift, and it suggests the shift comes mostly from the audio.

  6. The appendix covers several useful controls, including linear vs deep pretraining, night-time data, ROI decoding, preprocessing choices and a PopT/BrainBERT baseline.

Critical weaknesses
  1. Absolute gains are small, since the baseline ranks are about 0.45 to 0.47 and the pretrained model ranks are about 0.38 to 0.42 against a chance level of 0.5.

  2. The test splits are random 30-second chunks from one continuous audiobook, so nearby chunks in train and test may share slow signal drift and story content, which can inflate scores.

  3. Signals are low-passed at 50 Hz and downsampled to 40 Hz, which removes high gamma, and Figure 7 shows that gamma features help two subjects, so the main pipeline may be weaker than needed.

  4. The Mann-Whitney U test treats each test window as an independent sample from a single training run, so it ignores dependence between windows and variation across random seeds.

  5. The scaling claim rests on six data fractions per subject, pooled across three subjects with different baselines, so the "no plateau" claim is weakly supported.

  6. Ambient audio includes the patient's own speech, which mixes speech perception with speech production and movement-related brain signals, and the paper does not deal with this.

  7. The paper does not explain how the camera audio and iEEG clocks were synchronized or whether any neural response lag was modeled, and both could strongly affect alignment quality.

Questions for authors
  1. How large is the retrieval set used to compute the relative rank, and what top-1 or top-10 accuracy do you get?

2, Why does Deep(Audiobook) for Subject 1 seem close to 0.49 in Figure 9 but about 0.45 in Figure 2A, if these are the same baseline?

  1. Do the results hold if the audiobook is split into contiguous blocks, for example by chapter, instead of random 30-second chunks?

  2. How many random seeds did you run, and how much do the results vary across seeds?

  3. How did you align the video audio with the iEEG in time, and did you check that alignment?

  4. What happens if you pretrain using only the segments where Whisper detects speech, or only segments without the patient's own voice?

  5. Does a model pretrained on one subject's ambient data help when finetuned on another subject's task data?

Reviewer_AbrmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the scarcity of neural recordings, the authors propose pretraining a model on data collected outside controlled experiments. They use neural recordings from three participants, along with ambient audio recorded in the rooms where the participants stayed. Although this audio was collected outside the controlled task, the recordings came from the same participants and were made using the same setup and in the same rooms. The controlled task involved listening to an audiobook, and retrieving the correct sentence from neural data only. The authors adapt a contrastive learning model to this setting: neural signals are processed by a deep convolutional network, while audio is represented using a pretrained wav2vec model. They compare a model pretrained on the ambient recordings with one trained from scratch, and show that pretraining significantly improves the model’s rank on the speech task. However, the pretrained model cannot directly retrieve the audiobook audio in a zero-shot setting. The authors suggest that this may be because the ambient audio and audiobook recordings come from different distributions, an interpretation they support with UMAP visualizations of the resulting clusters. Finally, they show that fine-tuning the pretrained model further improves its rank.

Critical strengths

I consider the finding that pretraining on data collected from the same subjects and recording setup, but outside the controlled experiment, improves retrieval rank to be important and potentially novel. Previous work has demonstrated transfer across subjects and modalities; this paper instead shows the value of using additional, naturally collected data within the same recording context. Although the paper does not introduce a fundamentally new framework, it offers a practical result that could be useful to the community.

Critical weaknesses

I am not fully convinced by the analysis supporting the claim of “log-linear scaling of task performance with pretraining data quantity.” Figure 2B reports the log linear fit across subjects, but subjects differ in the amount of pretraining data available. As a result, the significant negative slope could reflect between-subject variability rather than a consistent relationship between pretraining data quantity and performance.

I recommend testing this relationship with a mixed-effects model that accounts for subject-level variation, for example by including random effects for subjects, and assessing whether the slope is significantly below zero. Or alternatively, fitting it separately for each subject. Since this is one of the paper’s main findings, the analysis should establish that the relationship holds across subjects. A similar analysis is also appropriate for Figure 4A.

The week-long datasets seem very heterogeneous (periods of activity, periods of rest, and so on), the authors select the test and validation set randomly, I am concerned these are not a faithful representation of the full dataset. I recommend evaluating how representative the sampled sets are, or using a split that accounts for this temporal and behavioral variability.

I think it would be important to compare the raw neural signals with the learnt brain embeddings, in terms of the UMAP clustering, to check if indeed they are now more aligned to the wav2vec features after learning, since with the analysis in Figure 6 I am not convinced that the “brain module learns to extract global wav2vec features to some extent”. The Pearson correlation comparison for the brain embedding and raw signals is in the appendix (Fig. 14), but this information should be in the main text.

Questions for authors

Please, see weaknesses above.

Furthermore, for the zero-shot results, have the authors thought of a simple, light weight, transformation that could be applied from the ambient recordings to the true recordings, that would allow for zero shot?

Reviewer_4GNHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The manuscript describes a method for identifying (retrieving) perceived audio from intracranial EEG data (3 participants). The basic paradigm (published previously) is to jointly embed neural data with audio data via a pretrained (frozen) wav2vec model (for audio), a (learned) CNN for neural data, and a CLIP-style loss. The main innovation of this study is to (pre)train the network on data from the participant's entire (week-long) stay in the hospital, rather than merely from the ~2 hour task (listening to an audio book), which in this case is used for "fine-tuning." The authors show that pretraining this way improves retrieval performance on the audiobook data; that retrieval improves in the number of pretraining data; and that it likewise improves approximately logarithmically in the number of fine-tuning data. "Zero-shot" (no fine-tuning) retrieval doesn't work (Fig. 3), purportedly because the task (audiobook) audio is distributed very differently from the non-task (ambient) sounds.

Critical strengths

This is a novel approach to an important problem, and the main results are clearly established, with moderate but not tiny effect sizes and appropriate statistical testing.

  • The study makes use of clinical monitoring data which is currently discarded would be very useful
  • Pretraining data sizes are substantial (100, 108, and 84 hour for the three subjects)
  • The scaling curve shows that adding more pretraining data helps, and that the curve has plateaued.
  • Representational analysis of the features learned during pretraining is done on four different variables.
Critical weaknesses

The manuscript (and its title) describe the task as "speech decoding," but all the reported results are from an audio retrieval task. It is true that the study focuses on retrieving (perceived) audio from an audiobook, but the fact that this is speech audio plays no role in the results: no attempt is made to (e.g.) classify words or sentences. Indeed, the retrieved segments correspond to time-averaged embeddings of random three-second segments, so it seems no granularity below three seconds is possible with this paradigm.

This is unfortunate for several reasons. First, it makes the title (and presentation) misleading. Second, it is something of a low bar, since for several years, speech has been successfully decoded at the level of words and phonemes from intracranial data---admittedly ECoG and not sEEG. But even sEEG has underwritten successful word classification. For example, Zheng 2024 ["Du-IN"] showed not only this, but that pretraining on data collected outside of the task improves accuracy. (The authors cite this paper, but only as an example of architectural innovations.)

Third, and perhaps most crucially, it is not clear that this approach would help actual speech decoding: The ambient sounds are apparently quite different from speech (e.g., Fig. 3), so although training on them improves generic retrieval of sounds, it is not clear that it helps with speech content. Indeed, at a granularity of 3 seconds, the extracted features seem more likely to correspond to something like sound "texture." It seems doubtful that even the speech envelope can be recovered, let alone lexical items.

More generally, the approach of averaging representations over 3-second intervals seems more appropriate to semantic decoding (and has been used in this context). But this is at odds with the training paradigm, which uses sounds of all sorts; and with the use of wav2vec to embed audio.

Finally, the innovation of this study is not really machine learning, but in use of data; ICLR might not be the appropriate venue.

Questions for authors
  • How big is the retrieval set? can the authors quantify the number of bits the best-performing decoder provides? How about top-1 and top-k accuracy?
  • The preprocessing of the neural data is described as filtering below 50 Hz and then downsampling to 40 Hz. I assume the downsampling was carried out by truncation in the frequency domain (otherwise there would be aliasing). But then why first downsample to 50 Hz?
  • In any case, it is unusual to throw away the high-gamma signal (~70 - 150 Hz), which is typically more closely related to speech content. --I now see in supplement that the authors did consider high-gamma, and that it improves performance (Fig. 7). Why is this not used in the main text? How does the network trained from scratch perform with high gamma?
  • Pretraining can sometimes improve performance merely by initializing the weights to a better overall size (mean and variance), rather than extracting useful information from the pretraining data. The authors should measure performance if the weights of the pretrained network are first permuted after pretraining but before fine-tuning (so that the mean and variance stay fixed to the values acquired during pretraining, but the filters themselves are destroyed).
  • The authors attribute the visible structure in wav2vec embedding space colored by recording hour (Fig. 6, row 1, column 2) to the existence of "a period likely corresponding to patient rest." I assume they mean that the structure reflects what time of day the audiobook data happened to have occured--is this right? Are there other hospital/EMU sounds that vary by time of day?
  • Why are the wav2vec embeddings interpolated up from 50Hz to 120Hz?
  • Was the audio resampled to the sampling rate wav2vec 2.0 is pretrained on?
  • Fig 2B shows points below 1 hour of pretraining data for participants 2 and 3; however, the text says models were pretrained with 10%, 20%, ..., 100% pretraining data.
  • How was the checkpoint or stopping point chosen for each pretraining run (for each pretraining data size)?
  • Were other audio feature extractors tried as targets in the CLIP loss such as e.g. HuBERT?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.