- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper uses about a week of clinical iEEG and room audio from epilepsy patients to pretrain a speech decoder before finetuning it on a short audiobook listening task. The model follows Défossez et al. (2023) and uses a CLIP loss to match 3-second brain windows with averaged wav2vec 2.0 features. In three participants, pretraining on 84 to 108 hours of ambient data lowers the retrieval rank on the true audiobook test set compared with training only on task data, and the gain grows roughly log-linearly with the amount of pretraining data. The authors also report that zero-shot transfer is close to chance, that wav2vec targets beat mel spectrogram targets in two of three participants, and that the brain embeddings are strongly organized by recording day.
Critical strengths
-
Using the week of clinical audio and iEEG is a smart idea that adds no extra burden on patients.
-
Pretraining raises downstream performance over the task-only baseline in all three participants.
-
The scaling curve from 10% to 100% of the pretraining data supports the claim that more ambient data helps.
-
The finding that brain embeddings group strongly by recording day is useful for anyone building long-term iEEG decoders.
-
The ambient audiobook condition helps separate audio domain shift from task shift, and it suggests the shift comes mostly from the audio.
-
The appendix covers several useful controls, including linear vs deep pretraining, night-time data, ROI decoding, preprocessing choices and a PopT/BrainBERT baseline.
Critical weaknesses
-
Absolute gains are small, since the baseline ranks are about 0.45 to 0.47 and the pretrained model ranks are about 0.38 to 0.42 against a chance level of 0.5.
-
The test splits are random 30-second chunks from one continuous audiobook, so nearby chunks in train and test may share slow signal drift and story content, which can inflate scores.
-
Signals are low-passed at 50 Hz and downsampled to 40 Hz, which removes high gamma, and Figure 7 shows that gamma features help two subjects, so the main pipeline may be weaker than needed.
-
The Mann-Whitney U test treats each test window as an independent sample from a single training run, so it ignores dependence between windows and variation across random seeds.
-
The scaling claim rests on six data fractions per subject, pooled across three subjects with different baselines, so the "no plateau" claim is weakly supported.
-
Ambient audio includes the patient's own speech, which mixes speech perception with speech production and movement-related brain signals, and the paper does not deal with this.
-
The paper does not explain how the camera audio and iEEG clocks were synchronized or whether any neural response lag was modeled, and both could strongly affect alignment quality.
Questions for authors
- How large is the retrieval set used to compute the relative rank, and what top-1 or top-10 accuracy do you get?
2, Why does Deep(Audiobook) for Subject 1 seem close to 0.49 in Figure 9 but about 0.45 in Figure 2A, if these are the same baseline?
-
Do the results hold if the audiobook is split into contiguous blocks, for example by chapter, instead of random 30-second chunks?
-
How many random seeds did you run, and how much do the results vary across seeds?
-
How did you align the video audio with the iEEG in time, and did you check that alignment?
-
What happens if you pretrain using only the segments where Whisper detects speech, or only segments without the patient's own voice?
-
Does a model pretrained on one subject's ambient data help when finetuned on another subject's task data?