- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes NOEMA for naturalistic speech decoding from fMRI. It uses a temporal convolutional network to process seven consecutive fMRI frames and aligns the brain representations with pretrained text, audio, and speech embeddings using contrastive learning.
Experiments on Cephalonauts One and LeBel show that NOEMA outperforms several MLP and Ridge baselines in speech segment retrieval. Ablation studies investigate the contributions of temporal context, multimodal targets, and frame ordering. The method also shows improved retrieval performance when additional unseen distractors are included.
Critical strengths
-
NOEMA consistently outperforms the evaluated baselines across two naturalistic speech datasets and six participants. The results on held-out sessions and additional retrieval candidates provide useful evidence of within-participant generalization.
-
The paper includes controlled comparisons of different temporal window sizes and stimulus representations. The results show that both temporal modeling and the combination of text, audio, and speech features contribute to retrieval performance.
Critical weaknesses
The proposed model mainly combines standard temporal convolutions, pretrained multimodal embeddings, and CLIP-style contrastive learning. These components are well established, and the paper does not introduce a substantially new temporal modeling or cross-modal alignment mechanism.
Questions for authors
Can the authors evaluate retrieval using temporally separated candidates or hard negatives from nearby speech segments? This would help determine whether NOEMA can distinguish fine-grained linguistic content.
3 comments
Thank you for your detailed response and the additional experiments. While these results strengthen the experimental evaluation, my main concern regarding the limited methodological novelty remains. The approach still primarily combines established techniques without introducing a substantially new modeling or alignment mechanism. Therefore, I will maintain my original rating.
We thank the reviewer for the follow-up and are glad to hear that the additional experiments strengthened the evaluation. We agree that NOEMA builds on established components, and we do not claim a new alignment mechanism. However, we believe the findings of the paper are useful precisely because they rely on simple, reproducible components, and they indicate what future fMRI speech decoders should investigate. Thank you again for your review and feedback.
These components are indeed established. Our contribution is to combine them for naturalistic speech decoding from fMRI, which, to our knowledge, had not been done, and to show with controlled comparisons that this combination works: on two datasets, with held-out sessions and stimuli absent from training, NOEMA retrieves the correct segment among 2,000 candidates within the top 10 for more than 42% of queries.
The revision strengthens this evidence. NOEMA outperforms two additional seven-frame baselines: a Transformer with three times as many parameters, and DynaDiff (Careil et al., 2025). It reaches 42.7% Top-10@2000 accuracy on Cephalonauts One and 43.2% on LeBel, against 38.0% and 38.7% for the Transformer and 38.5% and 41.3% for DynaDiff (Figure 3, Section 5.1). A new ablation of ten architecture variants also shows which design choices matter (Appendix C, Figure 8): NOEMA is the most accurate of the ten, and it benefits from sharing weights across frames in both its spatial projection and its temporal filters.
We thank the reviewer for this suggestion. We ran both evaluations on Cephalonauts One (Appendix G, Figures 14 to 16).
Temporally separated candidates: we retrieve each query among candidates at least 10.4 s apart, which share no text or audio context (about 413 candidates). NOEMA reaches 30.3% Top-1 accuracy, against 26.5% for DynaDiff, 26.3% for Transformer, 22.8% for Context MLP and 17.9% for MLP, and it is the most accurate decoder in all 15 subject–fold pairs (Figure 15).
Hard negatives: we rank each target only against its overlapping neighbors in the same run (about nine candidates; chance 11.2%). NOEMA reaches 38.8% Top-1 accuracy, against 27.9–32.9% for the baselines. Against the adjacent segment alone, NOEMA ranks the target higher in 68.9% of pairs, against 62.6–64.5% for the baselines (chance 50%). NOEMA leads every baseline in all 15 subject–fold pairs on both measures (Figure 16).
NOEMA therefore distinguishes nearby speech segments better than every baseline.