acceptodds
Under review as a conference paper at ICLR 2027

Decoding Naturalistic Speech from fMRI with Temporal Modeling and Multimodal Alignment

Abstract

Reconstructing what a person perceives from their brain activity is a stringent test of whether neural recordings carry meaningful information. Naturalistic speech offers a rich testbed for this challenge: it unfolds over multiple timescales, but fMRI's noisy, temporally smeared measurements make this structure hard to recover. We introduce NOEMA, a temporal convolutional decoder that aligns windows of consecutive fMRI measurements with multimodal stimulus embeddings. These embeddings combine complementary text, audio, and speech representations of the same stimulus segment, enabling its retrieval directly from brain activity. On two naturalistic speech datasets, Cephalonauts One and LeBel, NOEMA retrieves the correct segment among 2,000 candidates within the top 10 for more than 42% of held-out queries. This substantially outperforms baseline decoders, an advantage that persists when the candidate pool is extended with thousands of distractors absent from training. Controlled ablations establish the contributions of temporal modeling and complementary stimulus representations.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI, brain decoding, naturalistic speech, temporal modeling, multimodal alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_eHakOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NOEMA for naturalistic speech decoding from fMRI. It uses a temporal convolutional network to process seven consecutive fMRI frames and aligns the brain representations with pretrained text, audio, and speech embeddings using contrastive learning.

Experiments on Cephalonauts One and LeBel show that NOEMA outperforms several MLP and Ridge baselines in speech segment retrieval. Ablation studies investigate the contributions of temporal context, multimodal targets, and frame ordering. The method also shows improved retrieval performance when additional unseen distractors are included.

Critical strengths
  1. NOEMA consistently outperforms the evaluated baselines across two naturalistic speech datasets and six participants. The results on held-out sessions and additional retrieval candidates provide useful evidence of within-participant generalization.

  2. The paper includes controlled comparisons of different temporal window sizes and stimulus representations. The results show that both temporal modeling and the combination of text, audio, and speech features contribute to retrieval performance.

Critical weaknesses

The proposed model mainly combines standard temporal convolutions, pretrained multimodal embeddings, and CLIP-style contrastive learning. These components are well established, and the paper does not introduce a substantially new temporal modeling or cross-modal alignment mechanism.

Questions for authors

Can the authors evaluate retrieval using temporally separated candidates or hard negatives from nearby speech segments? This would help determine whether NOEMA can distinguish fine-grained linguistic content.

3 comments

Weaknesses: "The proposed model mainly combines standard temporal convolutions, pretrained multimodal embeddings, and CLIP-style contrastive learning. These components are well established, and the paper does not introduce a substantially new temporal modeling or cross-modal alignment mechanism."

These components are indeed established. Our contribution is to combine them for naturalistic speech decoding from fMRI, which, to our knowledge, had not been done, and to show with controlled comparisons that this combination works: on two datasets, with held-out sessions and stimuli absent from training, NOEMA retrieves the correct segment among 2,000 candidates within the top 10 for more than 42% of queries.

The revision strengthens this evidence. NOEMA outperforms two additional seven-frame baselines: a Transformer with three times as many parameters, and DynaDiff (Careil et al., 2025). It reaches 42.7% Top-10@2000 accuracy on Cephalonauts One and 43.2% on LeBel, against 38.0% and 38.7% for the Transformer and 38.5% and 41.3% for DynaDiff (Figure 3, Section 5.1). A new ablation of ten architecture variants also shows which design choices matter (Appendix C, Figure 8): NOEMA is the most accurate of the ten, and it benefits from sharing weights across frames in both its spatial projection and its temporal filters.

Question 1: "Can the authors evaluate retrieval using temporally separated candidates or hard negatives from nearby speech segments? This would help determine whether NOEMA can distinguish fine-grained linguistic content."

We thank the reviewer for this suggestion. We ran both evaluations on Cephalonauts One (Appendix G, Figures 14 to 16).

Temporally separated candidates: we retrieve each query among candidates at least 10.4 s apart, which share no text or audio context (about 413 candidates). NOEMA reaches 30.3% Top-1 accuracy, against 26.5% for DynaDiff, 26.3% for Transformer, 22.8% for Context MLP and 17.9% for MLP, and it is the most accurate decoder in all 15 subject–fold pairs (Figure 15).

Hard negatives: we rank each target only against its overlapping neighbors in the same run (about nine candidates; chance 11.2%). NOEMA reaches 38.8% Top-1 accuracy, against 27.9–32.9% for the baselines. Against the adjacent segment alone, NOEMA ranks the target higher in 68.9% of pairs, against 62.6–64.5% for the baselines (chance 50%). NOEMA leads every baseline in all 15 subject–fold pairs on both measures (Figure 16).

NOEMA therefore distinguishes nearby speech segments better than every baseline.

Thank you for your detailed response and the additional experiments. While these results strengthen the experimental evaluation, my main concern regarding the limited methodological novelty remains. The approach still primarily combines established techniques without introducing a substantially new modeling or alignment mechanism. Therefore, I will maintain my original rating.

We thank the reviewer for the follow-up and are glad to hear that the additional experiments strengthened the evaluation. We agree that NOEMA builds on established components, and we do not claim a new alignment mechanism. However, we believe the findings of the paper are useful precisely because they rely on simple, reproducible components, and they indicate what future fMRI speech decoders should investigate. Thank you again for your review and feedback.

Reviewer_9q3KOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces NOEMA, a participant-specific decoder that maps a seven-frame fMRI window to a concatenation of frozen text, audio, and speech embeddings. A spatial projection and two residual dilated temporal-convolution blocks predict the embedding associated with the central frame; contrastive training supports retrieval of the corresponding speech segment from a candidate set. Evaluation uses three participants from Cephalonauts One and three from LeBel, with five session-based splits and no shared stimuli between training, validation, and test sets. NOEMA achieves mean top-10 retrieval of 42.7% and 43.6%, respectively, among approximately 2,000 candidates, and top-1 retrieval of 12.0% and 13.6%. Comparisons with frame-wise models and a seven-frame Context MLP, context-length and frame-order ablations, and expanded candidate sets support the utility of temporal modeling. The contribution is improved finite-candidate speech-segment retrieval, with explicit acknowledgment that this does not establish unrestricted speech reconstruction.

Critical strengths
  1. A convincing and carefully scoped empirical improvement. Session-disjoint evaluation avoids the obvious leakage risk from adjacent fMRI frames and overlapping stimulus contexts. The advantage appears across both datasets and all six participants, including against a Context MLP with the same seven input frames. Top-1 results, full rank distributions, and expanded candidate sets support a consistent gain beyond the headline top-10 metric (Sections 4-5; Appendices E and I). The authors also correctly restrict the result to within-participant retrieval and describe the decoder as noncausal.

  2. Useful controls and explicit implementation details. Context-length ablations and retraining with both shared and sample-specific frame permutations test different aspects of temporal organization. In particular, the shared-permutation control leaves Context MLP essentially unchanged while reducing NOEMA's accuracy, supporting the relevance of the convolutional inductive bias (Appendix G). Validation-only hyperparameter search is documented, the architecture is supplied as standalone code in the PDF, and subject-level uncertainty is distinguished from fold variability (Appendix B; Figures 3-4). Together these make the empirical contribution understandable and assessable.

Critical weaknesses
  1. The representation ablations leave the nature of modality complementarity unresolved. The cumulative text, text-plus-audio, and text-plus-audio-plus-speech comparisons change both the training targets and the cosine geometry used to rank candidates (Sections 3.3 and 5.2). This is legitimate for the stated segment-identification task, but it does not identify each modality's unique contribution or distinguish additional semantic information from improved acoustic or temporal discrimination of segments. Audio-only, speech-only, and text-plus-speech results are needed to establish how much the full combination improves over the strongest simpler representation. Evaluating a fixed text-space readout would further test the narrower semantic interpretation. This limits the explanatory contribution of the representation analysis, while leaving the main same-target decoder comparison intact; it is a fixable concern rather than a reason to discount the retrieval gain.
Questions for authors
  1. Clarification/control affecting interpretation: Are audio-only, speech-only, and text-plus-speech results available under the same decoder and validation protocol? Do they establish a benefit from combining all three representations over the strongest simpler target? If making a claim about improved semantic decoding, does that benefit persist under a fixed text-space retrieval metric?
  2. Clarification of experimental selection: Section 3.2 motivates the 2.5-second lag by its performance, while Appendix C explicitly says the test-session lag curves were not used to select it. How was the default lag originally fixed, and which data, if any, were used for that choice? This is a request to clarify the chronology, not an allegation of test-set tuning.
5 comments

Weaknesses: "The representation ablations leave the nature of modality complementarity unresolved. The cumulative text, text-plus-audio, and text-plus-audio-plus-speech comparisons change both the training targets and the cosine geometry used to rank candidates (Sections 3.3 and 5.2). This is legitimate for the stated segment-identification task, but it does not identify each modality's unique contribution or distinguish additional semantic information from improved acoustic or temporal discrimination of segments. Audio-only, speech-only, and text-plus-speech results are needed to establish how much the full combination improves over the strongest simpler representation. Evaluating a fixed text-space readout would further test the narrower semantic interpretation. This limits the explanatory contribution of the representation analysis, while leaving the main same-target decoder comparison intact; it is a fixable concern rather than a reason to discount the retrieval gain."

Question 1: "Clarification/control affecting interpretation: Are audio-only, speech-only, and text-plus-speech results available under the same decoder and validation protocol? Do they establish a benefit from combining all three representations over the strongest simpler target? If making a claim about improved semantic decoding, does that benefit persist under a fixed text-space retrieval metric?"

We thank the reviewer for this analysis. Audio-only, speech-only, and text-plus-speech results are now available: we trained NOEMA, Context MLP, and two new strong baselines added to the paper, DynaDiff and Transformer (Section 4.2), on all seven combinations of text (T), audio (A), and speech (S) targets, with the same decoder and validation protocol (Appendix H.2, Figure 18). The table below reports all seven combinations on Cephalonauts One; the results on LeBel are in the paper (Figure 18).

Top-10@2000 accuracy (%) on Cephalonauts One, mean over 3 subjects × 5 folds. Best per column in bold.

ModelTAST+AT+SA+ST+A+S
Context MLP22.722.58.729.130.728.833.3
DynaDiff26.626.612.132.635.434.038.5
Transformer27.326.09.933.333.932.038.0
NOEMA27.627.31.335.238.837.942.7

Accuracy increases with the number of representations, for every model: each single representation is less accurate than any combination of two, and the combination of all three is the most accurate. It improves on the strongest simpler target for every model and on both datasets, by 1.7 to 4.1 percentage points. For NOEMA, the full target reaches 42.7% on Cephalonauts One, against 38.8% for T+S, and 43.2% on LeBel, against 40.8% for A+S. Notably, speech performs poorly as a target on its own (1.3% to 12.1%), yet it improves decoding when combined with the other representations: adding it to text and audio raises accuracy by 4.2 to 7.5 points, for every model.

Question 2: "Clarification of experimental selection: Section 3.2 motivates the 2.5-second lag by its performance, while Appendix C explicitly says the test-session lag curves were not used to select it. How was the default lag originally fixed, and which data, if any, were used for that choice? This is a request to clarify the chronology, not an allegation of test-set tuning."

We took δ=2.5\delta = 2.5 s from the Cephalonauts One benchmark (Collas et al., 2026) and used it unchanged for both datasets and all models. We did not tune it for any of the models or datasets, so no data from this paper were used for that choice. We clarified this in Section 3.2, which no longer motivates the value by its performance. The sweep in Appendix D (Figure 9) serves as a sensitivity analysis: NOEMA's Top-10@2000 accuracy stays within 10% of its value at the default offset for offsets from −1.25 to 4 s, compared with 1 to 3.25 s for the MLP.

Reviewer_9q3KOpenReview ↗

What the multimodal targets add to retrieval

Thank you for testing all seven combinations of text, audio and speech targets. On both datasets, the full target beats the best pair for every decoder, by 1.7–4.1 percentage points. That answers my question about whether all three targets help identify the segment. The hard-negative and non-overlapping candidate analyses strengthen that result. Using the 2.5 s lag from the Cephalonauts One benchmark also resolves my concern about how the lag was chosen.

One result puzzles me. With speech as the sole target on Cephalonauts One, NOEMA reaches 1.3% top-10 accuracy, compared with 8.7–12.1% for the other decoders. Yet adding speech to text and audio helps NOEMA substantially. Was the low speech-only score consistent across participants and folds? Did you see any signs of difficulty fitting that target?

I still have a question about the semantic interpretation. Changing the targets also changes the distances used to rank the candidates, so acoustic or timing cues could account for the gain. For the conditions that include text, could you rank the same candidates using only the predicted text component and the same frozen text embeddings in every condition? If that comparison is unavailable, I would limit the claim to improved segment retrieval. I am keeping my rating at 3.

One result puzzles me. With speech as the sole target on Cephalonauts One, NOEMA reaches 1.3% top-10 accuracy, compared with 8.7–12.1% for the other decoders. Yet adding speech to text and audio helps NOEMA substantially. Was the low speech-only score consistent across participants and folds? Did you see any signs of difficulty fitting that target?

Thank you for noticing this in the results. We checked every subject and fold. On Cephalonauts One, NOEMA stays at chance level (0.5–0.8% Top-10, chance ≈ 0.5%) in 14 of 15 subject–fold pairs. One likely cause is the contrastive temperature. All models use hyperparameters tuned on the full T+A+S target (as stated in Appendix B.2), and the speech targets are much more similar to one another than the text and audio targets, so small prediction errors are strongly amplified at NOEMA's tuned temperature (0.006). Retraining NOEMA on the speech-only target with a temperature of 0.02, all else unchanged, makes training succeed in all 15 pairs and raises Top-10 accuracy to 10.5% (range 8.0–14.2%), similar to the other decoders (8.7–12.1%). We chose this temperature to diagnose the failure rather than by validation, so we do not report it as a tuned result. The other decoders were tuned to higher temperatures (0.008–0.009), which likely leaves them less sensitive to this effect. When speech is combined with text or audio, these less similar targets drive the alignment, and adding speech improves retrieval in every subject and fold. We added this analysis to Appendix H.2.

Reviewer_ByN8OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This interesting paper tackles decoding of (perceived) naturalistic speech from fMRI, framed as retrieving the "correct segment" from about 2k candidates in a held-out data partition. The proposed model, NOEMA, takes a seven-frame fMRI window, projects each frame with shared weights, and mixes frames with two dilated temporal convolution blocks. Only the output at the central time position, which depends on all seven input frames, is mapped to the target embedding space, making the decoder non-causal over roughly ±6s (long time!) Targets are frozen embeddings from text, audio and speech encoders, normalised and concatenated (symmetric CLIP-style contrastive loss applied). Evaluation uses two listening datasets, Cephalonauts One & and LeBel (which I needed to look up and I admit a better understanding of these datasets would have helped me greatly, but didn't have time to investigate to the extent I would have liked).

NOEMA reaches around 43% top10@2000 accuracy, beating Ridge, single-frame MLP, MindEye-style MLP and a seven-frame Context MLP, Ablations (good to see!) show NOEMA gains more than the Context MLP from additional frames, relies on chronological frame order, and improves as audio and speech targets are added to text. The advantage persists when the candidate pool is extended with thousands of segments from unseen podcast episodes, and attribution maps highlight frontotemporal language regions (but, sadly, only provided for one fold of one subject, no null model or significance testing, so I don't know how much we can really trust this).

Critical strengths

The biggest issue I see in NeuroAI papers is a failure to control for baselines and a lack of investigation of all the failure modes other papers have fallen into and a failure to learn from those lessons. The main strengths in this paper, from what I can see, is that, to a moderate extent, this has happened. I wouldn't say it's flawless in that regard, but certainly leads me to more to trusting that that the authors are able to show their results are not (obviously) spurious. I say this in the context of an analysis that feels like it belongs more in the 2021-2023 era though, as I'll explain in the weaknesses below. The other is the production of the PyTorch code for reproducibility and candid assessment of their own weaknesses. I'm usually very forgiving if authors are upfront about this when it comes to novelty of claims.

Critical weaknesses
  • References and citations seem sparse - a lot has been done in this area and while some major works are cited, it feels like a very thin layer that reflects maybe an incomplete / non-comprehensive familiarity of related works in the area on the part of the authors
  • The related work feels like a patchwork of a bundle of related papers with sometimes only a thin thread linking them together, not really telling the story as emergent over time in a form of a developmental progression as I would have expected
  • There are citations to important references but no acknowledgement of the main processes of analysis. For example, the way that Huth and friends would deal with this is to resample the stimuli and have FIR delays incorporated into the analysis, while this boxcar-style motivation is more similar to the early fMRI alignment work in Bingel et al. (2017), which was fine for 2017 but not so much for 2026. I’d be happy to accept the motivation for this analysis if the more recent and standard one was recognised and argued against, but that doesn’t happen and it’s a gap that I must assume is due to the authors’ familiarity with how this is done in a state of the art way
  • The MLP and Ridge baseline only takes a central frame, right? While the other models see 7 frames? But the abstract and other parts of the paper often talk about “outperforming the baselines” - isn’t this unfair as they have seen fewer data points and have an impoverished signal to work with? I guess the Context MLP is contrasted, which does have the same input, but e.g. in Appendix C NOEMA is contrasted against just the MLP, and for conclusions to be drawn there, we’d expect the same inputs not 1-frame versus multiple, so I feel there is an issue here.
  • Separate linear projections in the Context MLP exist, while there’s a strong element of weight sharing in the NOEMA model, which we know can serve as a good regulariser. Therefore, conclusions made on the fact that temporal convolutions were used can’t be easily disentangled from the multiple differences between the two models under comparison. Applying shared-weight version without temporal convolution would disentangle the contributions and be a stronger result to present
Questions for authors
  1. I guess the only question I wanted to ask was why they didn't go down a route of using more normalised methods of looking at continuous stimuli and FIR methods and resampling and aligning, rather than using the fMRI blocks and coarsely linking them to the stimuli?
  2. How much do near-duplicate neighbours drive top10 accuracy? Text and audio targets use a 10s trailing window but markers are spaced about 2s apart, so adjacent candidates share about 80% of their content. A “hit” in the top 10 may be the true segment’s immediate neighbours ranking alongside it, and a miss may be a neighbour ranking first. How often is the top-ranked candidate a temporal neighbour of the target?
5 comments

Weakness 1: "References and citations seem sparse - a lot has been done in this area and while some major works are cited, it feels like a very thin layer that reflects maybe an incomplete / non-comprehensive familiarity of related works in the area on the part of the authors"

Weakness 2: "The related work feels like a patchwork of a bundle of related papers with sometimes only a thin thread linking them together, not really telling the story as emergent over time in a form of a developmental progression as I would have expected"

We thank the reviewer for this feedback. We rewrote the related work (Section 2) so that it follows a progression instead of listing papers.

The first paragraph now goes from the definition of decoding models (Naselaris et al., 2011), to the decoding of isolated words and sentences, and then to continuous language with naturalistic listening experiments and datasets (Huth et al., 2016; LeBel et al., 2023; Collas et al., 2026).

The paragraph on temporal integration now starts from why information about a speech segment is spread over several fMRI frames: cortical regions integrate language over windows ranging from words to paragraphs (Lerner et al., 2011), and the hemodynamic response spreads each event over several seconds (Glover, 1999). It recalls that encoding models handle this with FIR filters (Huth et al., 2016; LeBel et al., 2023). It then sorts decoders into two groups: those that combine frames with fixed weights (Bingel et al., 2016; Jalouzot et al., 2025; Collas et al., 2026), and those that learn how to combine frames, such as DynaDiff (Careil et al., 2025) and others (Xi et al., 2023; Chen et al., 2025; Lu et al., 2025). NOEMA belongs to the second group.

Weakness 3: "There are citations to important references but no acknowledgement of the main processes of analysis. For example, the way that Huth and friends would deal with this is to resample the stimuli and have FIR delays incorporated into the analysis, while this boxcar-style motivation is more similar to the early fMRI alignment work in Bingel et al. (2017), which was fine for 2017 but not so much for 2026. I’d be happy to accept the motivation for this analysis if the more recent and standard one was recognised and argued against, but that doesn’t happen and it’s a gap that I must assume is due to the authors’ familiarity with how this is done in a state of the art way"

Question 1: "I guess the only question I wanted to ask was why they didn't go down a route of using more normalised methods of looking at continuous stimuli and FIR methods and resampling and aligning, rather than using the fMRI blocks and coarsely linking them to the stimuli?"

We thank the reviewer for pointing out that our temporal alignment needs clearer positioning relative to established encoding approaches. We now acknowledge FIR-based encoding approaches explicitly in the related work (Section 2), where we also cite Bingel et al. (2016).

Encoding models of naturalistic language, such as those of Huth et al. (2016) and Tang et al. (2023), resample word-level features onto the acquisition grid and fit finite impulse response (FIR) filters over four delays (2, 4, 6 and 8 s) to model their delayed contribution to the BOLD response. A decoder faces the same delay, but on the brain side. NOEMA does not rely on a single frame at a fixed lag, nor on a boxcar average or an assumed hemodynamic response. It keeps seven individual frames, spanning about −4 to +9 s around the end of each stimulus segment, and learns how to combine them across temporal offsets. This is a learned, nonlinear analogue of FIR delays, applied to the fMRI input rather than to the stimulus features.

Our targets also differ from resampled word features. We compute segment-level embeddings from stimulus windows that end at time markers placed on a fixed grid, independent of the stimulus content and modality, rather than interpolating an irregular sequence of word embeddings. With this grid, sample boundaries never depend on word or sentence onsets. When samples are aligned to word onsets, their relative timing reveals word durations, which a decoder can exploit without using brain activity (Jayalath and Parker Jones, 2026). The fixed grid avoids this.

Our experiments evaluate this direct decoding formulation.

Weakness 4: "The MLP and Ridge baseline only takes a central frame, right? While the other models see 7 frames? But the abstract and other parts of the paper often talk about “outperforming the baselines” - isn’t this unfair as they have seen fewer data points and have an impoverished signal to work with? I guess the Context MLP is contrasted, which does have the same input, but e.g. in Appendix C NOEMA is contrasted against just the MLP, and for conclusions to be drawn there, we’d expect the same inputs not 1-frame versus multiple, so I feel there is an issue here."

Yes: Ridge, MLP and MindEye MLP take the central frame only, while Context MLP and NOEMA take seven frames (Section 4.2). We agree that conclusions about the architecture require baselines with the same input. The revised paper now has three of them: Context MLP and two new seven-frame baselines, DynaDiff and a Transformer, all tuned with the same protocol. NOEMA remains the most accurate on both datasets, with 42.7% Top-10@2000 accuracy on Cephalonauts One and 43.2% on LeBel, against at most 38.5% and 41.3% for the seven-frame baselines, and it outperforms the strongest of them in 14/15 and 15/15 subject–fold pairs of Cephalonauts One and LeBel, respectively (Figure 3, Section 5.1). The analyses of temporal context, stimulus representations, frame order and overlapping candidates are now made against these seven-frame baselines (Section 5.2, Appendices F to I).

The lag analysis (Appendix D in the revised paper) is a sensitivity analysis of the alignment offset, not an architecture comparison. It shows that the default offset suits both a one-frame and a seven-frame model. As the appendix notes, NOEMA’s lower sensitivity to the offset is expected from its seven-frame input, which spans about ±6 s around the central frame.

Weakness 5: "Separate linear projections in the Context MLP exist, while there’s a strong element of weight sharing in the NOEMA model, which we know can serve as a good regulariser. Therefore, conclusions made on the fact that temporal convolutions were used can’t be easily disentangled from the multiple differences between the two models under comparison. Applying shared-weight version without temporal convolution would disentangle the contributions and be a stronger result to present"

We thank the reviewer for this suggestion. We added the proposed baseline (Appendix C, Figure 8, “No TCN blocks, dense aggregation”): it keeps NOEMA’s shared spatial projection, applied to each of the seven frames, and replaces the temporal convolutions with a single dense layer over the projected frames.

This baseline reaches 40.5% Top-10@2000 accuracy on Cephalonauts One, below NOEMA’s 42.7% (−2.2 ± 0.5 points, 90% confidence interval across subjects), and NOEMA is more accurate in 13 of 15 subject–fold pairs. This shows that the temporal convolutions improve retrieval beyond what weight sharing alone provides. The baseline is itself well above Context MLP (33.3%), which shares no weights across frames, so both the shared projection and the temporal convolutions contribute.

We revised Sections 5.1 and 5.2 and the conclusion (Section 6) accordingly. Appendix C also reports eight other variants, which change how frames are aggregated, the number of TCN blocks, their dilations, and their kernel size. NOEMA is the most accurate of the ten.

Question 2: "How much do near-duplicate neighbours drive top10 accuracy? Text and audio targets use a 10s trailing window but markers are spaced about 2s apart, so adjacent candidates share about 80% of their content. A “hit” in the top 10 may be the true segment’s immediate neighbours ranking alongside it, and a miss may be a neighbour ranking first. How often is the top-ranked candidate a temporal neighbour of the target?"

We thank the reviewer for this question. We added a new study, “Retrieval among overlapping candidates” (Appendix G, Figures 14 to 16), on Cephalonauts One.

Across the five decoders, an overlapping neighbour takes the first rank in 11.1–16.7% of queries, against 0.4% for a random ranking (Figure 14). These neighbours account for many top-1 errors but contribute little to top-10 accuracy. Compared with random retrieval sets of the same size, removing overlapping candidates raises mean Top-1 by 1.3–2.8 percentage points and changes mean Top-10 by less than 0.5 percentage points (Figure 15).

NOEMA separates the target from its neighbours best. Among all ≈2,000 test candidates, when its top-ranked candidate lies within ±4 markers of the target, it is the target in 46.7% of queries, against 34.4–40.3% for the baselines. When each query’s retrieval set contains only the target and its neighbours within ±4 markers (about 9 candidates; chance 11.2%), NOEMA reaches 38.8% Top-1, against 27.9–32.9% for the baselines (Figure 16). The ranking of decoders is unchanged with or without overlapping candidates.

Representative results are summarized in the table below. Values show means over three subjects (five folds each). Random sets match each query’s non-overlapping set size and average 200 draws.

Accuracy (%)MLPContext MLPDynaDiffTransformerNOEMA
Top-1, random set (≈413, neighbours included)16.520.824.323.528.1
Top-1, non-overlapping set (≈413)17.922.826.526.330.3
Top-10, random set (≈413, neighbours included)50.859.863.963.667.7
Top-10, non-overlapping set (≈413)51.360.164.264.168.0
Top-1, target vs. its ±4 neighbours only (≈9; chance 11.2%)27.929.232.928.638.8
Reviewer_PuAnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces NOEMA, a temporal convolutional decoder that processes 7 consecutive fMRI frames containing cortical features, in order to produce a multimodal representation of a 10s window from the audio heard by the participant, obtained by concatenation of separately normalized representations from text, audio and speech encoders with the audio and text encoders processing 10s of audio and 10s of transcribed text, while the speech encoder operates on audio from last 2s of the window.
NOEMA applies a spatial projection - a shared linear projection to each frame, followed by GELU activation, dropout and 2 residual temporal convolution blocks with dilated convolutions, followed by a linear projection that converts the hidden representation of the middle of the 7 frames to the stimulus' vector representation. The model is trained using CLIP style loss to increase the cosine similarities between predictions and the matched target representations relative to unmatched pairs.

The target representations are chosen such that the vector representation obtained for stimulus ending at t corresponds to a 7 frame fMRI window centered at t + 2.5 seconds, to account for hemodynamic delay. The sensitivity to delay/offset values is evaluated by fitting and testing across different values.

Across two datasets Cephalonauts One and LeBel, 3 subjects each, and held-out sessions, NOEMA gets about 43% top-10 retrieval using cosine similarity, among about 2,000 clips, compared with about 34–36% for a 7-frame Context MLP trained on the same data. Training and evaluation are done using recordings involving the same participant.

Critical strengths

NOEMA consistently outperforms the evaluated baselines across both datasets and participants, using the same target representations and session splits. The paper supports these gains with substantial empirical analysis covering input frame count, stimulus-representation combinations, frame permutations, training-data scale, and retrieval-pool size, alongside more limited studies of alignment lag and speech window duration. Its strength lies in applying a straightforward temporal convolutional architecture to fMRI speech retrieval and systematically examining when and how the resulting gains arise.

Critical weaknesses

The baselines mentioned in the paper include Ridge Regression and architectures involving MLP layers only. There are no evaluations involving transformer based architectures, though there is literature involving transformers being used for fMRI decoding. and examples including UniCoRN and BP-GPT that are cited in the paper. A comparison with an architecture that uses a transformer across the fMRI frames, would strengthen the baselines section. Furthermore, a baseline that studies the importance of the temporal convolution blocks, such as context MLP along with the shared, per frame projection that is applied in NOEMA, is missing. Ablations concerned with dilation have not been studied.

Questions for authors
  1. Could the authors include into the baselines, an architecture involving a transformer operating over the projected fMRI frames, that uses positional encoding and evaluate it alongside NOEMA and context MLP across the reported comparisons ? trained under the same data, targets, loss and session splits.

  2. What was the exact reasoning behind the final reported value of 2.5s for the lag offset? Is the same value suitable for the LeBel dataset as well?

  3. Could the authors perform a T->T+S experiment, in addition to the existing T->T+A and T->T+A+S experiments to assess the contributions of Audio(A) and Speech(S) features in more detail ? This would show the importance of the longer Audio(A) features in comparison to the short context Speech(S) features.

4 comments

Weaknesses: "The baselines mentioned in the paper include Ridge Regression and architectures involving MLP layers only. There are no evaluations involving transformer based architectures, though there is literature involving transformers being used for fMRI decoding. and examples including UniCoRN and BP-GPT that are cited in the paper. A comparison with an architecture that uses a transformer across the fMRI frames, would strengthen the baselines section. Furthermore, a baseline that studies the importance of the temporal convolution blocks, such as context MLP along with the shared, per frame projection that is applied in NOEMA, is missing. Ablations concerned with dilation have not been studied."

Question 1: "Could the authors include into the baselines, an architecture involving a transformer operating over the projected fMRI frames, that uses positional encoding and evaluate it alongside NOEMA and context MLP across the reported comparisons? trained under the same data, targets, loss and session splits."

We thank the reviewer for these suggestions. We added the two baselines and the dilation ablation to the paper.

First, we added a Transformer baseline, built as requested (Section 4.2, Appendix B.2). It operates over the projected fMRI frames: it applies NOEMA’s shared spatial projection to each frame, adds learned positional embeddings, and predicts from the central frame after one self-attention block. It is trained on the same data, targets, loss and session splits as NOEMA, with the same hyperparameter search. This search selected a hidden width of 8,192, so the Transformer has 776.1M parameters, three times as many as NOEMA (253.8M). It reaches 38.0% Top-10@2000 accuracy on Cephalonauts One and 38.7% on LeBel, against 42.7% and 43.2% for NOEMA, which is more accurate in all 15 subject–fold pairs of each dataset (Figure 3, Section 5.1). It is included in all the reported comparisons: Top-1 accuracy and rank metrics (Appendix F), stimulus representations and temporal context (Section 5.2, Appendix H), frame order (Appendix I), and overlapping candidates (Appendix G).

Second, we added the proposed baseline for the temporal convolution blocks (Appendix C, Figure 8, “No TCN blocks, dense aggregation”). Like NOEMA, it applies the same spatial projection to each of the seven frames. Unlike NOEMA, it has no temporal convolution: a single dense layer combines the seven projected frames. It therefore sits between Context MLP, which shares no weights across frames, and NOEMA. On Cephalonauts One, sharing the spatial projection raises Top-10@2000 accuracy from 33.3% (Context MLP) to 40.5%, with 237M instead of 623M parameters. Adding the temporal convolutions, whose filters are also shared across frames, raises it further to 42.7% (NOEMA), a gain of 2.2 ± 0.5 points (90% confidence interval across subjects), observed in 13 of 15 subject–fold pairs.

Finally, we evaluated eight other variants of NOEMA to ablate the effect of each architectural choice: how frames are aggregated, the number of TCN blocks, their dilations, and their kernel size (Appendix C, Figure 8). NOEMA is the most accurate of the ten variants. In particular, removing dilation lowers accuracy from 42.7% to 39.1% (−3.6 ± 1.7 points, in all 15 subject–fold pairs). Using three TCN blocks (40.8%) or a single block with kernel size 7 (41.1%) does not improve on NOEMA either. We revised Sections 5.1 and 5.2 and the conclusion (Section 6) to describe NOEMA’s advantage in terms of these two forms of weight sharing.

Question 2: "What was the exact reasoning behind the final reported value of 2.5s for the lag offset? Is the same value suitable for the LeBel dataset as well?"

We took δ=2.5\delta = 2.5 s from the Cephalonauts One benchmark (Collas et al., 2026) and used it unchanged for both datasets and all models. We did not tune it for any of the models or datasets. We clarified this in Section 3.2 of the paper. The sweep in Appendix D (Figure 9) serves as a sensitivity analysis: NOEMA's Top-10@2000 accuracy stays within 10% of its value at the default offset for offsets from −1.25 to 4 s, compared with 1 to 3.25 s for the MLP.

Question 3: "Could the authors perform a T->T+S experiment, in addition to the existing T->T+A and T->T+A+S experiments to assess the contributions of Audio(A) and Speech(S) features in more detail ? This would show the importance of the longer Audio(A) features in comparison to the short context Speech(S) features."

We thank the reviewer for this suggestion. We ran the T→T+S experiment, together with all seven combinations of text (T), audio (A), and speech (S) targets, for NOEMA, Context MLP, DynaDiff, and Transformer (Appendix H.2, Figure 18). The table below reports the combinations of two or three representations on Cephalonauts One; the single-representation targets and the results on LeBel are in the paper (Figure 18).

Top-10@2000 accuracy (%) on Cephalonauts One, mean over 3 subjects × 5 folds. Best per column in bold.

ModelT+AT+SA+ST+A+S
Context MLP29.130.728.833.3
DynaDiff32.635.434.038.5
Transformer33.333.932.038.0
NOEMA35.238.837.942.7

Given text, speech adds more than audio: every model is more accurate with T+S than with T+A on both datasets. NOEMA reaches 38.8% with T+S against 35.2% with T+A on Cephalonauts One, and 40.2% against 31.3% on LeBel. Audio still adds information on top: the full T+A+S target is the best for every model, and raises NOEMA to 42.7% and 43.2%. NOEMA is the most accurate model whenever speech is combined with text or audio.

Reviewer_PuAnOpenReview ↗

Reply to the rebuttals

I thank the authors for providing relevant empirical results in response to my review. The authors have added ablations regarding dilation and other architectural choices in the TCN blocks of NOEMA, evaluated a single-block Transformer baseline with significantly more parameters than NOEMA, and examined the effects of speech (S) and audio (A) representations on retrieval performance. My concerns and questions have been substantially addressed. However, the effect of the number of blocks in the Transformer baseline remains unexplored. I will therefore increase my score to 3.

We thank the reviewer for their careful review of the paper and constructive feedback. We are glad that the architecture controls and the new ablations addressed your concerns. Indeed, the new experiments strengthen the paper and its claims. Regarding the number of Transformer blocks, a single self-attention block already attends over all seven frames, so it receives the same temporal context as NOEMA, and we tuned its width instead (Appendix B.2). We agree that depth is a natural next step to examine and we will explore that direction in our next steps.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall comment

We thank the reviewers for their careful reading and constructive suggestions. We revised the paper accordingly; all changes are marked in green in the new PDF. The main additions are:

  1. Two new strong baselines (Section 4.2, Figure 3): a Transformer over the projected fMRI frames and DynaDiff (Careil et al., 2025). Both take the same seven frames as NOEMA and use the same data, targets, loss, session splits and hyperparameter search. NOEMA remains the most accurate decoder: it reaches 42.7% Top-10@2000 accuracy on Cephalonauts One and 43.2% on LeBel, against 38.0% and 38.7% for the Transformer and 38.5% and 41.3% for DynaDiff, and it outperforms the strongest baseline in 14/15 and 15/15 subject–fold pairs of Cephalonauts One and LeBel, respectively (Section 5.1). Both baselines are included in all the analyses of the paper (Section 5.2, Appendices F to I).
  2. An ablation of NOEMA’s architecture (Appendix C, Figure 8): ten variants that change how frames are aggregated, remove the TCN blocks, or change their dilations and kernel size. NOEMA is the most accurate of the ten.
  3. All seven combinations of text, audio, and speech targets for NOEMA and the three seven-frame baselines (Appendix H.2, Figure 18). The combination of all three is the best target for every model on both datasets.
  4. Retrieval among overlapping candidates (Appendix G, Figures 14 to 16): retrieval sets without overlapping candidates, and hard negatives from nearby segments. NOEMA is the most accurate decoder in all 15 subject–fold pairs in both settings.
  5. A rewritten related work (Section 2), which now follows a progression, acknowledges FIR-based encoding models, and adds five references.

The conclusions of the original submission hold with these stronger baselines and additional controls.

Beyond the architectural comparison, our central contribution is a systematic study of how to build a decoding pipeline for continuous speech from fMRI. The experiments show the value of temporal context and complementary stimulus representations, while the architecture controls help disentangle the contributions of shared spatial projections and temporal modeling. We expect the best architectural choice to continue evolving with the amount of available data and advances in decoding methods. These findings provide empirical guidance for that evolution by identifying which pipeline choices contribute to retrieval performance under controlled comparisons.

We answer each reviewer’s points in the individual replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.