acceptodds
Under review as a conference paper at ICLR 2027

Synthesizing Speech from Intracranial Recordings with Conditional Flow Matching

Abstract

The task of synthesizing speech from neurological data has advanced significantly over the last five years, through a combination of higher-quality data and modern machine-learning techniques. To date, all such approaches have been based on discriminative modeling, i.e. building a deterministic map from neural data to speech audio (e.g., the spectrogram). However, there is reason to believe such models are unnecessarily restrictive. Inspired by the recent evolution in text-to-speech from discriminative to generative models, we reinterpret speech synthesis as a conditional-generation problem. We use conditional flow matching to learn a velocity field that transports a standard normal distribution to the distribution of log-mel spectrogram features, with velocity predictions conditioned on the corresponding neural data by way of a convolutional encoder. Discriminative modeling emerges as the special case in which the model is trained to predict only the initial velocity. We demonstrate our approach on (1) a set of stereo-EEG/speech audio recordings recently made by our group from two participants, and (2) a public data set of microelectrode recordings (single units) made from a person with severe dysarthria. In both cases, our conditional flow outperforms all competitors, in the latter case bringing word error rates of (transcribed) synthesized speech down from about 56% to 22%. Our model also requires about one fourth as many training trials to achieve comparable intelligibility. Overall, we show that conditional flow matching is a powerful approach to neural speech synthesis for datasets ranging from small (sEEG, 0.5 hours) to large (single units, 38 hours).

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain Machine Interfaces, Conditional Flow Matching, Neural Speech Synthesis, StrereoEEG, Utah Array

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rXryOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adapts conditional flow matching to synthesize acoustic features from intracranial neural recordings. A temporal CNN conditions a DiT velocity model, which is sampled with an ODE solver and vocoded to speech. The method is evaluated on two small sEEG datasets and the public T15 microelectrode dataset, including an architecture-matched MSE CNN-DiT control.

Critical strengths
  1. The paper gives a potential generative alternative to direct neural-to-spectrogram regression and evaluates it across both sEEG and microelectrode recordings. The architecture-matched MSE control, solver-budget analysis, and causal T15 setting are useful experimental design choices.
  2. The evaluation is multi-faceted and relatively transparent. The paper reports metrics that do not favor the proposed model, and the supplied code, configuration, and checkpoint make important parts of the T15 evaluation inspectable.
Critical weaknesses

Overall, This paper is technically incremental. The methodological novelty and task-specific mechanism are limited. The pipeline is essentially sEEG features→temporal CNN / resampling→standard DiT-conditioned CFM→standard HiFi-GAN/LPCNet\text{sEEG features} \rightarrow \text{temporal CNN / resampling} \rightarrow \text{standard DiT-conditioned CFM} \rightarrow \text{standard HiFi-GAN/LPCNet} Beyond standard neural-signal preprocessing and a temporal CNN conditioner, the method transfers established CFM, DiT, ODE sampling, and vocoder components to sEEG. Its central task-specific premise is that neural recordings underdetermine acoustics, so MSE conditional means blur speech and stochastic refinement yields a more plausible sample; this premise is not directly validated, and no sEEG-specific module or analysis establishes that the samples capture neural uncertainty rather than merely change the distortion--perception trade-off. The matched T15 control reinforces this concern: it ties flow on WER (0.22) and is better on non-DTW ESTOI (0.210 vs. 0.204) and frame-aligned correlation (0.604 vs. 0.570), while flow is better on DTW/perceptual metrics. Reported significance tests are against released Wairagkar audio rather than the matched control.

Questions for authors
  1. The paper states that saliency selection was made on one fold and then checks its stability across blocks. However, this does not remove the possibility that held-out data influenced the shared feature-selection step. Please repeat the NZ evaluation with channel selection nested inside each training fold, or fixed from an independent dataset, and report how the main results change.
  2. What property of sEEG makes the standard CFM/DiT transfer necessary beyond a generic stochastic decoder? Please provide paired flow-versus-matched-MSE comparisons for all T15 metrics, and explain why DTW-aligned/perceptual improvements outweigh tied WER and worse frame-aligned metrics for the intended use case.
2 comments
AuthorsOpenReview ↗

R[1/2]: Modeling the conditional distribution improves perceptual quality and intelligibility

We thank the reviewer for the their review. Below we answer the questions raised by the reviewer. Relevant results and explanations from the general response are cited rather than duplicated.

Validating the central premise (W1) The reviewer is concerned that we haven't shown directly (i) that "MSE conditional means blur speech and stochastic refinement yields a more plausible sample'' and (ii) "that samples capture neural uncertainty rather than merely change the distortion--perception trade-off.'' We agree that our demonstration in the original ms. is indirect, and have therefore added analyses addressing these two questions.

That MSE losses blur speech we believe has been firmly established in the TTS literature (see e.g. "Revisiting Over-Smoothness in Text to Speech," Ren et al. 2022). That the generative model yields a more plausible sample than a matched discriminative model we have shown directly in the ms., and our new results under our general comment on The role of flow matching in decoding performance. What remains is to show (ii).

A move along the distortion--perception frontier must pay in distortion for whatever superior perceptual quality it achieves. In our proposed model, this is not the case. That is, compared to the architecture-matched discriminative version of our model, a single draw from our CFM model has essentially the same frame-wise correlation, but superior intelligibility, spectral distortion, speaker similarity, and naturalness at once:

P0 block 4flow (one draw)matched regressor
rspr_{\mathrm{sp}} ↑\uparrow0.8070.816
ESTOI ↑\uparrow0.4110.380
STOI ↑\uparrow0.6620.648
MCD (dB) ↓\downarrow6.806.85
SpkSim ↑\uparrow0.7230.632
UTMOS ↑\uparrow1.8941.589
KID (ECAPA) ↓\downarrow1.473.89
KID (wav2vec 2.0) ↓\downarrow0.431.43

(In addition to the metrics used in the ms., we include kernel inception distance [KID], a measure of distributional match between synthesized and real speech.) Evidently, the CFM is on a different (superior) frontier of the perception--distortion trade-off, not merely a different position on the discriminative model's frontier. (We emphasize that the discriminative model is a strong baseline that already beats prior art.)

These analyses consider single draws; we turn now to their distribution. If the spread across samples were unrelated to the neural evidence, it would not change when the recording is degraded. Here we degrade the conditioning by \emph{graded} phase randomization: At each frequency, we add the same random phase to each of x% of the channels. When x=100, all statistics above second-order are wiped out. For each test trial, we draw K=16K=16 samples, score each draw, take the standard deviation of that trial's scores, and average it over the 20 trials of one held-out block per participant:

phase randomizedP0 mean rspr_{\mathrm{sp}}P0 spreadP0 vs. cleanP1 mean rspr_{\mathrm{sp}}P1 spreadP1 vs. clean
0% (unmodified)0.8080.00541.0×1.0\times0.7870.00781.0×1.0\times
25%0.7730.00721.3×1.3\times0.7490.00871.1×1.1\times
50%0.6340.02494.6×4.6\times0.6170.01702.2×2.2\times
75%0.5490.02384.4×4.4\times0.5320.02272.9×2.9\times
100%0.5170.03466.4×6.4\times0.4800.03454.5×4.5\times

Corrupting the recording increases the spread 6.4×6.4\times on P0 and 4.5×4.5\times on P1: as the neural evidence is destroyed the model becomes measurably less certain about its own prediction, so (as expected) the samples do carry information about how well the recording determines the acoustics.

Reporting significance against the matched (discriminative) control As requested, in the updated version of the ms., we now test the significance of our proposed flow model against the matched discriminative model (rather than Wairagkar). See R3 T15 in our general replies for more details.

Channel selection Please see point R4 in our general replies.

What property of sEEG makes this transfer necessary We do not claim that a property unique to sEEG makes CFM necessary. Rather, the incomplete and noisy nature of neural recordings motivates modeling the conditional distribution of speech. CFM provides a tractable way to learn this distribution. Our ablations separate the contributions of the CNN-DiT architecture, noise augmentation, training objective, and multi-step inference. They show that adding noise alone does not reproduce the gains of training across the interpolation path, while multi-step inference provides additional perceptual benefits. These findings support our choice of CFM, but do not rule out other conditional generative models.

AuthorsOpenReview ↗

R[2/2]: Modeling the conditional distribution improves perceptual quality and intelligibility

Why DTW-aligned/perceptual improvements outweigh tied WER and worse frame-aligned metrics for the intended use case In fact, once the language model is removed, the CFM model outperforms its discriminative counterpart even on WER and on CER; please see point R3 T15 in our general replies. We would also emphasize that WER is not the only or most important test of performance; if it were, we would simply decode text rather than speech audio.

Addressing the ethics flag. To promote reproducibility of neural speech synthesis methods in speech BCI, we have decided to release our in-house dataset after publication as per the IRB protocol. We understand that speech audio data is potentially person-identifiable, therefore, we will use random pitch shifting to anonymize both participants' (both P0 and P1) voice in the public release as in previous research (Verwoert et al. 2022, Sci. Data).

Reviewer_T2iSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper casts neural-to-speech synthesis as conditional generation. A 1-D dilated CNN encodes intracranial features into frame-aligned context, which is concatenated with the current noisy spectrogram and fed to an 8-block DiT (AdaLN-Zero, RoPE) that predicts a linear-path CFM velocity (Eq. 3, minibatch-OT coupling). Sampling uses 8 Euler steps, and scores are averaged over K = 8 draws per trial. Appendix A.1 notes that the discriminative MSE regressor is the special case where the FM loss is imposed only at t = 0. Experiments cover (i) two in-house sEEG participants (NZ0/NZ1, about 30 min each, leave-one-block-out CV) against five re-implemented baselines plus an architecture-matched "CNN-DiT mel regressor" (Table 1), and (ii) the public T15 dysarthric Utah-array dataset of Wairagkar et al. (2025), where a causal variant is compared with the released online audio (Table 2/12) and with a retrained Wairagkar decoder under varying training-set sizes (Fig. 2). Ablations cover the solver budget and the distortion-perception trade-off (Fig. 3), how the source is conditioned (Table 3), the probability path (Table 6), sampling variance (Table 8), sample averaging (Table 7), receptive field (Table 9) and baseline tuning (Table 11).

Critical strengths
  1. Well-motivated framing. The argument that MSE regression returns a conditional mean that may lie off the speech manifold, which shows up as blurry spectrograms, is reasonable and well known from TTS (Grad-TTS, Popov et al. 2021; Matcha-TTS, Mehta et al. 2024; Voicebox, Le et al. 2023; F5-TTS, Chen et al. 2025). Importing it into intracranial speech synthesis is a sensible step that the BCI-speech literature (Angrick et al. 2019/2021; Anumanchipalli et al. 2019; Metzger et al. 2023; Chen et al. 2024 NMI; Wairagkar et al. 2025) has largely not taken.
  2. The right controls are included. The architecture-matched MSE regressor (Table 1, Table 12) is exactly the control needed to separate the objective from the architecture. The authors report it even where it undercuts the headline claim (T15 WER is 0.22 for both models), and that honesty is welcome.
  3. Careful analysis of the distortion-perception trade-off. Fig. 3, Table 7 (averaging K samples moves toward the conditional mean: r_sp goes up, UTMOS/SpkSim go down) and Table 6 (straightness of the linear path vs. VP/VE paths; minibatch-OT coupling turns out to be unnecessary) are informative and honestly reported, including the null result on OT coupling.
  4. Use of a public, strong benchmark. Running on the T15 data with the original chained "train on all prior sessions" protocol (A.2.2), causal masking, and a receptive field matched to the 600 ms window (A.2.1) is a thoughtful attempt at a like-for-like comparison. The per-day breakdown (Fig. 5) and the data-efficiency sweep (Fig. 2) are useful.
  5. Statistical reporting is better than typical. The paper reports fold-level standard deviations, paired one-sided Wilcoxon tests, within-trial sampling variance (Table 8), and three training seeds for the ablations.
  6. Implementation detail is good (Tables 4/5, A.2), and the windowed overlap-add inference (Eq. 5-6) is clearly described.
Critical weaknesses
  1. The headline T15 claim is misattributed to flow matching. The abstract and Introduction say "our conditional flow ... bring[s] word error rates ... down from about 56% to 22%". However, Table 2/12 shows the architecture-matched CNN-DiT MSE regressor also reaches 0.22 median WER (IQR 0.00-0.44 vs. 0.00-0.40). Section 4.3 concedes that "word intelligibility is seemingly set by the architecture and the causal protocol." The flow objective's contribution on T15 is limited to ESTOI_DTW (0.355 → 0.404), SpkSim, and a small change in MCD. Frame-aligned r_sp actually gets worse (0.604 → 0.570, below Wairagkar's 0.658). There are further confounds:

    • The comparison is offline decoding versus the released online audio. That audio was produced by a real-time, closed-loop system with its own latency and engineering constraints.
    • The retrained-offline Wairagkar baseline appears only in the single-day data-efficiency sweep (Fig. 2). There it reaches 0.53 at best, so the "56%" figure is not a like-for-like baseline.
    • Fig. 2 omits the MSE regressor, so the "four-fold fewer sessions" claim (Section 4.3, abstract) again cannot be attributed to flow matching rather than to the CNN-DiT architecture.

    The abstract, Introduction (finding #2) and Conclusion should be rewritten to credit the architecture and protocol, not the generative objective.

  2. No controls for hallucination or grounding. A conditional generator with a strong prior can produce realistic, speaker-like spectrograms that are only weakly tied to the neural input. This is exactly what the metrics where the method shines reward. UTMOS is reference-free naturalness. SpkSim compares against a single speaker whose voice the generator has learned. Meanwhile, NZ intelligibility is near floor: median WER is 0.88 (NZ0) and 0.97 (NZ1) in Table 10, and Figs. 6-7 show fluent but wrong words ("put the box", "tempting jeremy is in there"). The paper never reports:

    • (a) the model run on shuffled or mismatched-trial neural inputs, or on time-reversed or phase-randomized sEEG;
    • (b) an unconditional or constant-context generator;
    • (c) chance levels for every metric (e.g., ESTOI/SpkSim/UTMOS of the ground truth against a random other trial).

    Without these, the roughly 25% SpkSim/UTMOS gains in Table 1 cannot be read as better decoding rather than better speech prior. The same worry applies to Whisper-based WER: Whisper medium.en has a strong language-model prior that rewards fluent, natural audio. T15's targets are TTS renderings of cued sentences. A human-transcription or closed-vocabulary listening test (as in Wairagkar et al. 2025 and Metzger et al. 2023), or at least an ASR without an LM, is needed.

  3. The evidence contradicts the "iterative refinement" thesis. The Conclusion argues that what distinguishes generative from discriminative models is "more than one step (NFE)". However:

    • In Fig. 3 (NZ0), the N = 1 flow already sits far above both the MSE regressor (□) and the t = 0-only flow (△) on SpkSim and UTMOS. N = 8 adds comparatively little.
    • In Table 6/7, ESTOI is 0.395 at N = 1 vs. 0.403 at N = 8.
    • Averaging K = 8 samples (moving toward the conditional mean) raises ESTOI to 0.415 (Table 7, row 3). The most intelligibility-relevant metric therefore benefits from mean-seeking. This undercuts the claim (Sections 1 and 4.2) that the conditional mean harms intelligibility.

    One possible explanation is that training across all t acts as a regularizer or data augmentation for the encoder in this tiny-data regime (about 300 training trials), since a one-step evaluation is not iterative refinement. This should be tested directly, e.g., with an MSE regressor trained with input-noise augmentation, or with the t = 0 model trained with noise concatenated, and the narrative revised accordingly.

  4. Possible leakage in the sEEG protocol, and test-fold reuse.

    • Channel selection (A.3.6) uses gradient saliency from a CNN trained on "a subset of the data", and Fig. 4 is framed around block 4. It is not stated whether block-4 test data informed the selection. The argument that "any upward bias would affect all algorithms equally" does not hold: a selection made by a convolutional MSE decoder can favour some architectures over others.
    • All ablations (Tables 3, 6, 7, 9, 11, Fig. 3) and baseline tuning use NZ0 block 4, which is also a test fold in Table 1. Hyperparameters were "extensively tuned on the NZ datasets" without a stated nested validation split.
    • The NZ checkpoint selection uses "validation MCD", but the validation set is never defined (A.2.2).
    • With roughly 3,000 candidate sentences sampled at random, the paper should state whether any sentence recurs across blocks.

    Please specify the inner/outer splits and rerun selection inside each fold.

  5. Statistical tests target the wrong comparison. The significance markers in Table 1 and Table 12 compare against the "strongest published baseline" or the online audio. The paper's central claim (generative vs. discriminative) requires paired tests against the architecture-matched MSE regressor, and on NZ the margins over it are modest for r_sp and MCD. Please report how the exact one-sided Wilcoxon p-values were computed with 8-9 folds, including the effective paired sample size and the handling of zeros and ties. Per-trial tests with fold-clustered errors would be more informative. Results for NZ0 vs. NZ1 should also be reported per fold.

  6. Data availability and benchmarking. The sEEG results rest on two unreleased participants. The paper does not evaluate on the public sEEG speech dataset of Verwoert et al. (2022), which has 10 participants and is cited in the paper, and which would allow independent verification and a fair comparison with published numbers. Several baselines are re-implementations with questionable provenance:

    • "CNN-LSTM (Quadri et al., 2024)" is a seizure-prediction paper, not a speech-synthesis method.
    • "VocalMind (He et al., 2025)" is a dataset paper.
    • The Metzger et al. (2023) pipeline was designed for 253-channel ECoG with HuBERT units and voice cloning, so applying it to 64 sEEG channels is not a fair representation.

    The authors' own tuning of the Wairagkar baseline (Table 11) shows non-causal lookahead adds about 0.12 ESTOI. It is not clear that Table 1's Wairagkar row uses the best non-causal variant, given that the proposed NZ encoder is non-causal (±312 ms).

  7. Novelty is moderate. Conditional flow matching for mel generation is standard (Matcha-TTS, Voicebox, F5-TTS). Generative priors for neural decoding are established in vision, e.g., diffusion for fMRI image reconstruction (Takagi & Nishimoto, CVPR 2023; Scotti et al., MindEye, NeurIPS 2023). The claim "to date, all such approaches have been based on discriminative modeling" is too strong:

    • Metzger et al. (2023) decode discrete units and resynthesize through a pretrained generative synthesizer.
    • Chen et al. (2024, NMI) use a differentiable speech synthesizer.
    • I believe adversarial (GAN) training has been explored for ECoG speech decoding (e.g., Wang et al., ISBI 2020), but please verify.

    The derivation in A.1 (FM at t = 0 equals regression) is straightforward. The contribution is therefore mainly empirical, and given weaknesses 1-3 the empirical case for flow matching specifically is weaker than presented.

  8. Practicality. 8 NFEs × 4-second windows × K draws is not streaming, and real-time decoding is explicitly not demonstrated (Section 5). The T15 comparison against a system that was real-time therefore favours the proposed method on constraints. Latency and RTF numbers should be reported for the causal model.

Minor Issues

  • A.2.2, lines 833-834: garbled text ("ratheirst stage", "previous s,").
  • "diffuson" (l. 477), "SPkSim" (l. 364), "Following the Wairagkar et al." (l. 358); LFC vs. LFP terminology is inconsistent (A.3.6).
  • Table 3 math rendering is broken in places, and its "Cond." column makes rows 2-5 hard to parse. The text refers to the "third row" as cutting the encoder to a linear layer, but the table's third row is a noisy source with Wx+b.
  • References: Blau & Michaeli is listed as "CVPR 2018, 2017"; Guided-TTS as "ICML 2022, 2023"; Lipman et al. links to Semantic Scholar rather than OpenReview.
  • Table 12: ESTOI without DTW is 0.204 for Ours vs. 0.210 for the regressor, yet Ours is bolded or starred against Wairagkar only. State this explicitly in the text.
  • Specify the ASR decoding settings (Whisper size for NZ; text normalization) and how per-trial WER is computed for "pooled draws".
  • The HiFi-GAN vocoded GT UTMOS of about 2.0 is low. Please comment, e.g., on recording conditions.
  • "NZ" dataset naming and "recently made by our group" may weaken anonymity.

Justification of Rating

The idea of replacing regression with conditional flow matching for intracranial speech synthesis is sensible and timely. The paper is carefully executed in many places, and it is commendably transparent: it includes an architecture-matched regressor and reports null results. However, the paper's most prominent claim, the T15 WER reduction from 56% to 22%, is equally achieved by the MSE regressor. The sEEG gains are concentrated in prior-sensitive perceptual metrics, with intelligibility near floor and no shuffled-input or chance controls. The paper's own ablations are consistent with a benefit from multi-t training rather than iterative refinement, but do not by themselves establish that mechanism. There are also unresolved test-fold reuse and channel-selection leakage questions, and the sEEG data are private. I would move toward weak acceptance if the authors:

  • show via shuffled or mismatched neural-input controls that the perceptual gains are neurally grounded;
  • provide paired tests against the matched regressor that remain significant on intelligibility-related metrics (ESTOI, WER) under nested selection;
  • add a public sEEG benchmark (e.g., Verwoert et al., 2022) or human listening tests;
  • reframe the claims to separate architecture from objective.

I would lower the score if the shuffled-input controls retain most of the SpkSim and UTMOS gains.

Questions for authors
  1. What does the flow model produce when conditioned on shuffled or mismatched-trial neural data, and on an all-zero or constant context? Please report r_sp, ESTOI, SpkSim, UTMOS and WER for these controls on NZ and T15, together with ground-truth-vs-random-trial chance levels. If SpkSim and UTMOS remain close to Table 1 values under shuffling, the perceptual gains are prior-driven. (Most important.)
  2. Can you provide paired significance tests of Ours vs. CNN-DiT MSE regressor on every metric, for both NZ and T15, and add the regressor to Fig. 2? Would you revise the abstract to attribute the 56% → 22% WER drop to the architecture and protocol?
  3. Was block 4's test data used in the channel-selection CNN or the hyperparameter search? What exactly is the NZ validation set for checkpoint selection? Can you redo selection nested within each CV fold?
  4. Does an MSE regressor trained with noise augmentation (e.g., the same DiT receiving a random y_t and t as nuisance inputs but regressing y_1) close the gap to the N = 1 flow in Fig. 3? This would separate "multi-t training as regularization" from "iterative refinement".
  5. For T15, what is the WER of the retrained offline Wairagkar decoder on all 8 evaluation days, rather than only day 172 in Fig. 2? Is the CNN-DiT regressor's advantage over it preserved? How much of the 0.56 → 0.22 gap is online-vs-offline?
  6. Can you report intelligibility from human listeners (at least for T15, which is public and de-identified), or from an ASR without a language-model prior, or using closed-set word identification?
  7. In Table 1, which Wairagkar variant (causal or 400 ms lookahead; Table 11) is used? Do all baselines get the same ±312 ms non-causal context as your NZ encoder?
  8. Do any sentences recur across NZ blocks, or between the read and recalled repetitions placed in different folds?
  9. Why does x0 = WE(x) + b in Table 3 give the best MCD (6.433) alongside a negative ESTOI and SpkSim? Is MCD computed on mel-cepstra or directly on the 80-bin log-mel, and is it reliable here?
3 comments
AuthorsOpenReview ↗

R[1/3]: Validated flow-matching gains, added controls, and refined claims

We thank the reviewer for the very detailed review. Below we address each concern (W1--W8) with new experimental results across the in-house sEEG and T15 datasets, and answer the nine questions raised by the reviewer. Relevant results and explanations from the general response are cited rather than duplicated.

Behavior under mismatched, time-reversed, phase-randomized, and constant conditioning This is an important methodological point in the reviews, and we agree these controls belonged in the submission. All are now run on P0 block~4 (20 held-out trials), and the table for P0 has been added to the revised manuscript. Both alignment conventions are shown for the two reference-based metrics, as in the T15 table below: our evaluation suite scores them either after a DTW warp, which permits re-timing and so raises their floor, or under a frame-aligned variant (which forbids it). WER is LM-free (\texttt{wav2vec2-large-960h}) and the transcriptions are generated by frame-wise argmax; no language model, lexicon, or beam search is employed.

conditioning supplied to the trained modelrspr_{\mathrm{sp}} (DTW)rspr_{\mathrm{sp}} (frame-al.)ESTOI (DTW)ESTOI (frame-al.)SpkSimUTMOSWER
unmodified sEEG0.8080.6190.4170.1880.7221.9090.94
mismatched trial0.5670.1280.1030.0180.6181.9201.00
time-reversed0.5020.0110.0660.0010.5351.6481.00
phase-randomized0.517−0.014-0.0140.069−0.009-0.0090.5661.6121.00
all-zero context−0.053-0.053−0.050-0.050−0.010-0.010−0.013-0.0130.0480.824silent
model trained on mismatched pairs0.375−0.008-0.0080.050−0.007-0.0070.2961.3571.30

The mismatched trial row shows that the model still produces natural, speaker-like speech from a genuine sEEG recording from the wrong trial: UTMOS is 1.920 against 1.909 for the correct recording, and SpkSim only falls from 0.722 to 0.618. The content, however, is missing, with ESTOI dropping from 0.417 to 0.103. The prior alone is therefore not what produces our results, because the output is tied to the neural input. Phase randomization is the clearest case: it preserves every channel's power spectrum to within 10−610^{-6} and the cross-channel correlations to within 10−810^{-8}, so the input remains entirely plausible sEEG and is simply the wrong sEEG. Performance still falls across the board --- ESTOI 0.069, SpkSim 0.566, UTMOS 1.612 --- as it does under time reversal, which likewise leaves the marginals and spectra exact. Replacing the recording altogether with zeros destroys the output entirely (ESTOI −0.011-0.011, UTMOS 0.824).

For the chance level, we also trained the model on randomly paired neural and acoustic data (randomly paired before each iteration); that model reaches ESTOI 0.050, SpkSim 0.296 and UTMOS 1.357, which is where every measure sits when no relationship can be learned. In short, the model draws its perceptual quality from the prior but remains grounded in the neural recording: synthesis quality survives only when the conditioning is both plausible \emph{and} correct. We have also run the same five controls, and the same chance-level null, on T15 as well (day~172, 112 held-out trials). Under the same two alignment conventions and the same LM-free recognizer, we observe similar trends across the control scenarios.

conditioning supplied to the trained modelrspr_{\mathrm{sp}} (DTW)rspr_{\mathrm{sp}} (frame-al.)ESTOI (DTW)ESTOI (frame-al.)SpkSimUTMOSWER
unmodified neural0.8410.5320.3740.1700.8671.8620.71
mismatched trial0.527−0.020-0.0200.0700.0040.7721.8291.13
time-reversed0.536−0.035-0.0350.0450.0010.6371.5901.02
phase-randomized0.571−0.035-0.0350.0720.0040.7411.6521.13
all-zero context——0.0000.0000.2031.274silent
model trained on mismatched pairs0.4950.0390.0590.0030.6901.5893.48

For T15, all-zero conditioning silences the decoder, so the transcript is empty and every mel band is constant: rspr_{\mathrm{sp}} is undefined rather than low. The predicted LPCNet features, which are not silenced, correlate at −0.004-0.004 against 0.535 when unmodified; the P0 pipeline's zero-context output keeps some band variance, hence the finite entries there.

non-LM based ASR transcription Please see the "T15" section of our general response.

Paired tests against the CNN-DiT regressor, and the abstract We have performed the significance tests between our flow model and its regressor-counterpart in all result tables for all datasets. Regarding the WER issue, we address this with quantitative results in the ``T15" section of the general response. We have also rewritten our contributions and claims accordingly in the abstract and introduction.

AuthorsOpenReview ↗

R[2/3]: Validated flow-matching gains, added controls, and refined claims

Block 4, the in-house sEEG data validation set, and nested selection For the in-house sEEG dataset, each held-out block is split into validation and test sets (50% of the trials in each). For channel selection and hyperparameter search, we had used the validation set of block~4. We have now moved to fold-specific channel selection, using the validation set of each held-out block (see point-3 of the general response). We thank the reviewer for pointing out this ambiguity. We have clarified the description of the training, validation, and test splits in section A.2.2 of the revised manuscript.

An MSE regressor with noise augmentation (Q4/W3) Please see section "The role of flow matching in decoding performance" of our general response.

The retrained offline Wairagkar decoder on all eight days We have included the offline implementation of Wairagkar et al. in all T15 results in the updated manuscript. Although it is markedly stronger than the online released audio, it is still worse compared to our CNN-DiT under both the MSE and the flow objectives.

Human listeners, an ASR without a language model, or closed-set identification (Q6/W1). It was not possible to manage human listeners in short notice, hence we have implemented an ASR without a language mode: Wav2vec2. Please see the "T15" section of our general response for details.

Which Wairagkar variant, and matched context for baselines For the in-house datasets we have used the best variant of Wairagkar et al. after finding the optimal context window size (non-causal; 600 ms past plus 400 ms future, a total of 1 s) through our search (details in Appendix Table 11). Going beyond 1 s hurts performance for the Wairagkar's streaming model, which predicts one frame at a time. The results in Table 1 are therefore the outputs of this optimal non-causal variant of the Wairagkar et al. model. All baselines other than Wairagkar et al. (non-streaming), including our architecture matched MSE-control get the same context size as our flow model.

Sentence recurrence across in-house dataset blocks and repetitions Yes, some sentence prompts recur across blocks. However, each read/recall pair is kept together in the same split; its two repetitions are not separated between training, validation, and test sets. Across train/test blocks, 16/160 senteces recurred for P0 and 18/180 for P1 (10% each). Excluding these pairs does not change the results of our CFM model by much (shown in the table), using the same checkpoints trained with nested channel selection.

ParticipantTest setrsp↑r_{\mathrm{sp}}\uparrowSpkSim ↑\uparrowESTOI ↑\uparrowMCD ↓\downarrow
P0All test trials0.8040.6460.3746.897
P0Exact repeats excluded0.8020.6440.3686.969
P1All test trials0.7550.6520.3875.280
P1Exact repeats excluded0.7560.6530.3875.334

The x0=WE(x)+bx_0 = W_E(x) + b row in Table 3, and how MCD is computed \ MCD is computed on mel-cepstral coefficients extracted from the synthesized and reference waveforms, not directly on the 80-bin log-mel spectrograms. The calculation excludes the zeroth coefficient and uses DTW alignment. MCD measures spectral-envelope distortion rather than intelligibility or speaker similarity. We include it because it is widely used in neural speech synthesis, including by Anumanchipalli et al. (2019, Nature) and Metzger et al. (2023, Nature). However, we agree that MCD should not be interpreted alone, but alongside intelligibility and perceptual measures.

Addressing the target of the statistical tests We agree, and the tables in the general response, as well as the revised manuscript, is now marked this way: every test compares the flow against the architecture-matched CNN-DiT regressor over the same 964 trials, under a paired one-sided Wilcoxon signed-rank test of the single hypothesis that the flow is better. Marks therefore appear only in our column, and three of the eleven rows carry n.s.
where that hypothesis is not supported --- Whisper WER, frame-aligned correlation and non-DTW ESTOI.

AuthorsOpenReview ↗

R[3/3]: Validated flow-matching gains, added controls, and refined claims

Addressing data availability and benchmarking We understand the concern regarding result reproducibility and generalization ability of the proposed method beyond the in-house datasets. To this end, we have decided to release the data in our GitHub repository following possible publication, in accordance with all human-subjects protocols required by our IRB. Furthermore, we analyzed our proposed method alongside the benchmark methods on a third dataset: Zada, which features a ECoG dataset (approx. 30 min.) from speech perception trials and features eight subjects. For brevity, we synthesize speech from ECoG trials and observe that our proposed method is more robust in comparison to both the benchmark methods and also its architecture matched MSE-control variant. We have incorporated these results to the revised version of our manuscript.

Addressing the novelty of the contribution We thank the reviewer for pointing out the recent use of generative priors in neural decoding tasks in modalities other than speech. What we wanted to convey in our claims is that previous speech decoding work have mostly employed a discriminative loss (MSE/cross-entropy) for learning acoustic targets from the neural data: Metzger et al. (2023, Nature) used a cross-entropy (with CTC) to predict the HuBERT discrete units from ECoG; Chen et al. (2024, NMI) used MSE to predict the interpretable speech parameters from ECoG. The speech synthesizers for these modular approaches were trained in a generative manner, however these do not involve the neural data at all. Finally, we thank the reviewer for mentioning the Wang et al. (2020, ISBI) paper. We verified it, and found that it indeed uses a generative (GAN-based) approach to learn speech synthesis through a classic VAE-style encoder-decoder structure. Although it has been trained with a GAN loss, it has been evaluated as a single reconstruction per trial using mean squared error and correlation coefficient only (their Table 1). No perceptual or intelligibility measure and no mechanism for drawing alternative acoustic samples for a singleton neural input is described. Our method in contrast learns a conditional mapping from the neural data manifold onto speech directly through flow-matching, and fully describes how to generate samples given any neural recording. Nonetheless, we have softened our claim regarding previous models being purely based on discriminative modeling and also cite the Wang et al. (2020, ISBI) paper in the introduction. We again thank the reviewer for pointing this out.

Addressing practicality, latency and RTF Latency and RTF for the causal T15 model are reported in our reply to Reviewer jXct: 3.97s per utterance against 22.7s trial windows, a real-time factor of 0.175, most of it the LPCNet vocoder. We claim no streaming operation with regards to our decoder.

Reviewer_jXctOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper is an application of CFM to neural data, along with needed statistical ablations and tests.

Critical strengths

I think it is a well-executed application of CFM on neural data.

The experimental rigor is well above the usual in this field: matched MSE control, path ablations with straightness metrics, source-distribution ablations, NFE × K grid, sampling variance / best-of-K, receptive field sweep, re-tuned baselines, significance tests. The honest finding is “iterative generation improves perceptual quality and speaker similarity at roughly same intelligibility”, which is a real but modest result. If the paper is rewritten around that claims, this is a decent paper.

Critical weaknesses

Limited Novelty: The model is basically Matcha-TTS/Voicebox recipe (linear path CFM, OT coupling, DiT with AdaLN-Zero, HiFi-GAN) with a CNN neural encoder replacing text encoder. Nothing in the architecture or objective is new. The conceptual part (discriminative = one-step special case, conditional mean = blurry spectrogram, distortion–perception trade-off) is a reasonable framing but it is a short derivation plus Blau & Michaeli, not a contribution by itself.

The main concern is the headline claim is not supported by their own control. On T15, the architecture-matched MSE regressor already gets 0.22 WER, same as the flow model (Table 2, 12). So the 56% → 22% in the abstract is from the backbone and training protocol, not from flow matching. They actually admit this in 4.3 (“intelligibility is seemingly set by the architecture and the causal protocol”) but the abstract and intro still credit the flow. Flow only improves perceptual metrics (ESTOI_DTW 0.355 → 0.404, SpkSim, UTMOS), and it is worse on frame-aligned r_sp (0.604 → 0.570) and non-DTW ESTOI. Same pattern on NZ, the MSE regressor already beats all published baselines on most metrics and flow adds on top on SpkSim/UTMOS/ESTOI only. Also the data-efficiency curve (Fig. 2) is flow vs. Wairagkar, not flow vs. matched regressor, so the “one fourth data” claim cannot be attributed to the generative objective either.

Some other things. T15 targets are TTS-synthesized proxy audio, so WER is measuring recovery of a synthetic voice, and they use Whisper while original used human transcribers, so 0.56 vs 0.22 is not a clean comparison. NZ results are still unintelligible (WER ~0.9–1.0), which they do acknowledge. No real-time demonstration even though framed as clinical. NZ dataset is in-house and probably not releasable.

As written, the abstract/and paper overclaims.

Questions for authors

Weaknesses Section

1 comment
AuthorsOpenReview ↗

more experiments back up the headline claim, but we also made our claims more nuanced

We thank the reviewer for the appreciative remarks on our experimental rigor and also for the constructive criticism of our method. Below we address each concerns (W1--W4) with a new set of analyses including measured latency for the causal model on the T15 dataset. We have also modified the claims of our paper to better reflect our results as per the concerns; see especially the abstract, the last paragraph of the Introduction, and the penultimate paragraph of the Conclusions.

  1. Novelty Please see the ``Technical Novelty" section of the general response above.

  2. Architecture-matched regressor vs. CFM on T15 Please see the The role of flow matching in decoding performance in our general replies.

  3. Whisper vs. human transcribers We believe the reviewer has misunderstood something here, but please correct us if we have mis-read his/her concern: We used Whisper for all reported WERs in the ms., including the WER for the Wairagkar model mentioned in the abstract. % Regarding the synthesized proxy audio:\ yes, perforce we have used this, since the participant cannot speak.

Following the reviewers' suggestions, we have now also evaluated models using an acoustic-only transcription model (see the T15 section of our general replies). We have clarified the evaluation of WER in the manuscript.

Real-time operation The CFM model we run on T15 is already causal, so the primary question is run time. Here we measure the time cost of one prediction at eight Euler steps, for 25 trials of day 172, on one A100 GPU. (Trial windows average 22.7~s of speech.) We also show the real-time factor (processing time divided by the duration of the audio produced) as in ASR studies (factor values below one indicate faster-than-real-time operation):

StageProcessing time (s)Real-time factor
Sampling (GPU)0.560.025
Vocoding (LPCNet, CPU)3.410.150
Total3.970.175

The total factor of 0.175 corresponds to about 5.7×5.7\times faster than real time, and most of the cost is the LPCNet vocoder we adopt from Wairagkar et al. The decoder itself is strictly causal on T15---left-padded convolutions, a lower-triangular attention mask in the velocity field and per-frame normalization. Therefore frame kk reads neural bins [k−60,k][k-60,k], i.e. 610 ms of trailing context with no lookahead, matching the information set of the baseline, which ran online in closed loop. We would still need to implement a streaming version of test-time generation, but since this is consistent with the current architecture, we would not even have to retrain the model.

Concluding remarks The reviewer suggested we revise our claims to something more modest, and we have now done so (especially in the abstract, intro, and conclusion), making sure to explicitly decouple the contributions from the architecture and generative objective. Still, we would not say that intelligibility is not improved by generative modeling. Removing the language model, as we have now done, reveals that the CFM model is extracting more intelligible speech (lower WER and CER) than its discriminative counterpart (see general replies). And the same holds for ESTOI (the DTW version, which is our default throughout).

We hope these revisions and the additional experiments address the concerns sufficiently for the reviewer to consider a more favorable score. We would be glad to run further experiments or analyses during the discussion period if that would be helpful.

Comments on the paper

5 comments
AuthorsOpenReview ↗

general replies to all reviewers

Several points were raised by multiple reviewers, so for convenience we address (and number) them here. Below and in the updated ms., we refer to the two participants in our study as P0 and P1. (The old label was not identifying, but was needlessly opaque.)

R1. Technical novelty

We agree that our model is similar to recent TTS models. But in the BMI literature, the situation is very different. Whereas TTS has been progressing through various generative models (diffusions, normalizing flows, and flow matching) over the last five years, BMI decoding has not really made the move to generative models at all. And our study doesn't merely show that flow matching is competitive at speech decoding, but improves the state of the art by a large margin.

We would also emphasize that not all methods in ML are applicable to BMI decoding, which faces a different set of constraints than TTS or ASR do: Neural data (1) have much lower SNR, (2) cannot be collected into massive data sets, and (3) are not interpretable even by human experts. One of the novel points of this study is that this makes generative models particularly well-suited to the decoding problem, indeed more than even to TTS. This is easy to overlook if one focuses only on the correlation coefficient, as several speech decoding studies have (Wang et al. (2020, ISBI), Verwoert et al. (2022, Sci. Data)). We believe this point is not currently appreciated in the neuroscience literature (or beyond, where large, clean datasets is the norm). It holds especially for stereo-EEG and ECoG.

R2. The role of flow matching in decoding performance

Multiple reviewers objected to our attribution of our performance gains to flow-matching and in particular "iterative refinement," especially in the abstract and intro and w.r.t. T15 (decoding from Utah array data). We agree that it would be better to separate out precisely the contributions of (1) the architecture, (2) the training (perhaps viewed merely as data augmentation [Rev. T2iS]), and (3) the test-time iterative refinement via numerical integration.

First, on both participants, the flow matching model is superior to its discriminative counterpart on all metrics except---as expected---correlation coefficient:

ParticipantModelrspr_\text{sp} ↑SpkSim ↑ESTOI ↑UTMOS ↑MCD ↓
P0CNN-DiT mel regressor0.8050.5590.3331.5647.086
Ours0.804 n.s.0.646 ***0.374 ***1.857 ***6.897 ***
P1CNN-DiT mel regressor0.7570.6230.3301.6505.404
Ours0.755 n.s.0.652 ***0.387 ***1.983 ***5.280 ***

Second, we ran a more fine-grained set of experiments on P0 (we will run for P1 as well if time permits), working our way up by degrees from the discriminative model to the full generative model:

ModelTrainingESTOI ↑SPK-SIM ↑UTMOS ↑
Wairagkar et al. (2025)Discriminative0.3290.3951.571
CNN-DiT mel regressor (existing baseline)Discriminative0.337 (+2.2%)0.595 (+50.7%)1.540 (−2.0%)
Mel predictor + neural input noiseDiscriminative0.346 (+5.0%)0.585 (+48.2%)1.624 (+3.4%)
Mel predictor + encoder output noiseDiscriminative0.340 (+3.3%)0.598 (+51.4%)1.530 (−2.6%)
Mel predictor, t = 0 onlyGenerative0.366 (+11.1%)0.638 (+61.5%)1.702 (+8.3%)
Mel predictor, all tGenerative0.409 (+24.2%)0.684 (+73.3%)1.774 (+12.9%)
Velocity predictor, all t (our flow, N = 1)Generative0.417 (+26.5%)0.700 (+77.2%)1.686 (+7.4%)
Velocity predictor, all t (our flow, N = 8)Generative0.416 (+26.2%)0.722 (+82.9%)1.906 (+21.3%)

Results vary by metric but, crudely, can be summarized as follows: For ESTOI and UTMOS, the architecture alone provides little to no improvement over the best baseline (Wairagkar), even with (standard) data augmentation. But architecture alone does substantially improve speaker similarity. For all three metrics, the gap between the the discriminative baseline and our proposed model is cut by about one third merely by moving to a generative model that predicts mels in one step ("t=0 only"). The remaining two thirds is roughly evenly split between the effects of training at multiple times tt (multiple "noise levels" if this is viewed as a data augmentation, as rev. T2iS suggests) and iteration at test time, although the latter doesn't do much for ESTOI.

This breakdown is more nuanced than our original framing, and we have now updated our claims in the abstract and introduction to reflect this.

AuthorsOpenReview ↗

more general comments to all reviewers

R3. T15

We also make a similar set of analyses on T15's Utah array data. Here we also take up three other reviewers' suggestions:

  1. Show results without Whisper (or any strong language model) [T2iS]
  2. Compare against the retrained (offline) Wairagkar-decoded audio, rather than the reported online versions [T2iS]
  3. Compute significance against the discriminative variant of our flow-matching model [jXct, T2iS, rXry]
MetricWairagkar et al. (online)Wairagkar et al. (offline repro.)CNN-DiT regressor (MSE)Ours (OT-CFM)
WER, Whisper ↓0.56 [0.29–0.83]0.33 [0.14–0.60]0.22 [0.00–0.44]0.22 [0.00–0.40] n.s.
WER, LM-free ↓1.00 [0.78–1.14]0.80 [0.67–1.00]0.71 [0.56–0.83]0.63 [0.44–0.80] ***
CER, LM-free ↓0.61 [0.50–0.73]0.48 [0.38–0.59]0.40 [0.30–0.50]0.34 [0.24–0.45] ***
rsp,DTWr_\text{sp,DTW} ↑0.850 ± 0.0390.874 ± 0.0390.890 ± 0.0360.898 ± 0.037 ***
rspr_\text{sp} (frame-aligned) ↑0.658 ± 0.0840.621 ± 0.0910.604 ± 0.0960.570 ± 0.095 n.s.
MCD w/o sil. (dB) ↓6.73 ± 0.666.07 ± 0.675.57 ± 0.645.43 ± 0.70 ***
MCD w/ sil. (dB) ↓2.96 ± 0.532.66 ± 0.472.43 ± 0.452.37 ± 0.45 ***
ESTOI$_\text{DTW}$ ↑0.253 ± 0.0550.319 ± 0.0580.355 ± 0.0640.404 ± 0.069 ***
ESTOI ↑0.177 ± 0.0570.213 ± 0.0620.210 ± 0.0660.204 ± 0.066 n.s.
UTMOS ↑1.73 ± 0.161.85 ± 0.161.88 ± 0.161.90 ± 0.10 **
SpkSim ↑0.741 ± 0.0740.785 ± 0.0650.801 ± 0.0630.835 ± 0.051 ***

A few points:

  1. In the original ms., we reported that the discriminative version of our model (CNN-DiT) matches the generative version in WER (0.22). But in fact there is a performance difference, which was masked by Whisper: without it, the generative version is better on WER and CER (see the first three rows of the table)---again confirming that architecture alone is not responsible for our performance gains.

  2. Offline Wairagkar is substantially better than online Wairagkar. (It's not clear why, since both models, like ours, are strictly causal.) But the gap with our model still remains and is large.

  3. Our CFM model outperforms its discriminative counterpart on all metrics except the non-DTW versions of rspr_{sp} and ESTOI.

R4. Channel selection

Technically, as reviewers pointed out, our channel selection saw the test block. This is very unlikely to have much of an effect, since it constitutes just 1/8th of the data, but to be safe we re-computed all results (training, testing, everything) with per-fold channel selection (i.e., without access to the test block). Results are qualitatively identical. We have (nevertheless) updated Table 1 in the ms. with the new versions.

R5. Another dataset and data availability

Some reviewers asked to see model performance on another publicly available dataset. We have therefore now evaluated our model and the baselines on three subjects from a public ECoG dataset on speech perception (Zada et al. (2025, Sci. Data)). Results are broadly similar to results on in-house dataset; see Appendix B. We think this is a nice addition since it extends our results to cover all three of the major intracranial recording modalities (microelectrode arrays, ECoG, and sEEG), and speech perception as well as speech production.

We will also make our new in-house sEEG dataset available upon publication. (The T15 dataset is already public.)

Dear Reviewer,

Thank you very much for your time and thoughtful review. As the discussion deadline approaches, could you please take a look at the authors' rebuttal and share your thoughts in the discussion?

Many thanks for your continued help!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.