- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper adapts conditional flow matching to synthesize acoustic features from intracranial neural recordings. A temporal CNN conditions a DiT velocity model, which is sampled with an ODE solver and vocoded to speech. The method is evaluated on two small sEEG datasets and the public T15 microelectrode dataset, including an architecture-matched MSE CNN-DiT control.
Critical strengths
- The paper gives a potential generative alternative to direct neural-to-spectrogram regression and evaluates it across both sEEG and microelectrode recordings. The architecture-matched MSE control, solver-budget analysis, and causal T15 setting are useful experimental design choices.
- The evaluation is multi-faceted and relatively transparent. The paper reports metrics that do not favor the proposed model, and the supplied code, configuration, and checkpoint make important parts of the T15 evaluation inspectable.
Critical weaknesses
Overall, This paper is technically incremental. The methodological novelty and task-specific mechanism are limited. The pipeline is essentially Beyond standard neural-signal preprocessing and a temporal CNN conditioner, the method transfers established CFM, DiT, ODE sampling, and vocoder components to sEEG. Its central task-specific premise is that neural recordings underdetermine acoustics, so MSE conditional means blur speech and stochastic refinement yields a more plausible sample; this premise is not directly validated, and no sEEG-specific module or analysis establishes that the samples capture neural uncertainty rather than merely change the distortion--perception trade-off. The matched T15 control reinforces this concern: it ties flow on WER (0.22) and is better on non-DTW ESTOI (0.210 vs. 0.204) and frame-aligned correlation (0.604 vs. 0.570), while flow is better on DTW/perceptual metrics. Reported significance tests are against released Wairagkar audio rather than the matched control.
Questions for authors
- The paper states that saliency selection was made on one fold and then checks its stability across blocks. However, this does not remove the possibility that held-out data influenced the shared feature-selection step. Please repeat the NZ evaluation with channel selection nested inside each training fold, or fixed from an independent dataset, and report how the main results change.
- What property of sEEG makes the standard CFM/DiT transfer necessary beyond a generic stochastic decoder? Please provide paired flow-versus-matched-MSE comparisons for all T15 metrics, and explain why DTW-aligned/perceptual improvements outweigh tied WER and worse frame-aligned metrics for the intended use case.
2 comments
R[2/2]: Modeling the conditional distribution improves perceptual quality and intelligibility
Why DTW-aligned/perceptual improvements outweigh tied WER and worse frame-aligned metrics for the intended use case In fact, once the language model is removed, the CFM model outperforms its discriminative counterpart even on WER and on CER; please see point R3 T15 in our general replies. We would also emphasize that WER is not the only or most important test of performance; if it were, we would simply decode text rather than speech audio.
Addressing the ethics flag. To promote reproducibility of neural speech synthesis methods in speech BCI, we have decided to release our in-house dataset after publication as per the IRB protocol. We understand that speech audio data is potentially person-identifiable, therefore, we will use random pitch shifting to anonymize both participants' (both P0 and P1) voice in the public release as in previous research (Verwoert et al. 2022, Sci. Data).
R[1/2]: Modeling the conditional distribution improves perceptual quality and intelligibility
We thank the reviewer for the their review. Below we answer the questions raised by the reviewer. Relevant results and explanations from the general response are cited rather than duplicated.
Validating the central premise (W1) The reviewer is concerned that we haven't shown directly (i) that "MSE conditional means blur speech and stochastic refinement yields a more plausible sample'' and (ii) "that samples capture neural uncertainty rather than merely change the distortion--perception trade-off.'' We agree that our demonstration in the original ms. is indirect, and have therefore added analyses addressing these two questions.
That MSE losses blur speech we believe has been firmly established in the TTS literature (see e.g. "Revisiting Over-Smoothness in Text to Speech," Ren et al. 2022). That the generative model yields a more plausible sample than a matched discriminative model we have shown directly in the ms., and our new results under our general comment on The role of flow matching in decoding performance. What remains is to show (ii).
A move along the distortion--perception frontier must pay in distortion for whatever superior perceptual quality it achieves. In our proposed model, this is not the case. That is, compared to the architecture-matched discriminative version of our model, a single draw from our CFM model has essentially the same frame-wise correlation, but superior intelligibility, spectral distortion, speaker similarity, and naturalness at once:
(In addition to the metrics used in the ms., we include kernel inception distance [KID], a measure of distributional match between synthesized and real speech.) Evidently, the CFM is on a different (superior) frontier of the perception--distortion trade-off, not merely a different position on the discriminative model's frontier. (We emphasize that the discriminative model is a strong baseline that already beats prior art.)
These analyses consider single draws; we turn now to their distribution. If the spread across samples were unrelated to the neural evidence, it would not change when the recording is degraded. Here we degrade the conditioning by \emph{graded} phase randomization: At each frequency, we add the same random phase to each of x% of the channels. When x=100, all statistics above second-order are wiped out. For each test trial, we draw K=16 samples, score each draw, take the standard deviation of that trial's scores, and average it over the 20 trials of one held-out block per participant:
Corrupting the recording increases the spread 6.4× on P0 and 4.5× on P1: as the neural evidence is destroyed the model becomes measurably less certain about its own prediction, so (as expected) the samples do carry information about how well the recording determines the acoustics.
Reporting significance against the matched (discriminative) control As requested, in the updated version of the ms., we now test the significance of our proposed flow model against the matched discriminative model (rather than Wairagkar). See R3 T15 in our general replies for more details.
Channel selection Please see point R4 in our general replies.
What property of sEEG makes this transfer necessary We do not claim that a property unique to sEEG makes CFM necessary. Rather, the incomplete and noisy nature of neural recordings motivates modeling the conditional distribution of speech. CFM provides a tractable way to learn this distribution. Our ablations separate the contributions of the CNN-DiT architecture, noise augmentation, training objective, and multi-step inference. They show that adding noise alone does not reproduce the gains of training across the interpolation path, while multi-step inference provides additional perceptual benefits. These findings support our choice of CFM, but do not rule out other conditional generative models.