acceptodds
Under review as a conference paper at ICLR 2027

MindVoice: Towards Intelligible Speech Reconstruction from Non-invasive Neural Signals via Linguistic Recovery

Abstract

Reconstructing speech from neural activity offers a direct window into human auditory perception and holds promise for safe and scalable speech brain-computer interfaces. Yet intelligible reconstruction from non-invasive neural signals remains challenging. Existing methods are predominantly acoustic-centric, directly mapping noisy neural recordings to entangled speech representations, which can preserve acoustic characteristics while failing to reliably recover the underlying linguistic content. We introduce *MindVoice*, a neural speech reconstruction framework that addresses this *intelligibility gap* through explicit linguistic recovery. A linguistic reconstruction stream estimates what was perceived, while a complementary acoustic stream captures fine-grained characteristics of how it was spoken. Leveraging pretrained speech priors to compensate for incomplete neural evidence, *MindVoice* integrates both streams through text-to-speech generation and in-context voice cloning to synthesize natural, intelligible speech. Experiments on EEG and MEG datasets demonstrate substantial intelligibility gains over existing non-invasive methods. Further analyses reveal an empirical trade-off between spectral fidelity and linguistic intelligibility, with *MindVoice* achieving a favorable balance. Together, these findings support explicit linguistic recovery as an effective route toward intelligible non-invasive neural speech reconstruction.

Primary area:
applications to neuroscience & cognitive science
Keywords:
neural decoding, neural speech reconstruction, neuroscience

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CZYUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MindVoice. MindVoice reconstructs perceived speech from EEG/MEG using two streams: an acoustic stream predicts Fish-Speech codec tokens, while a linguistic stream maps neural signals through a VQ mel representation and adapted Whisper to text. A frozen Fish-Speech generator combines predicted text and an acoustic prompt. The paper reports large WER improvements over acoustic baselines on Brennan EEG and MEG-MASC, plus stream ablations and noise-input controls.

Critical strengths
  1. The paper clearly exposes the tension between waveform/spectral fidelity and linguistic intelligibility. The acoustic-only, linguistic-only, and noise-stream controls are helpful for analyzing the two pathways.
  2. The method is technically concrete and the reported gains are large, making the question important if the evaluation split is leakage-free.
  3. The manuscript is generally well written and easy to follow. It provides a well-motivated approach, broad quantitative evaluation across multiple metrics, and supplementary interpretability and qualitative analyses.
Critical weaknesses
  1. The split protocol is insufficiently specified for continuous narrative recordings. The paper only states an 8:1:1 split of processed samples, while MEG and EEG samples are derived from the same stories/recordings and may be temporally adjacent. Without a story/session/time-block-disjoint split and explicit identifiers, the powerful adapted Whisper/Fish-Speech priors may exploit shared lexical or acoustic context rather than demonstrate out-of-context neural reconstruction. This is material to the central WER claim. This risk has also been raised in prior work on continuous narrative neural decoding. Moreover, the supplementary material contains only 60 audio examples and a manifest, while the paper states that code will be released in the future; the split and implementation therefore cannot currently be audited. The reported 95% CIs span five training seeds rather than independent participants, stories, or test samples, and thus do not quantify generalization uncertainty over those data units.
  2. The paper does not provide a well-supported rationale for selecting the Fish-Speech codec/TTS backbone, nor does it evaluate whether the conclusions are sensitive to this choice. Given that the final speech quality and intelligibility are strongly shaped by a frozen pretrained generator, it is unclear why other contemporary codec/TTS systems (e.g., CosyVoice or OmniVoice, where compatible) were not considered.
  3. The claimed “favorable balance” between spectral fidelity and intelligibility appears rather forced. Relative to MindVoice-A, MindVoice substantially worsens spectral metrics (EEG mel-MSE/mel-r: 0.168/0.555 to 0.287/0.382; MEG: 0.110/0.725 to 0.329/0.342) while improving WER. Moreover, in the MEG ablation, text-only MindVoice-L has better WER and ASR-BERT than the complete model (39.3%/0.516 vs. 43.5%/0.488). The paper gives no pre-specified utility criterion or human preference/intelligibility evaluation to establish that the selected operating point is favorable.
  4. The codebook design is insufficiently justified and ablated. The Fish-Speech codec itself is frozen rather than constructed by the authors, but the linguistic stream introduces a learned VQ autoencoder. The paper specifies the codebooks and loss weights but does not study the effect of VQ codebook size, structure, or loss design. It is therefore unclear whether the reported gains depend on these untested design choices.
Questions for authors

Please see Weaknesses 1--4. If the authors' response comprehensively and convincingly addresses these concerns, I would be willing to raise my rating.

  1. Exactly what is the split unit (utterance, sentence, recording, story, or subject)? Please release the split identifiers and report results with held-out story/session/time blocks that prevent adjacent or overlapping narrative context from crossing train/validation/test.
  2. Please justify the selection of Fish-Speech and clarify whether the results are robust to alternative compatible codec/TTS backbones.
  3. Please explain why the reported spectral-intelligibility trade-off should be considered favorable, ideally with a pre-specified decision criterion or human intelligibility/preference evaluation. Why is the complete model preferred when MindVoice-L has better MEG lexical scores?
  4. Please provide ablations of the learned VQ component, including codebook size, structure, and loss weights, and clarify how sensitive the main conclusions are to these choices.
Reviewer_pPmHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MindVoice reconstructs perceived speech from EEG/MEG through explicit linguistic recovery. A linguistic stream recovers transcripts, while an acoustic stream predicts codec tokens; a frozen Fish-Speech TTS model combines these outputs to generate speech. The central contribution is to improve the intelligibility of reconstructed speech by explicitly recovering linguistic content, rather than relying on acoustic reconstruction alone. Experiments on Brennan EEG and MEG-MASC report WERs of 38.1% and 44.3%, respectively, averaged over five training seeds. These results show substantial improvements over the evaluated acoustic baselines, while also revealing a trade-off with spectral fidelity.

Critical strengths
  • The paper addresses an important limitation of acoustic-centric decoding and reports substantial improvements in intelligibility across both EEG and MEG. I particularly appreciate that evaluation measures the linguistic content of the final generated waveform, rather than only the intermediate transcript. This makes the reported gains directly relevant to the speech reconstruction task.
  • The single-stream comparisons help clarify the roles of the two components. In the reported comparison, MindVoice-L achieves better WER than the full model, supporting the central importance of explicit linguistic recovery. Acoustic conditioning, in turn, improves selected fidelity metrics. These results offer a useful perspective on the trade-off between intelligibility and acoustic fidelity, rather than suggesting that combining the streams uniformly improves all aspects of reconstruction.
Critical weaknesses
  • Broader baseline coverage and clearer reproduction details would strengthen the comparative claims. Table 1 includes FESDE and constructed/internal baselines, but does not include the cited parallel-phoneme-prediction approach of Lee et al. (2025), “Enhancing Listened Speech Decoding from EEG via Parallel Phoneme Sequence Prediction”. Given the paper’s emphasis on linguistic recovery, this seems to be a particularly relevant comparison. Including it, or explaining the limitations that prevent a meaningful comparison, would help contextualize the contribution. It would also be helpful to distinguish more clearly the contribution of MindVoice from an existing neural-to-text decoder followed by the same TTS model. MindVoice-L is a useful internal reference, but its results appear to be limited to a single MEG run, and the paper explicitly notes that it should not be interpreted as a BrainECHO comparison. Finally, additional details about the “FESDE-style” implementation—particularly pretrained initialization, training budget, and checkpoint selection—would make the comparison easier to assess and reproduce. These concerns do not imply that the baselines were implemented incorrectly; rather, the current evidence leaves some uncertainty about the strength of the comparative and state-of-the-art claims.
  • Clarifying the data split would help establish the scope of generalization. Appendix B.1 specifies an 8:1:1 split, but I could not determine whether observations corresponding to identical stimulus utterances are grouped across subjects or repeated trials. This distinction matters because reconstruction of unseen linguistic content and reconstruction from new neural observations of previously encountered content represent different generalization settings. I do not view this ambiguity as evidence of leakage. However, clarifying the split construction and any overlap would help readers interpret what the reported WER improvements demonstrate.
Questions for authors
  • Could you clarify whether the splits are stimulus-disjoint and report any overlap in stimulus utterances, transcripts, and subjects across training, validation, and test sets? If stimuli overlap, could you also report performance on held-out utterances to help establish generalization to unseen content?

  • Could you specify which FESDE configuration was reproduced, including pretrained initialization, architectural changes, training budget, and checkpoint selection?

  • Could you explain why Lee et al. (2025) was not included in the comparison? A matched BrainECHO-plus-TTS comparison would also help clarify the contribution beyond existing neural-to-text decoding. Alternatively, could you explain to what extent MindVoice-L is equivalent to such a pipeline? Multi-seed results on both datasets would make this comparison more informative.

Reviewer_5yoLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, the author proposed MindVoice, a neural speech reconstruction framework that addresses this intelligibility gap through explicit linguistic recovery. In the experiments, the author showed some improvements in metrics such as mel-MSE, WER, and WavLM representation measures.

Critical strengths

The paper provides a good review of other works done in the literature, which is good for researchers who are new to the field.

Critical weaknesses

My biggest concern is that I strongly think the gain in "intelligibility" is purely from the decoder of the "linguistic recovery", and therefore, it is not fair to claim that "Together, these findings support explicit linguistic recovery as an effective route toward intelligible non-invasive neural speech reconstruction." It could simply be that the decoder is mapping the vocabulary of the dataset, how did the model perform on unseen sentences or vocabulary?

Moreover, I am very skeptical about the EEG-to-speech WER of 38.1% given how noisy the EEG signals are, and this could well just be the signals from the decoder. Therefore, I think the author should consider comparing a structure that does not require a decoder, since if the signals are mostly coming from the decoder, it is hard to estimate what is really being learned in the brain signal.

I think, from a technical point of view, the architecture isn't very exciting, as it combines different modules for synthesis. But in my opinion is fine. What is challenging to understand is the effectiveness of different blocks or components introduced in the framework. There is not a very strong connection or ablation study looking at which components contribute most to the synthesis, and as I suspect, most improvements might be from the pre-trained decoder of the model.

Also, how are the speaker characteristics and other properties like speaking rate and pitch being reconstructed? I feel Figure 4 could be used to answer these questions, and I am not entirely convinced to put reconstruction length comparisons in the main paper.

Questions for authors

My biggest concern is that I strongly think the gain in "intelligibility" is purely from the decoder of the "linguistic recovery", and therefore, it is not fair to claim that "Together, these findings support explicit linguistic recovery as an effective route toward intelligible non-invasive neural speech reconstruction." It could simply be that the decoder is mapping the vocabulary of the dataset, how did the model perform on unseen sentences or vocabulary?

Moreover, I am very skeptical about the EEG-to-speech WER of 38.1% given how noisy the EEG signals are, and this could well just be the signals from the decoder. Therefore, I think the author should consider comparing a structure that does not require a decoder, since if the signals are mostly coming from the decoder, it is hard to estimate what is really being learned in the brain signal. Could the author experiment without a decoder and see how it performs? Or test on unseen sentences, and if the performance remains high, I think it is highly possible that it is purely rely on the pre-trained knowledge of the decoder.

I think, from a technical point of view, the architecture isn't very exciting, as it combines different modules for synthesis. But in my opinion is fine. What is challenging to understand is the effectiveness of different blocks or components introduced in the framework. There is not a very strong connection or ablation study looking at which components contribute most to the synthesis, and as I suspect, most improvements might be from the pre-trained decoder of the model. Could the author explain how effective or important each module is?

Also, how are the speaker characteristics and other properties like speaking rate and pitch being reconstructed? I feel Figure 4 could be used to answer these questions, and I am not entirely convinced to put reconstruction length comparisons in the main paper. Could the authors provide more speech-based metrics or measures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.