- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MindVoice. MindVoice reconstructs perceived speech from EEG/MEG using two streams: an acoustic stream predicts Fish-Speech codec tokens, while a linguistic stream maps neural signals through a VQ mel representation and adapted Whisper to text. A frozen Fish-Speech generator combines predicted text and an acoustic prompt. The paper reports large WER improvements over acoustic baselines on Brennan EEG and MEG-MASC, plus stream ablations and noise-input controls.
Critical strengths
- The paper clearly exposes the tension between waveform/spectral fidelity and linguistic intelligibility. The acoustic-only, linguistic-only, and noise-stream controls are helpful for analyzing the two pathways.
- The method is technically concrete and the reported gains are large, making the question important if the evaluation split is leakage-free.
- The manuscript is generally well written and easy to follow. It provides a well-motivated approach, broad quantitative evaluation across multiple metrics, and supplementary interpretability and qualitative analyses.
Critical weaknesses
- The split protocol is insufficiently specified for continuous narrative recordings. The paper only states an 8:1:1 split of processed samples, while MEG and EEG samples are derived from the same stories/recordings and may be temporally adjacent. Without a story/session/time-block-disjoint split and explicit identifiers, the powerful adapted Whisper/Fish-Speech priors may exploit shared lexical or acoustic context rather than demonstrate out-of-context neural reconstruction. This is material to the central WER claim. This risk has also been raised in prior work on continuous narrative neural decoding. Moreover, the supplementary material contains only 60 audio examples and a manifest, while the paper states that code will be released in the future; the split and implementation therefore cannot currently be audited. The reported 95% CIs span five training seeds rather than independent participants, stories, or test samples, and thus do not quantify generalization uncertainty over those data units.
- The paper does not provide a well-supported rationale for selecting the Fish-Speech codec/TTS backbone, nor does it evaluate whether the conclusions are sensitive to this choice. Given that the final speech quality and intelligibility are strongly shaped by a frozen pretrained generator, it is unclear why other contemporary codec/TTS systems (e.g., CosyVoice or OmniVoice, where compatible) were not considered.
- The claimed “favorable balance” between spectral fidelity and intelligibility appears rather forced. Relative to MindVoice-A, MindVoice substantially worsens spectral metrics (EEG mel-MSE/mel-r: 0.168/0.555 to 0.287/0.382; MEG: 0.110/0.725 to 0.329/0.342) while improving WER. Moreover, in the MEG ablation, text-only MindVoice-L has better WER and ASR-BERT than the complete model (39.3%/0.516 vs. 43.5%/0.488). The paper gives no pre-specified utility criterion or human preference/intelligibility evaluation to establish that the selected operating point is favorable.
- The codebook design is insufficiently justified and ablated. The Fish-Speech codec itself is frozen rather than constructed by the authors, but the linguistic stream introduces a learned VQ autoencoder. The paper specifies the codebooks and loss weights but does not study the effect of VQ codebook size, structure, or loss design. It is therefore unclear whether the reported gains depend on these untested design choices.
Questions for authors
Please see Weaknesses 1--4. If the authors' response comprehensively and convincingly addresses these concerns, I would be willing to raise my rating.
- Exactly what is the split unit (utterance, sentence, recording, story, or subject)? Please release the split identifiers and report results with held-out story/session/time blocks that prevent adjacent or overlapping narrative context from crossing train/validation/test.
- Please justify the selection of Fish-Speech and clarify whether the results are robust to alternative compatible codec/TTS backbones.
- Please explain why the reported spectral-intelligibility trade-off should be considered favorable, ideally with a pre-specified decision criterion or human intelligibility/preference evaluation. Why is the complete model preferred when MindVoice-L has better MEG lexical scores?
- Please provide ablations of the learned VQ component, including codebook size, structure, and loss weights, and clarify how sensitive the main conclusions are to these choices.