- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work addresses a real problem of cross-study comparison in communication BCIs. Although this problem is broad, the paper focuses on the more specific challenge of comparing word error rates (WER) and accuracies across studies employing widely different vocabulary sizes and compositions. The proposed open-vocabulary mutual information (OVMI) metric captures how the vocabulary supported by a decoder relates to a broader reference language distribution, and how the decoder's performance within its supported vocabulary translates to an estimate of information conveyed over this reference distribution.
While useful, OVMI's main contribution appears to be enabling more meaningful comparisons between decoders evaluated on substantially different vocabulary sizes, particularly small versus large vocabularies. Describing it as a "common measure" of communication may therefore be an overstatement, as arguably some of the main obstacles to cross-study comparisons remain unaddressed, including differences in training and evaluation splits, evaluation protocols or experimental conditions.
The proposed metric also appears particularly relevant to studies in which word decoding is constrained to a predefined vocabulary. However, as the field increasingly moves toward effectively open-vocabulary decoding through language models (e.g., Cross-Species BIT, Zhang et al., 2025; BrainWhisperer, Boccato et al., 2026), the distinction between theoretically supported vocabulary and empirically demonstrated decoding capability becomes increasingly important.
Critical strengths
- Quantitative and principled approach to comparing decoders evaluated on substantially different vocabulary sizes and compositions
- Thorough mathematical formulation of the proposed metric
- An informative analysis of how the choice of reference language distribution, a subjective but important component of the benchmark, affects the resulting scores and comparisons
Critical weaknesses
-
Limited consideration of evaluation vocabulary. The retrospective cross-study evaluation does not explicitly account for the size or composition of the vocabulary represented in the training and test sets. Instead, it primarily relies on the decoder's supported output vocabulary and its reported aggregate accuracy or WER. Consequently, it is unclear how reliably the estimated OVMI reflects decoding capability over the broader reference distribution when evaluation has been performed on a much narrower set of words.
-
Applicability to modern open-vocabulary decoders. The lexical coverage factor C(S) may be less informative for modern attempted-speech decoders that use phoneme-to-text language models, such as Cross-Species BIT (Zhang et al., 2025) and BrainWhisperer (Boccato et al., 2026). These systems can theoretically support almost unrestricted output vocabularies, resulting in C(S) values close to one, regardless of how broadly their decoding performance has been empirically validated. Their OVMI scores would therefore rely predominantly on decoding fidelity estimated from aggregate WER, potentially extrapolating performance from a limited test distribution to a much broader reference vocabulary. This distinction deserves further investigation, especially given the paper's ambition to provide a common measure applicable across speech BCI systems.
-
Clarity of the Willett et al. (2023) comparisons. The presentation of Willett et al. (2023) is somewhat confusing, as the study is initially discussed in terms of its 50-word vocabulary and only later in terms of its 125k-word vocabulary. Furthermore, the labels "Willett 2023 isolated" and "Willett 2023 +LM" could be clarified. The original work employed LM-based decoding components, such as the WFST model, and also evaluated additional LLM-based sentence rescoring. It would help to specify exactly which experimental configurations are represented by these labels and what is meant by "+LM" in each case.
Questions for authors
- How does the size and lexical diversity of the test set affect the estimated OVMI score?
- Does OVMI account for the vocabulary of the training and validation/test sets, or only the decoder's theoretically supported output vocabulary?
- For open-vocabulary decoders, how does OVMI distinguish between theoretically supported words and words whose decoding performance has actually been validated?
Dear Reviewer,
Thank you very much for your time and thoughtful review. As the discussion deadline approaches, could you please take a look at the authors' rebuttal and share your thoughts in the discussion?
Many thanks for your continued help!