acceptodds
Under review as a conference paper at ICLR 2027

A Common Measure of Communication for Speech Brain-Computer Interfaces

Abstract

Speech decoding brain-computer interfaces (speech BCIs) translate neural activity into language, offering a path towards restoring speech for people with paralysis and, more broadly, enabling new forms of natural human-computer interaction. Despite this promise, the field lacks a common measure of progress because systems use different datasets, recording methods, types of speech, and vocabularies, so their reported scores are rarely comparable. Underlying this measurement problem are two unresolved questions: (i) what distribution of words should a speech BCI enable a user to communicate, and (ii) how much information from this distribution can a system convey. We address both by deriving open-vocabulary mutual information (OVMI), an information-theoretic quantity that measures the information conveyed by a decoder relative to a reference distribution over the words a user may wish to communicate. This allows capabilities measured under different conditions, such as distinct vocabularies, to be evaluated on a common reference scale. We show that ordinarily reported accuracy, word error rate (WER), and other metrics computed only over the words a system supports can overstate how much of a user's intended speech the system can communicate. We then use OVMI to compare existing systems, expose trade-offs between how much of the user's language a system supports and how accurately it decodes those words, show that these comparisons depend on what the user is expected to communicate, and demonstrate that selecting a vocabulary to maximise OVMI yields up to 16.3% relative improvement in accuracy across three speech domains. OVMI therefore provides the speech BCI community with a principled way to compare heterogeneous systems, improve vocabulary design, and measure progress in the field.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain-computer interfaces, neural decoding, thought-to-text, evaluation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_tq4EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work addresses a real problem of cross-study comparison in communication BCIs. Although this problem is broad, the paper focuses on the more specific challenge of comparing word error rates (WER) and accuracies across studies employing widely different vocabulary sizes and compositions. The proposed open-vocabulary mutual information (OVMI) metric captures how the vocabulary supported by a decoder relates to a broader reference language distribution, and how the decoder's performance within its supported vocabulary translates to an estimate of information conveyed over this reference distribution.

While useful, OVMI's main contribution appears to be enabling more meaningful comparisons between decoders evaluated on substantially different vocabulary sizes, particularly small versus large vocabularies. Describing it as a "common measure" of communication may therefore be an overstatement, as arguably some of the main obstacles to cross-study comparisons remain unaddressed, including differences in training and evaluation splits, evaluation protocols or experimental conditions.

The proposed metric also appears particularly relevant to studies in which word decoding is constrained to a predefined vocabulary. However, as the field increasingly moves toward effectively open-vocabulary decoding through language models (e.g., Cross-Species BIT, Zhang et al., 2025; BrainWhisperer, Boccato et al., 2026), the distinction between theoretically supported vocabulary and empirically demonstrated decoding capability becomes increasingly important.

Critical strengths
  • Quantitative and principled approach to comparing decoders evaluated on substantially different vocabulary sizes and compositions
  • Thorough mathematical formulation of the proposed metric
  • An informative analysis of how the choice of reference language distribution, a subjective but important component of the benchmark, affects the resulting scores and comparisons
Critical weaknesses
  • Limited consideration of evaluation vocabulary. The retrospective cross-study evaluation does not explicitly account for the size or composition of the vocabulary represented in the training and test sets. Instead, it primarily relies on the decoder's supported output vocabulary and its reported aggregate accuracy or WER. Consequently, it is unclear how reliably the estimated OVMI reflects decoding capability over the broader reference distribution when evaluation has been performed on a much narrower set of words.

  • Applicability to modern open-vocabulary decoders. The lexical coverage factor C(S) may be less informative for modern attempted-speech decoders that use phoneme-to-text language models, such as Cross-Species BIT (Zhang et al., 2025) and BrainWhisperer (Boccato et al., 2026). These systems can theoretically support almost unrestricted output vocabularies, resulting in C(S) values close to one, regardless of how broadly their decoding performance has been empirically validated. Their OVMI scores would therefore rely predominantly on decoding fidelity estimated from aggregate WER, potentially extrapolating performance from a limited test distribution to a much broader reference vocabulary. This distinction deserves further investigation, especially given the paper's ambition to provide a common measure applicable across speech BCI systems.

  • Clarity of the Willett et al. (2023) comparisons. The presentation of Willett et al. (2023) is somewhat confusing, as the study is initially discussed in terms of its 50-word vocabulary and only later in terms of its 125k-word vocabulary. Furthermore, the labels "Willett 2023 isolated" and "Willett 2023 +LM" could be clarified. The original work employed LM-based decoding components, such as the WFST model, and also evaluated additional LLM-based sentence rescoring. It would help to specify exactly which experimental configurations are represented by these labels and what is meant by "+LM" in each case.

Questions for authors
  • How does the size and lexical diversity of the test set affect the estimated OVMI score?
  • Does OVMI account for the vocabulary of the training and validation/test sets, or only the decoder's theoretically supported output vocabulary?
  • For open-vocabulary decoders, how does OVMI distinguish between theoretically supported words and words whose decoding performance has actually been validated?
Reviewer_3q1SOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces open vocabulary mutual information (OVMI) as a common information-theoretic scale for evaluating BCI’s with different experimental settings. The method is motivated by the fact that conventional methods are conditional on the words a decoder supports rather than the general intended lexical distribution the BCI can express. The method separates information about lexical identity from the fact that a word is merely present or not in a vocabulary. The work develops various estimators at different scales and compares non-invasive and invasive systems. The paper further demonstrates the practical usefulness of OVMI rigorously producing improvements over frequency based vocabulary selection across several language distributions.

Critical strengths

This is a simple, clear formulation of an important problem in BCI evaluation and provides value alongside established BCI benchmarks.

Evaluation is done at various levels and is comprehensive that in itself gives clear value for people in the BCI field.

The metric is theoretically motivated and practically useful.

Critical weaknesses

I see nothing as a significant weakness; but I am a bit skeptical of using 1- WER as a proxy for decoding accuracy in the retrospective comparisons, this is acknowledged but validation against word-aligned predctions would be helpful.

Questions for authors
  1. Appendix D3 measures approximation error using simulated channels, can you compare scalar OVMI with full-channel OVMI and report whether system rankings change?

  2. For the validation accuracy baseline in Appendix F can you make it clear if this baseline maximises in-vocabulary accuracy or open-vocabulary accuracy with out of vocabulary words counted as errors?

Reviewer_zSZ7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Open-Vocabulary Mutual Information (OVMI), which measures the lexical information that a speech brain-computer interface can convey by multiplying in-vocabulary mutual information with vocabulary coverage under the target language distribution. The authors use this metric to re-compare existing systems, analyze the impact of the reference language distribution on evaluation results, and apply it to vocabulary selection, achieving up to a 16.3% relative accuracy improvement across three speech domains.

Critical strengths

The problem is practically meaningful. Different speech BCIs use different vocabularies, and comparing only accuracy or WER can obscure differences in expressive coverage. This paper explicitly introduces the target language distribution, providing a clearer application context for evaluation.

The theoretical formulation is clear. Under the out-of-vocabulary abstention and word-level channel assumptions, the mutual information decomposition and the derivation of OVMI are fairly complete; the authors also provide three computation methods: full confusion matrix, per-word accuracy, and scalar statistics.

It addresses both evaluation and optimization. Beyond retrospective analysis, the vocabulary selection experiments demonstrate the metric's potential as an optimization objective; results under different reference distributions also show that system evaluation should be tied to users' communication needs.

Critical weaknesses

Weaknesses

  1. The scalar approximation is insufficient to support precise interpretation in cross-system comparisons. Section 3.2 assumes all words have identical accuracy and that errors are uniformly distributed. However, identical accuracy can correspond to different confusion structures and mutual information; therefore, accuracy, WER, and vocabulary alone cannot uniquely determine OVMI. Although Appendix D.3 provides simulation validation, it is fixed to the same 50-word vocabulary and is insufficient to verify ranking reliability under different vocabulary sizes, decoding paradigms, and language model conditions. Precise statements in the main text such as "conveys 93.7% of lexical information" should more explicitly flag their model dependence.

  2. There is an important gap in the conversion from WER to the word-level channel. The authors note that 1−WER1-\mathrm{WER} is a conservative proxy for the proportion of correctly recovered words, but this does not mean that the OVMI obtained by plugging it into a symmetric channel is also a lower bound on the actual mutual information. Moreover, WER is a frequency-weighted sequence metric, whereas the scalar estimator requires macro accuracy; the two are not equivalent. Deletions, insertions, and context-dependent errors also cannot be adequately described by a single accuracy parameter. It is recommended to use available aligned predictions to examine the bias of this conversion and its impact on ranking.

  3. Cross-reference-distribution reweighting lacks validation on real data. Appendix D.2 explicitly states the channel invariance assumption, which is commendable. However, errors in continuous speech decoding may change with context and language model priors. Therefore, scores after changing the reference distribution represent a reweighting of an existing channel and cannot be directly regarded as system performance in a new domain. It is recommended to compare reweighted estimates with actually measured OVMI on data with multiple domains, and to clarify the scope of this interpretation.

  4. The evidence scope of the vocabulary optimization experiments is limited. All three domains come from the same subject in LibriBrain100, and the candidate word pool is limited to 250 words, which is insufficient to show that the gains generalize to other subjects, datasets, or decoders. Appendix Figure 9 already includes a validation-set accuracy selection baseline, but the main text mainly emphasizes the improvement over relative-frequency selection and lacks a direct comparison with this stronger baseline. In particular, since the final evaluation metric is open-vocabulary accuracy, it is necessary to explain the additional value of OVMI optimization over directly optimizing that accuracy.

Questions for authors
  1. Can you compare full-channel, per-word, and scalar OVMI on real decoding results, and test whether rankings remain stable across different vocabulary sizes and decoders?

  2. For systems that report only WER, how do you handle the difference between micro and macro accuracy? Do the existing uncertainty intervals reflect only sampling error, rather than also including errors from channel approximation and vocabulary substitution?

  3. In Figure 9, does OVMI selection show a stable advantage over validation-set accuracy selection? If the two are close, what is the unique benefit of OVMI as a vocabulary optimization objective?

Comments on the paper

1 comment

Dear Reviewer,

Thank you very much for your time and thoughtful review. As the discussion deadline approaches, could you please take a look at the authors' rebuttal and share your thoughts in the discussion?

Many thanks for your continued help!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.