acceptodds
Under review as a conference paper at ICLR 2027

MAS-BIT: A Multi-Agent System for End-to-End BraIn-to-TexT Decoding

20 views

Abstract

Speech brain-computer interfaces aim to restore communication by translat- ing neural activity into text, yet end-to-end decoders continue to trail cascaded systems in accuracy. We introduce a Multi-Agent System for BraIn-to-Text (MAS-BIT), the first multi-agent framework for end-to-end brain-to-text decod- ing from intracortical recordings. MAS-BIT coordinates three specialized agents: a neural agent that interprets continuous neural representations, a phonetic agent that decodes soft phoneme posteriors, and an arbitration agent that integrates their complementary evidence with direct access to neural representations to generate the final transcription. The expert agents communicate with the arbiter through token-embedding-based latent communication, constructing continuous messages as probability-weighted combinations of token embeddings. These messages re- tain alternative token hypotheses and provide a differentiable interface through which sentence-level supervision jointly trains the neural encoder and specialized agents. MAS-BIT achieves state-of-the-art end-to-end decoding performance on the Brain-to-Text ’24 and ’25 benchmarks, with word error rates of 8.418% on ’24 and 4.878% on the ’25 public test set (5.288% on the private test set). These results narrow the gap to cascaded systems without ensembling independently trained models or using external language-model rescoring.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain-computer interfaces, brain-to-text decoding, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ACncOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MAS-BIT decodes attempted speech from intracortical recordings. Neural and phonetic experts send soft token messages to an arbiter, which also reads the neural recordings directly. The system builds on pretrained BIT encoders with a shared language-model backbone, separate adapters, and alternating updates, and it is evaluated on two participants.

Critical strengths
  • The contribution is a useful architectural advance. BIT already connects neural representations to language models [1]; LegoSLM bridges speech recognizers and language models through posterior-weighted embeddings[2]; CIPHER exchanges soft embeddings between language models without training [3]. MAS-BIT meaningfully extends these ideas by jointly training complementary neural and phonetic specialists through the final transcription objective.

  • For researchers with scarce neural supervision, this offers a way to use phonetic structure without forcing every decision through a discrete phoneme sequence. Because the arbiter sees the neural recordings directly, it can also recover information that the expert drafts leave out. Sharing pretrained weights and dropping a separate language-model rescoring stack make the approach practical.

References

[1] Yizi Zhang, Linyang He, Chaofei Fan, Tingkai Liu, Han Yu, Trung Le, Jingyuan Li, Scott Linderman, Lea Duncker, Francis R. Willett, Nima Mesgarani, and Liam Paninski. A Cross-Species Neural Foundation Model for End-to-End Speech Decoding. International Conference on Learning Representations, 2026.

[2] Rao Ma, Tongzhou Chen, Kartik Audhkhasi, and Bhuvana Ramabhadran. LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriors. Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 18171–18186, 2025.

[3] Chau Pham, Boyi Liu, Yingxiang Yang, Zhengyu Chen, Tianyi Liu, Jianbo Yuan, Bryan A. Plummer, Zhaoran Wang, and Hongxia Yang. Let Models Speak Ciphers: Multiagent Debate through Embeddings. International Conference on Learning Representations, 2024.

Critical weaknesses
  • The matched single-decoder comparison supports the architecture as a whole. It does not separate specialization from the extra adapters, the intermediate transcription supervision, and the repeated decoding. The comparison with published BIT results also changes the language-model backbone and the decoding procedure. The improvement is credible, but the claim that splitting the input views into specialist roles explains it is less secure.

  • The soft-message analysis shows that keeping alternatives helps for T12, yet collapsing messages to their top token barely changes aggregate performance for T15. The trained hard-text control removes both the alternatives and the backward gradient path, so it cannot show whether the gain comes from richer evidence at inference time or from better training.

  • The host-memory savings over the reproduced cascade are substantial, however, the efficiency analysis is incomplete. Inference needs the complete utterance, expert generation, replay, and arbitration, and the paper reports neither latency nor accelerator memory. MAS-BIT uses much less host memory than the conventional cascade tested here. However, memory-efficient alternatives such as LightBeam [4] already exist, so this comparison alone does not establish a deployment advantage over other available approaches.

References

[4] Ebrahim Feghhi, Junlin Hu, Nima Hadidi, and Jonathan C. Kao. LightBeam: An Accurate and Memory-Efficient CTC Decoder for Speech Neuroprostheses. arXiv preprint arXiv:2603.14002, 2026.

Questions for authors

Q1. Does specialization keep its advantage over a refinement baseline matched for adapters, supervision, and decoding passes?

Q2. What happens when soft messages are kept but arbiter-to-expert gradients are stopped?

Q3. What latency and accelerator memory does the full pipeline need compared with a memory-efficient cascade?

9 comments

We sincerely thank the reviewer for a thorough, insightful review and for recognizing MAS-BIT as a practical, neural-grounded advance building upon BIT [1], LegoSLM [4], and CIPHER [5]. In the revised paper, we unify MAS-BIT into one cross-participant model across T12 and T15, lowering holdout WER from 8.418% and 4.878%/5.288% to 6.929% and 3.606%/3.892% (-32.2%/-53.5% relative to BIT's ensembles, and surpassing BIT's 4.06% cascade on T15 public). All new controls and benchmarks below use this unified framework.

“Q1. Does specialization keep its advantage over a refinement baseline matched for adapters, supervision, and decoding passes?”

Response: Thank you for proposing this exact control. Yes—dedicated neural and phonetic specialization outperforms sequential dual-view refinement across all three holdout splits (Section 5.1, Appendix A.6, Table 16). Under the identical frontend, 65-epoch schedule, and selection rule, Sequential dual-view refinement matches MAS-BIT's 3×rank-8 adapters, three sentence CE + CTC losses, and three generations + two top-16 replays (R1R_1: [ZN;ZP]→M1[Z_N;Z_P]\to M_1; R2R_2: [ZN;ZP;M1]→M2[Z_N;Z_P;M_1]\to M_2; AA: [ZN;M1;M2][Z_N;M_1;M_2]), evaluated alongside Two neural experts and single-agent controls with Rank-24 or Borealis-5B's 4B audio-LLM decoder [3] (chosen because Qwen3-ASR [6] publicly releases weights only up to 1.7B, with no ~5B / 3×1.7B3\times 1.7\text{B} variant):

Official Holdout WER (%)Capacity, Losses & PassesT12T15 PubT15 Priv
Single agent (rank-8 / Rank-24)1×r8 / 1×r24, 1 CE, 1 gen9.988 / 10.2316.121 / 5.7876.669 / 6.889
Single agent (Borealis-5B 4B, r13)+2.6% LoRA, 1 CE, 1 gen12.4788.5158.553
Sequential dual-view refinement3×r8, 3 CE + CTC, 3 gen + 2 replay7.1593.9394.739
Two neural experts (separate neural projectors)3×r8, 3 CE + CTC, 3 gen + 2 replay7.0513.9093.939
MAS-BIT (NN: ZNZ_N, PP: ZPZ_P, AA)3×r8, 3 CE + CTC, 3 gen + 2 replay6.9293.6063.892

MAS-BIT improves upon Sequential dual-view refinement by 0.23 pp on T12 and 0.33/0.85 pp (8.5%/17.9% relative) on T15 public/private; sentence-pooled RSA confirms more distinct geometry between ZNZ_N and ZPZ_P (0.582/0.639) than between two neural views (0.950/0.949; Figure 6C, Appendix E.3).

“Q2. What happens when soft messages are kept but arbiter-to-expert gradients are stopped?”

Response: Thank you for suggesting this direct test. (1) Stopped message gradients: Training Detached latent messages—keeping top-16 weighted embeddings, message dropout, ZNZ_N, and expert CE losses while detaching replay logits—raises holdout WER to 7.349% (+0.42 pp) on T12 and 3.666%/4.315% (+0.06/+0.42 pp) on T15 (vs. Hard-text: 7.159% and 4.181%/4.409%; Table 16). (2) Fixed-weight Top-kk: Across all 880 T12 and 1,426 T15 validation trials in FP32 with weights, drafts, and ZNZ_N fixed, varying k∈1,2,4,8,16k \in \\{1,2,4,8,16\\} cuts edits monotonically on both participants: 598→563→563→562→557 (-41 edits, -0.746 pp) on T12 and 449→429→425→421→420 (-29 edits, -0.307 pp) on T15 (Figure 5, Table 7). Both inference alternatives and message gradients contribute.

“Q3. What latency and accelerator memory does the full pipeline need compared with a memory-efficient cascade?”

Response: Thank you for pointing us to LightBeam [2]. Across all 1,200 T12 and 1,450 T15 holdout utterances in FP32 (batch size 1)—covering encoding, drafting, replay, and arbitration—we benchmark MAS-BIT against the K=100K=100 WFST + OPT-6.7B cascade and BIT + LightBeam (beam 1,000/900, 100k lexicon, 4-gram KenLM, Llama-3.2-1B adapter) on server and consumer GPUs (Section 5.4, Appendix C, Tables 10–12):

Platform & Method (FP32, batch 1)RAM T12/T15 (GiB)GPU T12/T15 (GiB)Decode T12/T15 (s)
A40 — Cascade (K=100K=100)321.8 / 334.825.211 / 25.2742,789.6 / 3,761.6 (1.8×)
A40 — BIT + LightBeam17.2 / 31.812.553 / 14.7503,563.5 / 6,008.0 (2.3×/2.9×)
A40 — MAS-BIT (ours)12.7 / 31.18.559 / 8.5651,538.8 / 2,091.4 (1.28/1.44 s/utt)
RTX 5090 Laptop — LightBeam17.3/32.012.629/14.7853,204.6/6,021.1
RTX 5090 Laptop — MAS-BIT12.6/31.08.662/8.6481,210.2/1,690.6
RTX 4090 Desktop — LightBeam17.4/32.012.648/14.8871,483.5/2,332.7
RTX 4090 Desktop — MAS-BIT12.8/31.28.695/8.695550.5/775.1

Sharing one frozen 1.7B backbone across three rank-8 adapters keeps MAS-BIT at 8.559–8.695 GiB peak GPU memory in FP32 (vs. 12.553–14.887 GiB for LightBeam) while decoding 2.3×–3.6× faster (2.6×/3.6× on 5090 Laptop; 2.7×/3.0× on 4090 Desktop).

[1] Yizi Zhang, Linyang He, Chaofei Fan, Tingkai Liu, Han Yu, Trung Le, Jingyuan Li, Scott Linderman, Lea Duncker, Francis Willett, Nima Mesgarani, and Liam Paninski. A cross-species neural foundation model for end-to-end speech decoding. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (eds.), International Conference on Learning Representations, volume 2026, pp. 30773–30797, 2026. URL ICLR 2026 paper.

[2] Ebrahim Feghhi, Junlin Hu, Nima Hadidi, and Jonathan C. Kao. Lightbeam: An accurate and memory-efficient ctc decoder for speech neuroprostheses, 2026. URL arXiv:2603.14002.

[3] VikhrModels. Borealis: Audio-language model for speech understanding, 2025. URL Borealis-5b-it model card.

[4] Rao Ma, Tongzhou Chen, Kartik Audhkhasi, and Bhuvana Ramabhadran. LegoSLM: Connecting LLM with speech encoder using CTC posteriors. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 18171–18186, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.985. URL ACL Anthology.

[5] Chau Pham, Boyi Liu, Yingxiang Yang, Zhengyu Chen, Tianyi Liu, Jianbo Yuan, Bryan Plummer, Zhaoran Wang, and Hongxia Yang. Let models speak ciphers: Multiagent debate through embeddings. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (eds.), International Conference on Learning Representations, volume 2024, pp. 51899–51928, 2024. URL ICLR 2024 paper.

[6] Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, and Junyang Lin. Qwen3-ASR technical report, 2026. URL arXiv:2601.21337.

[7] Zehui Feng, Weichuan Wang, Xiaohan Chen, Cuntai Guan, and Ting Han. Do speech BCIs need larger models? Rethinking neural decoding beyond scaling, 2026. URL arXiv:2610.23175.

We also added a concurrent October 2026 (post-ICLR-submission) baseline, BEST (no external LM) [7] (12.61% T12 holdout WER; 5.17% T15 holdout WER, public/private subset unspecified; Table 1)—even with this latest work included, MAS-BIT remains the state-of-the-art end-to-end speech BCI decoder.

Reviewer_KiVBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MAS-BIT combines neural and phonetic experts with an arbiter using latent messages and staged training. Benchmark comparisons and ablations on two participants show improved end-to-end decoding, with WERs of 8.418% on T12 and 4.878%/5.288% on T15 public/private subsets.

Critical strengths

MAS-BIT reduces WER by 17.6% on T12 and 37.1% on T15 public compared with the published BIT end-to-end ensemble. These are substantial improvements without combining independently trained models or using external language-model rescoring.

Using the same backbone, encoder initialization, and data, MAS-BIT outperforms a single agent receiving both input views by 3.22 points on T12 and 2.64/2.15 points on T15 public/private. This supports the value of the full architecture.

On validation data, the arbiter makes fewer word errors than an oracle that selects the better complete expert draft. It also correctly decodes 59 T12 and 52 T15 sentences that both experts get wrong, showing benefits beyond draft selection.

Critical weaknesses

The single-agent baseline uses one adapter and generates one transcript, while MAS-BIT uses three adapters, generates two drafts, and then produces a final transcript. It is unclear how much of the improvement comes from expert specialization rather than the extra parameters and computation.

The hard-text control both removes alternative tokens and blocks gradients through the messages. This comparison cannot separate the benefit of extra token choices from the benefit of letting the arbiter’s loss train the experts.

The reproduced cascade performs worse than the published BIT cascade on T15 public (5.939% vs. 4.06% WER). MAS-BIT also fine-tunes its encoder, while the cascade keeps it frozen. The comparison therefore cannot separate improvements from encoder adaptation and multi-agent decoding.

Questions for authors
  1. Does MAS-BIT still outperform a baseline with the same number of trainable parameters and decoding passes?

  2. What explains the cascade reproduction gap on T15 public (5.939% vs. 4.06% WER)? How does the cascade perform using MAS-BIT’s fine-tuned encoder and phoneme readout?

3 comments

We thank the reviewer for the thoughtful review and for recognizing MAS-BIT's gains over BIT's ensemble and single-agent baselines. In the revised paper, we unify MAS-BIT into one cross-participant model across T12/T15, lowering holdout WER from 8.418% and 4.878%/5.288% to 6.929% and 3.606%/3.892% (-32.2%/-53.5% vs. BIT's ensembles); all controls share this setup.

“Does MAS-BIT still outperform a baseline with the same number of trainable parameters and decoding passes?”

Response: Thank you for this central question. Yes—MAS-BIT outperforms controls matched in trainable parameters, backbone capacity, supervision, and passes (Section 5.1, Figure 2B–C, Appendix A.6, Table 16). Under the identical 65-epoch schedule and selection rule, Sequential dual-view refinement matches MAS-BIT's 3×rank-8 parameters, three CE + CTC losses, and three generations + two replays (R1R_1: [ZN;ZP]→M1[Z_N;Z_P]\to M_1; R2R_2: [ZN;ZP;M1]→M2[Z_N;Z_P;M_1]\to M_2; AA: [ZN;M1;M2][Z_N;M_1;M_2]). We also test single-agent controls with Rank-24 (α=96\alpha=96) and Borealis-5B's 4B audio-LLM decoder [3] (r13, +2.6% LoRA; chosen because Qwen3-ASR [6] publicly releases weights only up to 1.7B, with no ~5B / 3×1.7B3\times 1.7\text{B} variant):

Holdout WER (%)SetupT12T15 PubT15 Priv
Single agent (r8 / Rank-24)1×r8 / 1×r24, 1 gen9.988 / 10.2316.121 / 5.7876.669 / 6.889
Single agent (Borealis-5B 4B)r13 (+2.6%), 1 gen12.4788.5158.553
Sequential refinement3×r8, 3 gen + 2 replay7.1593.9394.739
MAS-BIT (ours)3×r8, 3 gen + 2 replay6.9293.6063.892

Capacity scaling leaves a 2.18–5.55 pp gap, while view specialization outperforms Sequential refinement across splits (-0.23 pp T12; -0.33/-0.85 pp, 8.5%/17.9% relative, T15).

“The hard-text control both removes alternative tokens and blocks gradients through the messages. This comparison cannot separate the benefit of extra token choices from the benefit of letting the arbiter’s loss train the experts.”

Response: Thank you for this insightful point; we added two tests (Section 5.1, Appendix B.3, Tables 7 & 16): (1) Training gradients (Detached latent messages): Keeps top-16 weighted embeddings while detaching replay logits to stop arbiter-to-expert gradients, raising WER to 7.349% (+0.42 pp) on T12 and 3.666%/4.315% (+0.06/+0.42 pp) on T15 (vs. Hard-text: 7.159% and 4.181%/4.409%). (2) Inference alternatives (fixed-weight Top-kk): Fixing weights, drafts, and ZNZ_N on all 880/1,426 validation trials, varying k∈1,2,4,8,16k \in \\{1,2,4,8,16\\} cuts edits from 598 to 557 (-41, -0.746 pp) on T12 and 449 to 420 (-29, -0.307 pp) on T15 (Figure 5, Table 7). Both contribute.

“What explains the cascade reproduction gap on T15 public (5.939% vs. 4.06% WER)? How does the cascade perform using MAS-BIT's fine-tuned encoder and phoneme readout?”

Response: Thank you for raising both questions. (1) Why we reproduced the cascade and the cause of the gap: Rather than overlooking the gap, we plotted both our reproduced cascade and the published BIT cascade in Figure 4B and Table 14. We reproduced the cascade specifically to evaluate candidate limits KK, as Zhang et al. [1] reported only K=100K=100 without smaller-KK results. Their published cascade used Stage-2 phoneme-only encoder checkpoints, whereas the public BIT release provides only Stage-3 sentence-decoding checkpoints; when we contacted them, the BIT authors told us they could no longer locate those phoneme-decoding weights. For a fair comparison across KK, MAS-BIT is initialized from the same released BIT encoder checkpoints used by our reproduced cascade (5.939% best T15 public at K=25–75K=25\text{--}75; 5.969% at K=100K=100). (2) Adapted-encoder cascade and closing the cascade gap: Rerunning the FP32 WFST + 5-gram + OPT-6.7B pipeline across all 13 limits up to K=100K=100 with MAS-BIT's fine-tuned encoder branch and CTC readout separates both effects (Section 5.4, Figure 4B, Tables 14–15):

Holdout WER (%)T12T15 PubT15 Priv
Published BIT cascade (K=100K=100)6.354.06—
Reproduced cascade, released BIT (K=100K=100)7.8775.9696.026
Fine-tuned-encoder cascade (K=100K=100)7.2954.6964.629
MAS-BIT (ours, greedy E2E)6.9293.6063.892

Encoder adaptation lowers K=100K=100 WER by 0.58/1.27/1.40 pp, yet MAS-BIT outperforms this adapted cascade at all 13 limits (-0.37/-1.09/-0.74 pp at K=100K=100). Even against the published BIT cascade, MAS-BIT is close on T12 (6.929% vs. 6.35%) and surpasses it on T15 public (3.606% vs. 4.06%, -0.454 pp / 11.2% relative) via greedy decoding without WFST or external LMs—substantially narrowing the end-to-end vs. cascade gap as the new end-to-end SOTA.

[1] Yizi Zhang, Linyang He, Chaofei Fan, Tingkai Liu, Han Yu, Trung Le, Jingyuan Li, Scott Linderman, Lea Duncker, Francis Willett, Nima Mesgarani, and Liam Paninski. A cross-species neural foundation model for end-to-end speech decoding. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (eds.), International Conference on Learning Representations, volume 2026, pp. 30773–30797, 2026. URL ICLR 2026 paper.

[3] VikhrModels. Borealis: Audio-language model for speech understanding, 2025. URL Borealis-5b-it model card.

[6] Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, and Junyang Lin. Qwen3-ASR technical report, 2026. URL arXiv:2601.21337.

[7] Zehui Feng, Weichuan Wang, Xiaohan Chen, Cuntai Guan, and Ting Han. Do speech BCIs need larger models? Rethinking neural decoding beyond scaling, 2026. URL arXiv:2610.23175.

We also added a concurrent October 2026 (post-ICLR-submission) baseline, BEST (no external LM) [7] (12.61% T12 holdout WER; 5.17% T15 holdout WER, public/private subset unspecified; Table 1)—even with this latest work included, MAS-BIT remains the state-of-the-art end-to-end speech BCI decoder, outperforming all prior and concurrent end-to-end methods by a substantial margin (6.929% on T12 and 3.606%/3.892% on T15 public/private). Since the discussion period is drawing to a close, we wanted to kindly check in to see whether our new experiments and clarifications have addressed all of your concerns. Please let us know if anything remains unclear—and if your concerns have been resolved, we would be sincerely grateful if you would consider raising your score in light of these revisions.

Reviewer_8uYfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a multi-agent system for end to end brain to text decoding, MAS-BIT. It utilizes a frozen backbone ASR model as the shared backbone for all the agents, with lightweight adapters to specialize each agents role. There are three agents, one a neural expert, a phonetic expert and an arbiter which takes the inputs of the first two experts to decode into text. A staged cirricular training is employed and latent communication is used instead of text, which enables gradient flow in an end to end manner during training. Experiments show improved performance as well as avoiding huge memory constraints since the agents share one set of backbone parameters and needs only the lightweight LORA params to distinguish them.

Critical strengths
  1. The separation of the neural expert and the phonetic expert is very interesting and a fresh perspective on how to approach this problem, allowing a richer representation to be extracted from the same signal via separate linguistic lenses.
  2. The latent communication which enables differentiability and allows training of the the neural encoder via all the downstream linguistic objectives is very nice.
Critical weaknesses
  1. The proposed method significantly reduces memory requirements and trained end to end, however the inference pipeline requires several sequential back and forth passes, which weakens the efficiency claims and also the end to end mechanism.
  2. Hallucination and error propagation from the experts to arbiter could be problematic, when both experts are wrong the arbiter alone cannot recover without an external LM.
Questions for authors

See weaknesses above.

4 comments

We sincerely thank the reviewer for providing a thoughtful review. We are glad the reviewer views separating the neural and phonetic experts through distinct linguistic lenses as a fresh perspective and recognizes the value of differentiable latent communication on a shared backbone. In the revised paper, we unify MAS-BIT into one cross-participant model jointly trained across T12 and T15, lowering holdout WER from 8.418% and 4.878%/5.288% to 6.929% and 3.606%/3.892%. All new experiments below use this unified model. We also added a concurrent October 2026 (post-ICLR-submission) baseline, BEST (no external LM) [7] (12.61% T12 holdout WER; 5.17% T15 holdout WER, public/private subset unspecified; Table 1)—even with this latest work included, MAS-BIT remains the state-of-the-art end-to-end speech BCI decoder.

“The proposed method significantly reduces memory requirements and trained end to end, however the inference pipeline requires several sequential back and forth passes, which weakens the efficiency claims and also the end to end mechanism.”

Response: We sincerely appreciate the reviewer bringing both points to our attention.

(1) Complete-pipeline efficiency across server and consumer GPUs. Although encoding, expert drafting, prefix replay, and arbitration run sequentially, all three roles share one frozen 1.7B backbone in GPU memory (switching rank-8 LoRA adapters) and decode short utterances (≤24/29\le 24/29 tokens) without lexicon beam search or candidate-list LLM rescoring. Across all 1,200 T12 and 1,450 T15 holdout utterances in FP32 (batch size 1), MAS-BIT outperforms both the K=100K=100 WFST + OPT-6.7B cascade [1] and BIT + LightBeam [2] (Section 5.4, Appendix C, Tables 10–12):

Platform & Method (FP32, batch 1)RAM T12/T15 (GiB)GPU T12/T15 (GiB)Decode T12/T15 (s)
A40 — Cascade (K=100K=100) [1]321.8 / 334.825.211 / 25.2742,789.6 / 3,761.6
A40 — BIT + LightBeam [2]17.2 / 31.812.553 / 14.7503,563.5 / 6,008.0
A40 — MAS-BIT (ours)12.7 / 31.18.559 / 8.5651,538.8 / 2,091.4
RTX 5090 Laptop — LightBeam17.3/32.012.629/14.7853,204.6/6,021.1
RTX 5090 Laptop — MAS-BIT12.6/31.08.662/8.6481,210.2/1,690.6
RTX 4090 Desktop — LightBeam17.4/32.012.648/14.8871,483.5/2,332.7
RTX 4090 Desktop — MAS-BIT12.8/31.28.695/8.695550.5/775.1

MAS-BIT achieves a 1.8× speedup over the K=100K=100 cascade (2.95× lower GPU memory; 25.4×/10.8× lower RAM) and a 2.3×–3.6× speedup over LightBeam across all GPUs, averaging 0.46/0.53 s per utterance on RTX 4090.

(2) End-to-end optimization across passes. Prefix replay builds differentiable token mixtures (mˉtr=∑v∈Ktrat,vrEv\bar{m}_t^r = \sum_{v \in \mathcal{K}_t^r} a_{t,v}^r E_v) so the arbiter loss updates expert adapters, interfaces, and encoders. Training Detached latent messages (keeping identical multi-pass inference and top-16 soft embeddings while detaching replay logits) raises holdout WER to 7.349% (+0.42 pp) on T12 and 3.666%/4.315% (+0.06/+0.42 pp) on T15 (Section 5.1, Table 16).

“Hallucination and error propagation from the experts to arbiter could be problematic, when both experts are wrong the arbiter alone cannot recover without an external LM.”

Response: Thank you for raising this important concern. Rather than relying only on discrete drafts, the arbiter prefix ZA=[ZN;MN;MP]Z_A = [Z_N; M_N; M_P] integrates direct neural features ZNZ_N, top-16 token alternatives in (MN,MP)(M_N, M_P), and the 1.7B LLM's priors (trained with 15% message dropout). Across all 880 T12 (5,496 words) and 1,426 T15 (9,445 words) validation utterances in FP32 (Section 5.2, Appendix B, Tables 4–7 & 16):

  • (1) Error reduction and preservation of correct expert outputs: Arbitration cuts word edits from 778/801 (neural/phonetic) to 557 on T12 (-28.4%/-30.5%) and from 627/703 to 420 on T15 (-33.0%/-40.3%), preserving 392/394 (99.5%) T12 and 973/973 (100.0%) T15 trials where both experts are correct.
  • (2) Exact recovery when both experts err: Among 305 T12 and 262 T15 both-wrong trials, the arbiter recovers 31 T12 and 49 T15 exact sentences without an external LM; 26/31 and 38/49 lie outside all minimum-edit draft alignments, including 26 and 36 with a correct word absent from both drafts (Figure 3, Appendix B.1).
  • (3) Causal dependence on ZNZ_N: Zeroing ZNZ_N at fixed messages eliminates all 26/26 and 38/38 outside-graph recoveries, and swapping ZNZ_N with a length-matched same-participant trial eliminates all 25/25 and 36/36 paired recoveries (Appendix B.2). Consistently, a retrained Message-only arbiter (omitting ZNZ_N) raises holdout WER to 8.445% (+1.52 pp) and 4.939%/5.492% (+1.33/+1.60 pp) (Table 16).

[1] Yizi Zhang, Linyang He, Chaofei Fan, Tingkai Liu, Han Yu, Trung Le, Jingyuan Li, Scott Linderman, Lea Duncker, Francis Willett, Nima Mesgarani, and Liam Paninski. A cross-species neural foundation model for end-to-end speech decoding. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (eds.), International Conference on Learning Representations, volume 2026, pp. 30773–30797, 2026. URL ICLR 2026 paper.

[2] Ebrahim Feghhi, Junlin Hu, Nima Hadidi, and Jonathan C. Kao. Lightbeam: An accurate and memory-efficient ctc decoder for speech neuroprostheses, 2026. URL arXiv:2603.14002.

[7] Zehui Feng, Weichuan Wang, Xiaohan Chen, Cuntai Guan, and Ting Han. Do speech BCIs need larger models? Rethinking neural decoding beyond scaling, 2026. URL arXiv:2610.23175.

Reviewer_8uYfOpenReview ↗

Response to rebuttal

Thanks for the detailed responses. My original issues were addressed well enough, however, I am a little skeptical of the unified training. It does seem that the performance is significantly better than the participant specific model, however, it is a significant deviation from the original paper version, and I do not support such significant changes in the rebuttal phase as it would require another full round of reviewing.

We sincerely thank the reviewer for following up, and we are very glad to hear that our responses addressed your original concerns. Regarding the unified training, we completely understand your concern and would like to respectfully clarify that this is only an encoder-level detail adjustment that does not change the major claims or core contributions of our MAS system: as shown in Figure 1 of the original submission, our core multi-agent framework—the neural expert, phonetic expert, and signal-grounded arbiter sharing one frozen LLM backbone with lightweight LoRA adapters and differentiable latent-message replay—remains unchanged, and we only unify the encoder frontend by combining the two BIT-initialized encoders through participant-specific input projections and shared neural and phonetic interfaces (paired with a streamlined staged-to-joint training schedule) so T12 and T15 share one model instead of two separate copies. The manuscript diff appears extensive mainly because, to maintain strict apples-to-apples consistency, we reran all original ablation controls and auxiliary analyses under this unified setup (alongside the new reviewer-requested controls and additional controls to further solidify the paper) and added a matched separate-participant training control to directly isolate and verify the benefit of cross-participant training. We hope this clarifies why the text updates look broad while our central MAS design, major claims, and contributions remain unchanged, and we sincerely appreciate your time and thoughtful feedback.

Comments on the paper

2 comments

We sincerely thank all three reviewers (8uYf, KiVB, and ACnc) for their thoughtful and constructive reviews. We are glad the reviewers recognize the value of coordinating complementary neural and phonetic experts through differentiable latent communication on a shared language-model backbone. In addition, we believe that the insightful questions and control experiments suggested across the reviews have substantially strengthened our paper.

In the revised manuscript, we unify MAS-BIT from separate participant-specific models into one cross-participant framework jointly trained across participants T12 and T15 via participant-specific input projections, two BIT-initialized Transformer encoders, shared interfaces, and a shared frozen Qwen3-ASR-1.7B backbone with three lightweight rank-8 LoRA adapters (Sections 3 & 5, Table 1, Figure 4). We also added a concurrent October 2026 (post-ICLR-submission) baseline, BEST (no external LM) [7] (12.61% T12 holdout WER; 5.17% T15 holdout WER, public/private subset unspecified; Table 1)—even with this latest work included, MAS-BIT remains the state-of-the-art end-to-end speech BCI decoder:

Method (Official Holdout Corpus WER %, lower is better)System SetupT12 HoldoutT15 PublicT15 Private
BIT End-to-End [1]1 E2E model15.6711.06—
BIT End-to-End + Ensemble [1]10 / 29 models + GPT10.227.76—
Published BIT Cascade, K=100K=100 [1]WFST + 5-gram + OPT-6.7B6.354.06—
MAS-BIT (Original submission, separate models)1 E2E model / participant8.4184.8785.288
MAS-BIT — Separate participant training (Revised ablation)1 E2E model / participant7.9714.1214.629
MAS-BIT (Revised unified cross-participant model)1 shared E2E model (greedy)6.9293.6063.892
  • New state-of-the-art end-to-end decoding: One jointly trained MAS-BIT model lowers holdout WER by 17.7%, 26.1%, and 26.4% relative from our original submission, outperforms matched Separate participant training by 1.042 pp (13.1% relative) on T12 and 0.515 / 0.737 pp (12.5% / 15.9% relative) on T15 while serving both participants with a single model, cuts WER by 32.2% and 53.5% relative versus BIT's 10-/29-model ensembles, and surpasses BIT's published K=100K=100 cascade (4.06%) on T15 public by 0.454 pp (11.2% relative) using greedy decoding without any WFST, lexicon, or external LM reranker.
  • Specialization vs. matched capacity and decoding passes (KiVB Q1, ACnc Q1; Section 5.1, Table 16): Under the same frontend, initialization, and 65-epoch schedule, MAS-BIT outperforms single-agent controls with a Rank-24 adapter (10.231% T12; 5.787% / 6.889% T15) or a 4B Borealis-5B audio-LLM decoder [3]—chosen because Qwen3-ASR [6] publicly releases weights only up to 1.7B, with no ~5B (3×1.7B3\times 1.7\text{B}) variant—(12.478%; 8.515% / 8.553%), as well as Sequential dual-view refinement matched in 3×rank-8 adapters, intermediate supervision, and 3 generation + 2 replay passes (7.159% T12; 3.939% / 4.739% T15).
  • Disentangling token alternatives, message gradients, and neural grounding (KiVB W2, ACnc Q2, 8uYf W1–W2; Section 5.1, Appendix B, Tables 4–7 & 16): New Detached latent messages (7.349%; 3.666% / 4.315%), Message-only arbiter (8.445%; 4.939% / 5.492%), fixed-weight Top-kk (-41 / -29 edits on T12/T15), and ZNZ_N intervention analyses isolate each mechanism.
  • Fine-tuned-encoder cascade & multi-GPU LightBeam efficiency (KiVB Q2, ACnc Q3, 8uYf W1; Section 5.4, Appendix C, Tables 10–12 & 15): MAS-BIT outperforms the cascade even when using MAS-BIT's fine-tuned encoder (7.295%; 4.696% / 4.629% at K=100K=100), and runs 2.3×–3.6× faster than BIT + LightBeam [2] with lower host RAM and GPU memory across A40, RTX 5090 Laptop, and RTX 4090 Desktop GPUs.

We address each reviewer comment point by point below. Please let us know if we have not fully addressed any of your questions.

[1] Yizi Zhang, Linyang He, Chaofei Fan, Tingkai Liu, Han Yu, Trung Le, Jingyuan Li, Scott Linderman, Lea Duncker, Francis Willett, Nima Mesgarani, and Liam Paninski. A cross-species neural foundation model for end-to-end speech decoding. In C. Vondrick, B. Hariharan, C. Raffel, L. Pinto, D. Yang, and A. Faust (eds.), International Conference on Learning Representations, volume 2026, pp. 30773–30797, 2026. URL ICLR 2026 paper.

[2] Ebrahim Feghhi, Junlin Hu, Nima Hadidi, and Jonathan C. Kao. Lightbeam: An accurate and memory-efficient ctc decoder for speech neuroprostheses, 2026. URL arXiv:2603.14002.

[3] VikhrModels. Borealis: Audio-language model for speech understanding, 2025. URL Borealis-5b-it model card.

[6] Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, and Junyang Lin. Qwen3-ASR technical report, 2026. URL arXiv:2601.21337.

[7] Zehui Feng, Weichuan Wang, Xiaohan Chen, Cuntai Guan, and Ting Han. Do speech BCIs need larger models? Rethinking neural decoding beyond scaling, 2026. URL arXiv:2610.23175.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.