- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Author of LENS propose a change in the training framework for invasive, intracortical BCI for brain-to-text decoding. They kept the two stage approach (neural data -> phoneme decoding model trained with CTC + WFST model for phoneme2text mapping), but they combine 3 different level of supervision for the encoder: covariance aware phoneme similarity within CTC (the piece I liked the most), word-level alignment with FastText embeddings and also sentence level alignment using E5 embeddings. This requires two stage of pretraining, the first one to get a normal phoneme decoding model, from which soft alignments are computed, then finetuning the additional heads and finally fine-tuning the whole model with the 3 different losses. Everything happens at training time, while the inference procedure is left unchanged and comparable and compatible with other approches on the same benchmark. They tested it on Card et al and Willett et al BCI data released with Brain2Text '24 and '25 benchmarks. Trained separately on each benchmark they achieved SOTA WER in both.
Critical strengths
The results are impressive and the idea to improve the baseline decoder is sound. Some strengths of this work:
- The idea to enrich to supervision is relevant and a practical contribution.
- The formulation is clear and the advantage of informing the CTC based on covariance affinity (but removing the shared logit component) is cool
- Good controls, authors included multiple seeds, ablation of each component and CTC-only comparison
- Of course, the WER on public benchmark is SOTA, suggesting that underlying contributions are helping.
Critical weaknesses
During my read of the paper I had two feelings: one is excitment for the good engineering, while the other one was a lack of novelty from the scientific point of view. These are the weaknesses driving my evaluation:
-
The paper starts describing the lack of use of higher hierarchy supervision for the encoder and propose the word/sentence guidance. They do cite some of the recent work, but I think some of the important ones are not present. The most obvious absent there is Boccato et al, BrainWhisperer, which tackles the same problem (in a different way relying on encoder-decoder CTC + CrossEntropy and pretrained Whisper adaption, but still injecting word and sentence level supervision) on the same data. Zhang et al cross-species work is correctly cited (in this case this the information flows with a contrastive loss + freeze and connection with LLMs). Probably worth mentioning also d'Ascoli et al 2025 and Defossez et al 2023 which proposed similar kind of alignments (even though they focus mostly on MEG data).
-
The headline gains attribution seems incomplete. The historical baselines (TMT included) differ in the training configuration and the ensemble results also depend on LLM trained separately different from the baselines one.
-
By looking at the ablation the improvement due to linguistic supervision seems to be quite minor (few percentage points in the WER). It's a bit unclear to me whether it's just compounding effect of the SOTA results mostly depend on other component?
-
The main novelty to me is the use of the similarity weighted CTC via the covariance matrix. I wasn't able to correctly get the intuition of whether this is a re-implementation or re-thinking of the Zhou et al 2021 word (Towards Accurate Phonetic Error Detection Through Phoneme Similarity Modeling) or if there is some novelty element here.
Questions for authors
I appreciate the engineering effort and the strong reported performance. To address the concerns above, please clarify the methodological novelty and provide evidence separating the contribution of the proposed supervision from other system improvements. In particular:
-
Position the contribution relative to prior work. Please discuss BrainWhisperer (Boccato et al.) and relevant neural-to-linguistic alignment approaches, including d’Ascoli et al. (2025) and Défossez et al. (2023). Which specific capability or methodological contribution distinguishes LENS from existing approaches combining phoneme supervision with higher-level linguistic guidance?
-
Isolate the gains under matched experimental conditions. Please report a CTC-only baseline using the same encoder, training budget, checkpoint-selection procedure, and decoding configuration as LENS. For the ensemble results, could you compare baseline and LENS checkpoints using the same ensemble size and language-model refinement procedure? This would help establish how much of the headline improvement is attributable to the proposed supervision.
-
Quantify the incremental contribution of linguistic supervision. Please report paired performance differences and confidence intervals between full LENS and the CAPS→CTC control. Does PMLA provide comparable benefits when initialized from standard CTC? This would clarify both the practical magnitude of its contribution and whether it specifically benefits from combination with CAPS.
-
Clarify and validate the novelty of covariance-aware CTC. Please explain the mathematical and conceptual differences between CAPS and the similarity-weighted CTC formulation of Zhou et al that you cite. In particular, what does incorporating representation covariance add beyond the existing emission-weighting framework? A direct paired comparison against classifier-only similarity and a relevant prior similarity kernel, under matched tuning conditions, would strengthen this claim.
7 comments
RE: W1, Q1
RE: W1, Q1 — Positioning LENS relative to prior linguistic supervision
Thank you for highlighting these studies and asking us to clarify LENS's contribution. Our specific contribution is posterior-guided word pooling for joint word- and sentence-level representation supervision within an attempted-speech phoneme decoder, where no word timings are available to anchor word-level targets.
We have expanded the Introduction to discuss BrainWhisperer, d’Ascoli et al. (2025), and Défossez et al. (2023), alongside BIT and Brain2Qwerty v2. We also added BrainWhisperer's published result to Section 3.1 and Table 1.
1. The mechanism of linguistic supervision.
BrainWhisperer combines phoneme CTC with autoregressive text-token prediction through an adapted Whisper encoder–decoder. LENS introduces explicit representation-level supervision: PMLA constructs soft word-level neural representations from CTC posteriors and aligns them with pretrained word embeddings, alongside utterance-level sentence alignment. The distinction concerns how linguistic representations are constructed and supervised, rather than the general use of higher-level guidance.
| Prior work | Linguistic supervision | Distinction of LENS |
|---|---|---|
| BrainWhisperer | Phoneme CTC and autoregressive text-token prediction through adapted Whisper. | Explicit word- and sentence-embedding objectives within phoneme-decoder training. |
| BIT | Utterance-level contrastive matching and integration with an audio-language model; also evaluates cascaded decoding. | Localized word matching alongside sentence alignment within the phoneme encoder. |
| d’Ascoli et al. (2025) | Word-embedding alignment using known word onsets in perception tasks. | Word correspondence estimated from phoneme CTC posteriors without annotated timings. |
| Défossez et al. (2023) | Alignment with pretrained speech features of synchronized audio for perceived-speech retrieval. | Transcript-derived linguistic targets for attempted-speech transcription. |
| Brain2Qwerty v2 | CTC-predicted spaces segment MEG typing features; DTW matches segments to words for embedding alignment. Neural tokens condition an LLM at inference. | Soft occupancies assigned directly to reference-word occurrences, agreement-based weighting, and training-only linguistic projection heads. |
2. Obtaining word-level correspondence in attempted speech.
The perception studies above obtain temporal anchors from the stimulus. In the self-paced attempted-speech datasets considered here, transcripts specify the word sequence but provide no annotated word timings. PMLA derives correspondence through reference-conditioned CTC forward–backward, retaining uncertainty over valid alignments and avoiding a separate segment-to-word matching step.
Word occupancies are precomputed from a frozen phoneme decoder and remain fixed during fine-tuning. Agreement with a separate target-conditioned alignment model weights each word's supervision. Pooled word features are aligned with FastText embeddings and an utterance representation with an E5 embedding, alongside phoneme CTC. The projection heads are discarded after training. Together with CAPS, these objectives connect phoneme-, word-, and sentence-level supervision while retaining the established phoneme-to-word decoding pipeline.
3. Controlled evidence supporting this contribution.
-
Linguistic information in the encoder. Appendix F.1, Table 8 evaluates raw features at the layer supervised by the word objective using common source alignments and a cross-recording-day prototype readout, without the learned projection head or downstream LM. Relative to CAPS→CTC, PMLA improves word-identification accuracy by 0.68 and 0.43 percentage points on B2T ’24 and ’25, with paired-bootstrap 95% intervals [0.37, 1.03] and [0.21, 0.67]. Improvements occur in all ten seeds on both datasets.
-
Soft temporal pooling. Over five matched seeds with agreement weights unchanged, soft source pooling gives the lowest mean validation WER: 9.01% / 2.85% on B2T ’24 / ’25, versus 9.16% / 2.89% for hard Viterbi and 9.10% / 2.91% for oracle pooling (Appendix E.1, Table 5). These comparisons evaluate pooling within PMLA.
-
Sentence-target correspondence. With CTC and word supervision retained, corresponding E5 targets yield lower mean WER than shuffled targets: 8.92% versus 9.05% on ’24 and 2.93% versus 3.03% on ’25 over ten matched seeds. They also outperform the tested random bag-of-words targets (Appendix F.2, Table 9).
Finally, PMLA improves over matched CTC-only continuation from both standard-CTC and CAPS initializations on both datasets (Appendix D.1, Table 3; ten matched seeds; one-tailed paired tests ). We discuss gain magnitude and attribution further in W2/Q2 and W3/Q3, and the covariance-aware contribution of CAPS in W4/Q4.
RE: W2, Q2
RE: W2, Q2 — Matched training and ensemble comparisons
We thank the reviewer for this helpful suggestion. We agree that matched comparisons are important for distinguishing the contribution of supervision from changes in training and decoding. The revision now includes these comparisons at both the single-model and ensemble levels.
1. Matched single-model controls.
We compare standard CTC and CAPS initial training, each followed by either CTC-only continuation or PMLA. These conditions use the same encoder backbone and data pipeline, 600 initial-training epochs followed by 600 main fine-tuning epochs, checkpoint selection by validation phoneme error rate, and the same WFST + OPT decoding configuration. These budgets cover the main decoder-training stages; PMLA additionally includes offline supervision preparation and projection-head warm-up.
The full validation factorial is reported in Appendix D.1, Table 3. We also added matched test-set comparisons over the same ten seeds (Appendix D.2, Table 4).
Single-model test WER (%), mean ± s.e.m.:
| Training schedule | B2T ’24 | ’25 Public | ’25 Private |
|---|---|---|---|
| CTC→CTC | 6.15 ± 0.07 | 3.20 ± 0.09 | 3.21 ± 0.05 |
| CAPS→CTC | 6.02 ± 0.06 | 3.26 ± 0.10 | 3.15 ± 0.05 |
| CAPS→PMLA (LENS) | 5.94 ± 0.06 | 3.21 ± 0.10 | 3.10 ± 0.05 |
Under the extended fine-tuning paradigm, LENS has lower mean WER than CAPS→CTC on all three test splits, which reflects the improvement by PMLA. LENS also achieves the lowest mean on ’24 and the ’25 private set. These comparisons account for the gains available through continued CTC training. We report the paired differences and their uncertainty in W3/Q3.
2. Matched ensemble comparison.
Appendix G.1, Table 13 compares two pools of 20 checkpoints: ten initial CTC models with their ten CTC-only descendants, versus ten CAPS models with their ten PMLA descendants. Both use the same Qwen refinement procedure described in Appendix G.
Test WER (%) after Qwen refinement; ensemble point estimates:
| Checkpoint pool | B2T ’24 | ’25 Public | ’25 Private |
|---|---|---|---|
| CTC + CTC→CTC | 4.25 | 1.70 | 1.73 |
| CAPS + CAPS→PMLA (LENS) | 4.02 | 1.55 | 1.43 |
With checkpoint count and refinement procedure held fixed, LENS reduces WER by 0.230, 0.146 and 0.296 percentage points, respectively. These correspond to relative word-error reductions of approximately 5.4%, 8.6% and 17.1%.
The advantage therefore remains after both checkpoint pools receive the same refinement procedure. Together, the matched single-model and ensemble results support a contribution from the proposed supervision within the complete decoding system. The factorial comparison separately identifies the contributions of CAPS and PMLA, while the ensemble comparison evaluates their combined benefit.
General Response
We thank the reviewer for the careful reading and constructive suggestions, which helped us clarify LENS’s methodological contribution and better isolate the gains from the proposed supervision. We have expanded the related-work discussion, added matched controls and analyses, and extended the existing B2T ’25 ablations from five to ten matched seeds. Changes in the manuscript are marked in blue: