acceptodds
Under review as a conference paper at ICLR 2027

LENS: Linking Phoneme, Word, and Sentence Supervision for Brain-to-Text Decoding

Abstract

Brain–computer interfaces offer a path to restoring communication by decoding intended speech from neural activity. High-performing brain-to-text systems combine connectionist temporal classification (CTC)-based phoneme decoding with language models, yet standard CTC neither explicitly encodes phoneme relations nor directly supervises word- and sentence-level neural representations. Pretrained linguistic representations could provide that direct supervision, but connecting words to neural features is challenging without annotated word timings. We propose **LENS**, a framework that **l**inks phon**e**me-, word-, and se**n**tence-level **s**upervision for brain-to-text decoding. LENS combines classifier geometry with neural representation covariance to assign bounded credit to phoneme alternatives during CTC supervision. Building on phoneme-level training, we introduce Posterior-Guided Multiscale Linguistic Alignment: model-derived soft alignment couples neural features with pretrained word embeddings without annotated word timings, complemented by utterance-level alignment with pretrained sentence representations. LENS achieves state-of-the-art word error rates on the Brain-to-Text ’24 and ’25 benchmarks without changing the established decoding pipeline. Together, these findings support combining phoneme supervision informed by neural evidence and phoneme relations with word- and sentence-level linguistic guidance for more accurate brain-to-text decoding.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain-computer interfaces, intracortical speech decoding, connectionist temporal classification, phoneme similarity, multiscale supervision, contrastive learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bpN1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Author of LENS propose a change in the training framework for invasive, intracortical BCI for brain-to-text decoding. They kept the two stage approach (neural data -> phoneme decoding model trained with CTC + WFST model for phoneme2text mapping), but they combine 3 different level of supervision for the encoder: covariance aware phoneme similarity within CTC (the piece I liked the most), word-level alignment with FastText embeddings and also sentence level alignment using E5 embeddings. This requires two stage of pretraining, the first one to get a normal phoneme decoding model, from which soft alignments are computed, then finetuning the additional heads and finally fine-tuning the whole model with the 3 different losses. Everything happens at training time, while the inference procedure is left unchanged and comparable and compatible with other approches on the same benchmark. They tested it on Card et al and Willett et al BCI data released with Brain2Text '24 and '25 benchmarks. Trained separately on each benchmark they achieved SOTA WER in both.

Critical strengths

The results are impressive and the idea to improve the baseline decoder is sound. Some strengths of this work:

  1. The idea to enrich to supervision is relevant and a practical contribution.
  2. The formulation is clear and the advantage of informing the CTC based on covariance affinity (but removing the shared logit component) is cool
  3. Good controls, authors included multiple seeds, ablation of each component and CTC-only comparison
  4. Of course, the WER on public benchmark is SOTA, suggesting that underlying contributions are helping.
Critical weaknesses

During my read of the paper I had two feelings: one is excitment for the good engineering, while the other one was a lack of novelty from the scientific point of view. These are the weaknesses driving my evaluation:

  1. The paper starts describing the lack of use of higher hierarchy supervision for the encoder and propose the word/sentence guidance. They do cite some of the recent work, but I think some of the important ones are not present. The most obvious absent there is Boccato et al, BrainWhisperer, which tackles the same problem (in a different way relying on encoder-decoder CTC + CrossEntropy and pretrained Whisper adaption, but still injecting word and sentence level supervision) on the same data. Zhang et al cross-species work is correctly cited (in this case this the information flows with a contrastive loss + freeze and connection with LLMs). Probably worth mentioning also d'Ascoli et al 2025 and Defossez et al 2023 which proposed similar kind of alignments (even though they focus mostly on MEG data).

  2. The headline gains attribution seems incomplete. The historical baselines (TMT included) differ in the training configuration and the ensemble results also depend on LLM trained separately different from the baselines one.

  3. By looking at the ablation the improvement due to linguistic supervision seems to be quite minor (few percentage points in the WER). It's a bit unclear to me whether it's just compounding effect of the SOTA results mostly depend on other component?

  4. The main novelty to me is the use of the similarity weighted CTC via the covariance matrix. I wasn't able to correctly get the intuition of whether this is a re-implementation or re-thinking of the Zhou et al 2021 word (Towards Accurate Phonetic Error Detection Through Phoneme Similarity Modeling) or if there is some novelty element here.

Questions for authors

I appreciate the engineering effort and the strong reported performance. To address the concerns above, please clarify the methodological novelty and provide evidence separating the contribution of the proposed supervision from other system improvements. In particular:

  1. Position the contribution relative to prior work. Please discuss BrainWhisperer (Boccato et al.) and relevant neural-to-linguistic alignment approaches, including d’Ascoli et al. (2025) and Défossez et al. (2023). Which specific capability or methodological contribution distinguishes LENS from existing approaches combining phoneme supervision with higher-level linguistic guidance?

  2. Isolate the gains under matched experimental conditions. Please report a CTC-only baseline using the same encoder, training budget, checkpoint-selection procedure, and decoding configuration as LENS. For the ensemble results, could you compare baseline and LENS checkpoints using the same ensemble size and language-model refinement procedure? This would help establish how much of the headline improvement is attributable to the proposed supervision.

  3. Quantify the incremental contribution of linguistic supervision. Please report paired performance differences and confidence intervals between full LENS and the CAPS→CTC control. Does PMLA provide comparable benefits when initialized from standard CTC? This would clarify both the practical magnitude of its contribution and whether it specifically benefits from combination with CAPS.

  4. Clarify and validate the novelty of covariance-aware CTC. Please explain the mathematical and conceptual differences between CAPS and the similarity-weighted CTC formulation of Zhou et al that you cite. In particular, what does incorporating representation covariance add beyond the existing emission-weighting framework? A direct paired comparison against classifier-only similarity and a relevant prior similarity kernel, under matched tuning conditions, would strengthen this claim.

7 comments
AuthorsOpenReview ↗

General Response

We thank the reviewer for the careful reading and constructive suggestions, which helped us clarify LENS’s methodological contribution and better isolate the gains from the proposed supervision. We have expanded the related-work discussion, added matched controls and analyses, and extended the existing B2T ’25 ablations from five to ten matched seeds. Changes in the manuscript are marked in blue:

  • W1, Q1: expanded positioning relative to BrainWhisperer and neural-to-linguistic alignment approaches (Introduction), and inclusion of BrainWhisperer in the benchmark comparison (Section 3.1, Table 1).
  • W2, Q2: added matched single-model test comparisons (Appendix D.2, Table 4) and a matched ensemble comparison with the same checkpoint count and language-model refinement procedure (Appendix G.1, Table 13).
  • W3, Q3: added CTC→PMLA to complete the factorial comparison (Appendix D.1, Table 3), with paired improvements and confidence intervals reported below; added analyses of word identity in neural representations and sentence-target correspondence (Appendix F.1–F.2, Tables 8–9).
  • W4, Q4: clarified the mathematical role of covariance and the distinction from prior similarity-weighted CTC (Section 2.3, Appendix A); added an articulatory-kernel control and comparisons with CTC-aligned neural similarity (Appendix H.4–H.5, Tables 18–19), complementing the existing classifier-only ablation (Table 2).
AuthorsOpenReview ↗

RE: W1, Q1

RE: W1, Q1 — Positioning LENS relative to prior linguistic supervision

Thank you for highlighting these studies and asking us to clarify LENS's contribution. Our specific contribution is posterior-guided word pooling for joint word- and sentence-level representation supervision within an attempted-speech phoneme decoder, where no word timings are available to anchor word-level targets.

We have expanded the Introduction to discuss BrainWhisperer, d’Ascoli et al. (2025), and Défossez et al. (2023), alongside BIT and Brain2Qwerty v2. We also added BrainWhisperer's published result to Section 3.1 and Table 1.

1. The mechanism of linguistic supervision.

BrainWhisperer combines phoneme CTC with autoregressive text-token prediction through an adapted Whisper encoder–decoder. LENS introduces explicit representation-level supervision: PMLA constructs soft word-level neural representations from CTC posteriors and aligns them with pretrained word embeddings, alongside utterance-level sentence alignment. The distinction concerns how linguistic representations are constructed and supervised, rather than the general use of higher-level guidance.

Prior workLinguistic supervisionDistinction of LENS
BrainWhispererPhoneme CTC and autoregressive text-token prediction through adapted Whisper.Explicit word- and sentence-embedding objectives within phoneme-decoder training.
BITUtterance-level contrastive matching and integration with an audio-language model; also evaluates cascaded decoding.Localized word matching alongside sentence alignment within the phoneme encoder.
d’Ascoli et al. (2025)Word-embedding alignment using known word onsets in perception tasks.Word correspondence estimated from phoneme CTC posteriors without annotated timings.
Défossez et al. (2023)Alignment with pretrained speech features of synchronized audio for perceived-speech retrieval.Transcript-derived linguistic targets for attempted-speech transcription.
Brain2Qwerty v2CTC-predicted spaces segment MEG typing features; DTW matches segments to words for embedding alignment. Neural tokens condition an LLM at inference.Soft occupancies assigned directly to reference-word occurrences, agreement-based weighting, and training-only linguistic projection heads.

2. Obtaining word-level correspondence in attempted speech.

The perception studies above obtain temporal anchors from the stimulus. In the self-paced attempted-speech datasets considered here, transcripts specify the word sequence but provide no annotated word timings. PMLA derives correspondence through reference-conditioned CTC forward–backward, retaining uncertainty over valid alignments and avoiding a separate segment-to-word matching step.

Word occupancies are precomputed from a frozen phoneme decoder and remain fixed during fine-tuning. Agreement with a separate target-conditioned alignment model weights each word's supervision. Pooled word features are aligned with FastText embeddings and an utterance representation with an E5 embedding, alongside phoneme CTC. The projection heads are discarded after training. Together with CAPS, these objectives connect phoneme-, word-, and sentence-level supervision while retaining the established phoneme-to-word decoding pipeline.

3. Controlled evidence supporting this contribution.

  • Linguistic information in the encoder. Appendix F.1, Table 8 evaluates raw features at the layer supervised by the word objective using common source alignments and a cross-recording-day prototype readout, without the learned projection head or downstream LM. Relative to CAPS→CTC, PMLA improves word-identification accuracy by 0.68 and 0.43 percentage points on B2T ’24 and ’25, with paired-bootstrap 95% intervals [0.37, 1.03] and [0.21, 0.67]. Improvements occur in all ten seeds on both datasets.

  • Soft temporal pooling. Over five matched seeds with agreement weights unchanged, soft source pooling gives the lowest mean validation WER: 9.01% / 2.85% on B2T ’24 / ’25, versus 9.16% / 2.89% for hard Viterbi and 9.10% / 2.91% for oracle pooling (Appendix E.1, Table 5). These comparisons evaluate pooling within PMLA.

  • Sentence-target correspondence. With CTC and word supervision retained, corresponding E5 targets yield lower mean WER than shuffled targets: 8.92% versus 9.05% on ’24 and 2.93% versus 3.03% on ’25 over ten matched seeds. They also outperform the tested random bag-of-words targets (Appendix F.2, Table 9).

Finally, PMLA improves over matched CTC-only continuation from both standard-CTC and CAPS initializations on both datasets (Appendix D.1, Table 3; ten matched seeds; one-tailed paired tests p<0.05p<0.05). We discuss gain magnitude and attribution further in W2/Q2 and W3/Q3, and the covariance-aware contribution of CAPS in W4/Q4.

AuthorsOpenReview ↗

RE: W2, Q2

RE: W2, Q2 — Matched training and ensemble comparisons

We thank the reviewer for this helpful suggestion. We agree that matched comparisons are important for distinguishing the contribution of supervision from changes in training and decoding. The revision now includes these comparisons at both the single-model and ensemble levels.

1. Matched single-model controls.

We compare standard CTC and CAPS initial training, each followed by either CTC-only continuation or PMLA. These conditions use the same encoder backbone and data pipeline, 600 initial-training epochs followed by 600 main fine-tuning epochs, checkpoint selection by validation phoneme error rate, and the same WFST + OPT decoding configuration. These budgets cover the main decoder-training stages; PMLA additionally includes offline supervision preparation and projection-head warm-up.

The full validation factorial is reported in Appendix D.1, Table 3. We also added matched test-set comparisons over the same ten seeds (Appendix D.2, Table 4).

Single-model test WER (%), mean ± s.e.m.:

Training scheduleB2T ’24’25 Public’25 Private
CTC→CTC6.15 ± 0.073.20 ± 0.093.21 ± 0.05
CAPS→CTC6.02 ± 0.063.26 ± 0.103.15 ± 0.05
CAPS→PMLA (LENS)5.94 ± 0.063.21 ± 0.103.10 ± 0.05

Under the extended fine-tuning paradigm, LENS has lower mean WER than CAPS→CTC on all three test splits, which reflects the improvement by PMLA. LENS also achieves the lowest mean on ’24 and the ’25 private set. These comparisons account for the gains available through continued CTC training. We report the paired differences and their uncertainty in W3/Q3.

2. Matched ensemble comparison.

Appendix G.1, Table 13 compares two pools of 20 checkpoints: ten initial CTC models with their ten CTC-only descendants, versus ten CAPS models with their ten PMLA descendants. Both use the same Qwen refinement procedure described in Appendix G.

Test WER (%) after Qwen refinement; ensemble point estimates:

Checkpoint poolB2T ’24’25 Public’25 Private
CTC + CTC→CTC4.251.701.73
CAPS + CAPS→PMLA (LENS)4.021.551.43

With checkpoint count and refinement procedure held fixed, LENS reduces WER by 0.230, 0.146 and 0.296 percentage points, respectively. These correspond to relative word-error reductions of approximately 5.4%, 8.6% and 17.1%.

The advantage therefore remains after both checkpoint pools receive the same refinement procedure. Together, the matched single-model and ensemble results support a contribution from the proposed supervision within the complete decoding system. The factorial comparison separately identifies the contributions of CAPS and PMLA, while the ensemble comparison evaluates their combined benefit.

Reviewer_yCroOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The work proposes two additions to brain2text decoding. First, it develops CPAS, a way to incorporate phoneme relations into CTC by capturing correlations between phoneme classifier outputs from a pretrained decoder. Second, it adds word and sentence contrastive losses using pretrained language embeddings and estimated word alignments.

The method is implemented on top of TMT model for brain2text application and is evaluated on two intracortical speech benchmarks brain2text24 and brain2text25. Ablations show some improvements over standard CTC. Larger gains are achieved from model ensemble and language-model refinement. The reported test results are competitive with prior methods.

Critical strengths

S1 The recovery of phoneme relations to relax CTC supervision is interesting and makes sense as second order improvement over CTC. It appears before the decoder and thus does not require changing the decoding architecture.

S2 Evaluated on 2 datasets and compared with prior methods. The improvements are modest but fairly consistent. This suggests that CTC training can benfit from additional regularization.

Critical weaknesses

W1. The first proposed regularization, phoneme relation correlation (CAPS) assumes that correlations between phoneme classifier outputs reflect meaningful phoneme similarities. However, I am concrened that these correlations could capture specific dataset statistics rather than actual similarities in neural representations. Effectively, it overfits to the dataset correlations between phonemes.The paper does not sufficiently convince or explain what exactly does CAPS capture. Comparing affinity matrices across datasets could perhaps shed more light.

W2. The improvements from individual components are relatively small. In particular, CAPS provides almost no improvement on B2T25, and the benefit of covariance based similarity over simpler classifier weight similarity is limited. It is unclear whether the additional complexity is justified by the gains.

W3. The motivation for sentence-level supervision is not clear. Semantic embeddings may not preserve distinctions important for actual transcription. The paper does not investigate whether this supervision improves phoneme decoding through better neural representations or mainly introduces linguistic biases. The contribution of this component also appears small.

Questions for authors

Q1. The authors claim that the proposed components can be applied to different decoding architectures, but evaluate them only with TMT. Why was TMT chosen? Is there evidence that the improvements would hold with other architectures?

Q2. What is the stability of phoneme similarity matrices learned by CAPS? across training subsets and participants.

Q3. How CAPS compares with phoneme similarities estimated directly from CTC aligned neural representations?

7 comments
AuthorsOpenReview ↗

General Response and RE: W1, Q2, Q3 (1/2)

We thank the reviewer for the careful reading and the concrete suggestions, several of which directly shaped this revision. New material is in blue:

  • W1, Q2, Q3: Section 4, "What Does the CAPS Affinity Capture?" (Fig. 3); Appendix A, "What CC Depends On" (Eq. 16); Appendix H: stability across seeds, recording days, covariance estimates and participants (H.3, H.6), transcript and articulatory controls (H.4), CTC-aligned neural similarity (H.5), cross-participant transfer (H.7), unseen recording days (H.8).
  • W2: ten matched seeds on both benchmarks (Section 3.1, Table 2); factorial ablation and matched test-set comparisons (Appendix D, Tables 3–4); matched ensemble comparison (Appendix G.1, Table 13).
  • W3: sentence-target controls (F.2, Table 9), word identity in neural representations (F.1, Table 8), and utterances with unseen vocabulary (F.3, Table 10).
  • Q1: why the TMT, and CAPS and PMLA with a GRU encoder (Appendix I, Tables 23–26, Fig. 10).

RE: W1, Q2, Q3 (1/2) — What CAPS captures, and whether it reflects neural similarity or dataset statistics

Short answer. CC measures how similar the neural representations of two phonemes are, after normalizing for noise. Five results support this:

  • CC agrees closely with phoneme similarity estimated directly from CTC-aligned neural input features.
  • It is reproducible across seeds, disjoint recording days, covariance estimates and two participants.
  • It is not explained by phoneme frequency or co-occurrence.
  • A kernel estimated on one participant gives the same WER on the other.
  • Its WER benefit holds on recording days not used to build it.

We agree this is the central question for CAPS, and the original submission did not answer it well. Below we take the reviewer's three requests in turn. This comment covers what CC measures and how it compares with neural similarity (Q3). The next covers stability (Q2), dataset statistics (W1) and whether the kernel overfits.

1. What CC measures (Appendix A). CC is the correlation, across training frames, of two phonemes' class-relative logit responses (Eq. 14). Equivalently, it is a covariance-weighted cosine of the centered classifier weights (Eq. 15). Appendix A derives what this means for the representations.

  • Assumptions: phoneme classes are Gaussian with a shared within-class covariance Σw\Sigma_w, and the CTC head is calibrated. A converged CTC head behaves this way, because its logit gradient is y−γy - \gamma: a stationary head is the logistic regression of its own alignment occupancies.
  • Result: the centered weights are w~c=Σw−1(μc−μˉ)\tilde w_c = \Sigma_w^{-1}(\mu_c - \bar\mu). CC is then a noise-normalized cosine between the phoneme-conditional mean representations (Eq. 16). This is the quantity that similarity from aligned representations estimates (Q3).

The derivation also predicts how dataset statistics can enter CC:

  • Phoneme priors log⁡πc\log \pi_c enter only the classifier biases, which CC removes.
  • Phoneme frequencies can reach CC only through the between-class part of Σ^\hat\Sigma. That route can be tested directly by reweighting Σ^\hat\Sigma (point 4).

For the same reason, rebalancing phoneme frequencies when training the teacher would change only its biases. It would therefore not change CC.

2. Comparison with similarity estimated directly from CTC-aligned representations (Q3; Section H.5, Table 19, Fig. 9). We compared CC with two similarities computed directly from aligned data. For both, a standard-CTC model trained on the other recording days forced-aligns the ground-truth phonemes of each held-out day.

  • NZN_Z, the CTC-aligned teacher representations. The class mean of each phoneme is taken over the teacher's pre-classifier representations at its aligned frames. Similarity is a whitened cosine between these means, computed across folds.
  • NXN_X, the neural input features themselves. Each phoneme's response is estimated by regressing the features on time-lagged phoneme indicators, and the same cross-fold whitened cosine is applied.

Table R1. Spearman correlation over 741 phoneme pairs; one-sided Mantel p<10−4p < 10^{-4} for every row.

CC compared with'24'25
NZN_Z: teacher representations, CTC-aligned0.9190.862
NXN_X: neural input features, hard (Viterbi) alignment0.8260.820
Articulatory similarity0.3060.292

CC closely tracks the similarity of the aligned teacher representations (NZN_Z), as Appendix A predicts. Because NZN_Z comes from the same teacher that defines CC, part of that agreement is expected. NXN_X is the stricter test. It agrees with CC at 0.83 and 0.82. Fig. 9 shows that NXN_X reproduces the vowel–consonant split and the manner-class blocks of CC. Articulatory similarity agrees much less, and it explains no variance in CC beyond NXN_X.

AuthorsOpenReview ↗

RE: W1, Q2, Q3 (2/2)

RE: W1, Q2, Q3 (2/2) — Stability, dataset statistics, and overfitting

Continuing from the previous comment, this part covers stability across training subsets and participants (Q2), dataset statistics (W1), and whether the kernel overfits its training data.

3. Stability across training subsets and participants (Q2; Section H.3, Tables 14–16, 20; Figs. 3, 6, 7). Following the reviewer's suggestion, we compared affinity matrices across training subsets, covariance estimates and participants. Each row below is paired with the value that means "no change" (two teachers that differ only in seed) or "chance" (the 95th percentile of a phoneme-label permutation null).

Table R2. Spearman correlation over 741 entries (CC) or 1,482 entries (GG), except the last row. For the participant rows, Mantel p≤5×10−4p \le 5\times10^{-4}.

ComparisonMeasureAgreementNo change / chance
Training subsets ('24)
Disjoint halves of the recording days (even vs. odd days, no shared session)GG0.989Different seeds: 0.994
Each half vs. full dataGG0.990Different seeds: 0.994
Covariance estimate ('24 / '25)
Session-balanced Σ^\hat\SigmaCC0.999 / 0.998Different seeds: 0.993 / 0.955
Phoneme-balanced Σ^\hat\SigmaCC0.990 / 0.958Different seeds: 0.993 / 0.955
Seeds and participants
Different seeds, '24 / '25CC0.993 / 0.955—
GG0.994 / 0.954—
'24 vs. '25 participant, ten-seed meansCC0.931Chance: 0.076
GG0.926Chance: 0.074
'24 vs. '25 participant, each phoneme's top-3 alternatives (Jaccard)GG0.48Chance: 0.05

Changing half of the recording days, or how frames are weighted in Σ^\hat\Sigma, changes the affinity about as much as changing the seed. Only changing the participant has a larger effect. Here the two benchmarks differ in participant, electrode count and encoder width, and agreement still remains far above chance. A teacher trained on half of the '24 days agrees with '25 as well as a full-data teacher does (0.900 vs. 0.903; Table 16). The shared structure therefore does not depend on which recording days are used.

The cross-participant agreement is not just the vowel–consonant split (Fig. 3a):

  • Restricted to vowel pairs, the seed means agree at 0.83; for consonant pairs, at 0.80.
  • These within-class pairs receive almost all CAPS credit: less than 0.01% ('24) and 0.5% ('25) of credit crosses the vowel–consonant boundary.

4. Dataset statistics (W1; Sections H.4 and H.6, Tables 17 and 20).

  • Frequency. CC is unrelated to phoneme-frequency similarity (ρ=0.011\rho = 0.011 and 0.017; p≥0.54p \ge 0.54). The derivation says frequency could enter only through Σ^\hat\Sigma. Reweighting Σ^\hat\Sigma so that every phoneme counts equally changes CC no more than a seed change does (0.990 / 0.958 vs. 0.993 / 0.955).
  • Co-occurrence. CC is negatively correlated with within-word adjacency PMI (ρ=−0.51\rho = -0.51 and −0.42-0.42). Phonemes that often occur next to each other, such as consonant–vowel pairs, are dissimilar in CC. A kernel that overfit co-occurrence would show the opposite sign.
  • Articulatory features. The relation is moderate (ρ=0.31\rho = 0.31 / 0.29, Fig. 8). Articulatory similarity explains no variance in CC beyond NXN_X.

5. Does the kernel overfit its training data? (Sections H.4, H.7, H.8; Fig. 3c; Tables 18, 21, 22) If CAPS exploited dataset-specific correlations, a kernel from another dataset should perform worse. It does not:

  • Cross-participant transfer. On '24, replacing the native kernel with one estimated entirely on the '25 participant gives the same WER: 9.32% vs. 9.34%, against 9.90% for CTC. On '25, both kernels reach 3.13%.
  • Unseen recording days. Models trained on half of the '24 days, with that half's kernel, still lower WER on validation trials from the other half: Δ=−0.59\Delta = -0.59, 95% CI [−1.15, −0.03] (Table 22).
  • Source of the relations. An articulatory kernel with the same credit budget reaches 9.70%, significantly worse than CAPS at 9.34% (Table 18). Across the articulatory, classifier-weight and CAPS kernels, WER falls as agreement with NXN_X rises (0.39, 0.81, 0.83). Relaxation alone does not explain the gain: spreading the same budget uniformly (UNIFORM) gives 9.66% on '24 and no gain on '25 (3.29%; Table 2).

Summary. CAPS captures a noise-normalized similarity of the phoneme-conditional neural representations. It matches similarity measured directly from aligned neural data, it is as stable across disjoint recording days and covariance estimates as across seeds, and it agrees far above chance across two participants. Frequency and co-occurrence do not explain it, and a kernel from the other participant works as well as the native one.

RE: W2 — Size of the gains, and regarding covariance

The revision reports ten matched seeds on both benchmarks (Section 3.1). With ten seeds, each component is significant on its own on both: CAPS against CTC (Table 2), and PMLA against the CTC-only continuation from both standard-CTC and CAPS initializations (Table 3; '24: 9.22%∗^* and 8.92%∗^*, '25: 3.05%∗^* and 2.93%∗^*).

We agree that each component's gain is modest in absolute terms. Below we explain why we think the gains are still meaningful and what they cost.

The gains are consistent, significant and add together.

  • Phoneme training. Over ten matched seeds, CAPS lowers validation WER from 9.90% to 9.34% on '24 and from 3.28% to 3.13% on '25. These are relative reductions of 5.7% and 4.6%, both p<0.05p < 0.05 (Table 2), on a '25 baseline that is already near 3% WER.
  • Longer training. The advantage persists after 600 further epochs of standard CTC: CAPS→CTC reaches 9.06% / 3.01%, vs. 9.37% / 3.09% for CTC→CTC.
  • Combination with PMLA. The factorial ablation (Appendix D.1, Table 3) shows that CAPS initialization helps under both fine-tuning objectives. The combination (LENS) is lowest: 8.92% / 2.93%.
  • Test sets. Single-model LENS reaches 5.94% ('24) and 3.21% / 3.10% ('25 public / private). These are 6.5%, 20.9% and 32.5% relative below the strongest listed single-model comparators (Table 1).
  • Ensembles. With the same checkpoint budget and LLM refinement, the LENS ensemble reaches 4.02% ('24) and 1.55% / 1.43% ('25 public / private), against 4.25% and 1.70% / 1.73% for a CTC-only ensemble. These are 5.4%, 8.6% and 17.1% relative reductions (Appendix G.1, Table 13). The gain from the proposed supervision therefore persists after ensembling.

Covariance vs. classifier-weight similarity. CLF is the classifier-only ablation of CAPS, not a separate simpler method. It already uses the proposed kernel, exact-phoneme anchor and credit budget, and it improves over CTC on '24. The CLF-vs.-CAPS comparison therefore isolates a single design choice: whether the affinity accounts for how the representations vary. The WER difference between CLF and CAPS is small (9.47% vs. 9.34% on '24; 3.17% vs. 3.13% on '25), and we do not claim that covariance is the main source of the gain. Its main role is the one W1 asks about: it ties the kernel to the neural representation.

  • Theory. Without the ridge term, CC is invariant to any invertible linear reparametrization of the hidden space. The weight cosine is invariant only to rotations, so it depends on an arbitrary basis. Under the model of Appendix A, CC becomes a noise-normalized similarity of class means.
  • Structure matters more than relaxation. Kernels rank by their agreement with NXN_X: articulatory 9.70%, CLF 9.47%, CAPS 9.34% (Table 18). Relaxation without structure does not help: UNIFORM gives 9.66% and 3.29%, with no gain on '25. On '25, only CAPS is significantly better than CTC.

Cost. CAPS adds very little to training and nothing at inference:

  • The teacher is the standard-CTC baseline, which is trained anyway. A kernel can even be reused from another participant (Section H.7).
  • One pass of the frozen teacher over a sample of training frames estimates a d×dd\times d covariance (d=384d = 384 / 512). A 41×4141\times41 kernel is then computed once.
  • During training, the only extra step is one 41×4141\times41 matrix product per frame.
  • The CTC topology, decoder and inference are unchanged, and the teacher is not used at inference.
  • λΣ\lambda_\Sigma and τph\tau_{\mathrm{ph}} are shared across benchmarks; only ρ\rho differs (Table 28). Compared with CLF, covariance adds only the Σ^\hat\Sigma estimate.

PMLA’s additional work is concentrated in training. an oracle is trained for each seed, and linguistic embeddings and alignments are computed offline. However, text embeddings are cached and shared across runs, while each seed’s alignments and oracle agreement weights are cached for reuse. Neither the oracle nor the text encoders is therefore recomputed during the subsequent 600-epoch fine-tuning stage. Both PMLA and the CTC-only controls perform this model fine-tuning, with PMLA adding pooling, small projection heads and auxiliary losses. The matched comparisons show that these additions improve validation WER beyond continued CTC training alone (Table 3). The oracle, text encoders and auxiliary heads are not used at inference without any extra computational cost.

Reviewer_du1TOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LENS, a framework that improves brain-to-text decoding by incorporating supervision at the phoneme, word, and sentence levels. The method consists of two main components:

  1. Covariance-Aware Phoneme Supervision (CAPS), which uses classifier geometry and neural representation covariance to construct a similarity-aware credit kernel for CTC training.

  2. Posterior-Guided Multiscale Linguistic Alignment (PMLA), which leverages CTC posterior probabilities to obtain soft word-level temporal alignments without annotated word timestamps, and aligns neural representations with pretrained FastText word embeddings and E5 sentence embeddings.

The framework retains the established CTC-based decoding pipeline at inference. Experiments on Brain-to-Text '24 and '25 demonstrate strong single-model performance, achieving WERs of 5.94% and 3.28%/3.11%, respectively. Additional ensemble experiments further improve performance.

Critical strengths
  1. Strong empirical performance. The method achieves state-of-the-art single-model results on both Brain-to-Text benchmarks. These improvements are meaningful for neural speech decoding, where accurate transcription remains challenging.

  2. A well-motivated approach to multiscale supervision. The paper addresses an important limitation of phoneme-level CTC training: its lack of explicit supervision for higher-level linguistic representations. The use of CTC posteriors to derive soft word alignments without annotated timestamps is a practical way to incorporate pretrained linguistic representations into neural decoding.

  3. Reasonably comprehensive ablation studies. The authors compare covariance-aware phoneme supervision with uniform relaxation and classifier-only similarity, and separately investigate word-level supervision, sentence-level supervision, and oracle-derived confidence weighting. I particularly appreciate the extended CTC-only fine-tuning control, which helps distinguish improvements due to the proposed objectives from those attributable to additional training.

  4. Compatibility with existing decoding pipelines. The proposed auxiliary supervision does not require replacing the established phoneme-to-word decoding procedure, making the approach relatively straightforward to integrate with existing systems.

Critical weaknesses
  1. The individual contributions of CAPS and PMLA are not fully isolated. Although the paper includes a CAPS→CTC fine-tuning control, it does not report the corresponding CTC→PMLA experiment. A complete 2×2 factorial ablation would clarify whether PMLA provides similar benefits without CAPS and whether the two components exhibit a genuine complementary interaction.

  2. The incremental improvements from multiscale linguistic supervision are relatively modest. On the validation sets, full LENS improves WER from 9.06% to 8.92% on B2T '24 and from 2.95% to 2.85% on B2T '25 relative to CAPS→CTC. Although the authors report statistically significant improvements, the gains are relatively small considering the additional linguistic encoders, projection heads, alignment estimation, and oracle-based weighting. Moreover, word-only supervision slightly worsens performance on B2T '25, suggesting that the effectiveness of individual objectives deserves further investigation.

  3. The evidence for CAPS is less consistent across datasets. CAPS improves initial-stage validation WER substantially on B2T '24 (9.90% to 9.34%), but the corresponding improvement on B2T '25 is much smaller (3.16% to 3.12%) and is not marked statistically significant. While the benefit appears to persist after continued CTC training, the statistical significance of this particular comparison is not explicitly reported.

  4. Some improvements in the ensemble setting are difficult to attribute to the proposed supervision. The ensemble combines multiple neural checkpoints with a separately fine-tuned Qwen-based voting model. Consequently, the ensemble results demonstrate strong overall system performance but provide limited evidence about the isolated effectiveness of CAPS and PMLA. The single-model results are more convincing in this regard.

Questions for authors
  1. Can the authors provide a CTC→PMLA experiment? This would complete the factorial comparison between CAPS and PMLA and help establish whether their benefits are independent, complementary, or conditional on one another.

  2. How robust are the CAPS improvements on B2T '25? Could the authors report paired per-seed improvements, confidence intervals, and significance tests for CTC→CTC versus CAPS→CTC? This would help assess whether the relatively small gains are consistent across runs.

  3. How important is the quality of the CTC-derived word alignment? Since PMLA depends on soft alignments obtained from a pretrained phoneme decoder, it would be informative to compare against hard CTC alignments or simpler temporal pooling strategies. Furthermore, the oracle-based confidence weighting appears to account for a substantial part of the incremental PMLA improvement. Could the authors further analyze this effect?

  4. Can the contribution of the neural supervision methods be separated more clearly from the ensemble decoding strategy? A comparison using matched neural checkpoint counts and identical language-model refinement would help isolate the source of the reported gains.

5 comments
AuthorsOpenReview ↗

General Response and RE: W1, Q1

We thank the reviewer for the careful reading and constructive suggestions, which helped us strengthen the controlled comparisons and clarify the contribution. New material is in blue:

  • W1, Q1: added CTC→PMLA to complete the 2×2 factorial comparison of initialization and fine-tuning objectives (Appendix D.1, Table 3).
  • W2: expanded the existing B2T ’25 linguistic-objective ablations to ten seeds (Table 2); added analyses of word identity in neural representations and sentence-target controls (Appendix F.1–F.2, Tables 8–9).
  • W3, Q2: expanded the B2T ’25 CAPS comparisons to ten matched seeds (Section 3.1, Table 2), with paired per-seed results, confidence intervals and significance tests for initial training and CTC-only continuation reported below.
  • Q3: added hard and oracle-derived pooling controls, temporal-alignment perturbations and shuffled agreement weights (Appendix E.1–E.3, Tables 5–7), complementing the existing oracle-weighting ablation (Table 2).
  • W4, Q4: added a matched ensemble comparison with the same checkpoint count and language-model refinement procedure (Appendix G.1, Table 13), while using the controlled single-model experiments as the primary evidence for CAPS and PMLA.

RE: W1, Q1 — Separating the contributions of CAPS and PMLA

We thank the reviewer for this helpful suggestion. We have completed the factorial comparison, allowing PMLA’s contribution to be assessed from standard-CTC as well as CAPS initialization on both benchmarks.

1. The factorial comparison.

Each condition uses 600 epochs of initial training followed by 600 epochs of main decoder fine-tuning. The encoder backbone, data pipeline, checkpoint-selection criterion and downstream decoding configuration are shared. The CTC-only continuation accounts for the benefit of further optimization.

Table R1. Factorial comparison. Validation WER (%), mean ± s.e.m. over ten matched seeds per dataset (Appendix D.1, Table 3).

InitializationFine-tuningB2T ’24B2T ’25
CTCCTC9.37 ± 0.113.09 ± 0.08
CTCPMLA9.22 ± 0.123.05 ± 0.07
CAPSCTC9.06 ± 0.073.01 ± 0.09
CAPSPMLA8.92 ± 0.052.93 ± 0.08

2. PMLA benefits both initializations.

With standard-CTC initialization, PMLA significantly improves validation WER over CTC-only continuation: from 9.37% to 9.22% on B2T ’24 and from 3.09% to 3.05% on B2T ’25 (one-tailed paired t-tests over ten matched seeds: p=0.045p=0.045 and p=0.042p=0.042, respectively).

With CAPS initialization, PMLA further improves validation WER from 9.06% to 8.92% on ’24 and from 3.01% to 2.93% on ’25 (one-tailed paired t-tests over ten matched seeds: p<0.001p<0.001 and p=0.0015p=0.0015, respectively).

On ’24, PMLA provides similar mean improvements from the two initializations; on ’25, the observed improvement is larger from CAPS initialization. Its benefit therefore extends to standard-CTC-trained decoders and does not depend on CAPS.

The completed factorial supports complementary contributions from the two training stages. PMLA improves recognition beyond continued CTC optimization from either starting point, while combining CAPS and PMLA achieves the lowest mean validation WER on both benchmarks.

RE: W2 — Contribution of linguistic supervision

We thank the reviewer for examining the individual objectives and their practical benefit. The revised ’25 experiments now use ten matched seeds, compared with five in the submission. CAPS→CTC and LENS now achieve 3.01% and 2.93% validation WER, respectively. Word-only supervision and CAPS→CTC both achieve 3.01%, so the slight worsening observed in the original five-seed comparison is no longer present at the reported precision.

1. Consistent improvement beyond continued CTC training.

Table R2. Linguistic-objective ablations. Validation WER (%), mean ± s.e.m. over ten matched seeds per dataset (Table 2).

Fine-tuning objectiveB2T ’24B2T ’25
CTC only (CAPS→CTC)9.06 ± 0.073.01 ± 0.09
CTC + word9.04 ± 0.063.01 ± 0.10
CTC + sentence9.03 ± 0.062.98 ± 0.08
CTC + word + sentence (PMLA)8.92 ± 0.052.93 ± 0.08

PMLA improves nine of ten validation seeds on each benchmark, with one tie on ’24. Its paired WER reductions are 0.140 percentage points on ’24 (two-sided paired-t 95% CI [0.083, 0.197]) and 0.075 on ’25 ([0.033, 0.117]). These correspond to approximately 1.5% and 2.5% relative word-error reduction, calculated from the unrounded means.

The full combination gives the lowest mean WER on both benchmarks, improving beyond the gains available through continued CTC optimization.

2. What word supervision contributes.

To examine what the word objective learns, we evaluate word identification from raw block-5 neural features using common source alignments and a cross-recording-day prototype readout. Neither the learned linguistic projection nor the downstream language model is used (Appendix F.1, Table 8).

Table R3. Word identification from raw neural features. Mean accuracy (%) over ten matched seeds per dataset.

ModelB2T ’24B2T ’25
CTC-continuation control90.1891.94
Word-only90.9592.35
PMLA90.8692.37

Word-only improves identification in all ten seeds on both datasets. Paired-bootstrap 95% intervals for its gains are [0.46, 1.11] and [0.23, 0.61] percentage points. Thus, even where its standalone WER gain is limited, word supervision makes lexical identity more accessible in the encoder. PMLA retains this lexical benefit while producing better complete-utterance recognition.

The sentence-target experiments provide complementary evidence: corresponding pretrained sentence targets outperform shuffled embeddings and random bag-of-words targets on both datasets over ten matched seeds (Appendix F.2, Table 9). Together, these results support combining localized word supervision with transcript-matched utterance supervision.

3. Preparation cost and inference requirements.

PMLA adds preparation cost: an oracle is fitted for each seed, alignments are extracted, and projection heads receive 20 epochs of preparation. Frozen text embeddings are cached and shared across runs, while each seed’s alignments and oracle agreement weights are cached for reuse.

Both PMLA and the CTC-only control then receive 600 epochs of main decoder fine-tuning. During this stage, the auxiliary branches share the encoder forward pass and use cached targets; the oracle and text encoders are not evaluated.

At inference, the oracle, text encoders and linguistic projection heads are unnecessary, and the established phoneme-to-text decoding pipeline is retained. The additional preparation therefore supports a repeatable recognition improvement without requiring these auxiliary models during deployment.

AuthorsOpenReview ↗

RE: W3, Q2

RE: W3, Q2 — Robustness of the CAPS improvement on B2T ’25

Thank you for this point. We agree that the ’25 evidence for CAPS in the submission was weaker than the ’24 evidence, and that the continued-training comparison needed its own test. We made two changes: we use ten matched seeds on ’25, and we report paired per-seed statistics for both the initial-stage and continued-training comparisons.

1. Ten matched seeds on ’25. The previously submitted ’25 results used five seeds, while ’24 used ten. The initial-stage and factorial comparisons now use ten matched seeds (Section 3.1), so the ’25 values in Tables 2 and 3 have changed: for example, CTC is now 3.28% instead of 3.16%. With ten seeds, CAPS lowers ’25 validation WER from 3.28% to 3.13%, which is significant (one-tailed paired t-test, p=0.041p=0.041; Table 2).

2. Paired per-seed statistics (Q2). Each CAPS model is paired with the standard-CTC model of the same seed. Δ\Delta = CAPS − CTC, so a negative value means CAPS is better. We report a two-sided 95% t interval for the mean Δ\Delta and the one-tailed paired t-test used throughout the paper. Relative changes use the corresponding standard-CTC condition as the reference.

Table R4. Paired comparisons over ten matched seeds. WER differences are in percentage points.

ComparisonBenchmarkMean Δ\Delta [95% CI]RelativePaired t, p (one-tailed)
CAPS vs. CTC’24−0.56 [−0.99, −0.13]−5.7%0.0078
CAPS vs. CTC’25−0.15 [−0.33, 0.02]−4.7%0.041
CAPS→CTC vs. CTC→CTC’24−0.31 [−0.69, 0.08]−3.3%0.053
CAPS→CTC vs. CTC→CTC’25−0.08 [−0.19, 0.03]−2.6%0.070

Table R5. Per-seed WER on ’25 (%). Differences are computed before rounding.

SeedCTCCAPSΔ\DeltaCTC→CTCCAPS→CTCΔ\Delta
03.142.76−0.382.9962.732−0.264
13.033.140.113.0072.859−0.148
22.922.85−0.072.7532.552−0.201
33.433.500.073.2503.2820.032
43.263.360.103.3033.113−0.190
53.773.59−0.183.4833.5790.096
63.343.09−0.253.3773.092−0.285
73.172.99−0.182.9963.1660.170
82.812.75−0.062.7532.7850.032
93.973.28−0.692.9652.912−0.053
Mean ± s.e.m.3.28 ± 0.113.13 ± 0.09−0.153.09 ± 0.083.01 ± 0.09−0.08

3. What the results show.

  • Initial stage. CAPS lowers WER by a similar relative amount on both benchmarks (5.7% on '24, 4.7% on '25). The absolute '25 gain is smaller because the baseline WER there is about three times lower (3.28% vs. 9.90%). Ten seeds therefore establish the direction of the '25 gain more firmly than its size.
  • After continued training. The mean advantage remains on both benchmarks (3.3% and 2.6% relative, p=0.053p = 0.053 and 0.070). The mean advantage persists after continued training on both benchmarks.
  • After PMLA fine-tuning. CAPS initialization gives 8.92% ('24) and 2.93% ('25), compared with 9.22% and 3.05% from standard-CTC initialization (Table 3). On '25, the CAPS-initialized model therefore has the lowest mean WER at every stage: 3.13% vs. 3.28%, 3.01% vs. 3.09%, and 2.93% vs. 3.05%.
  • The gain comes from the kernel's structure, not from relaxation alone. On '25, equal credit to every alternative (UNIFORM, 3.29%) gives no gain, and classifier-weight similarity (CLF, 3.17%) gives a smaller gain that is not significant. Only CAPS improves significantly over CTC (Table 2).

Comments on the paper

1 comment

Dear Reviewer,

Thank you very much for your time and thoughtful review. As the discussion deadline approaches, could you please take a look at the authors' rebuttal and share your thoughts in the discussion?

Many thanks for your continued help!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.