acceptodds
Under review as a conference paper at ICLR 2027

Neural Evidence Transport: Reusing Brain Decoders Across Speech Tokenizers

Abstract

Speech tokenizers define different vocabularies for the same audio, preventing a brain decoder trained on one vocabulary from directly serving another. We introduce Neural Evidence Transport to reuse a participant’s decoder with a frozen target model. The method divides source probabilities by reference token frequencies and averages the resulting ratios through a soft bridge estimated from ordinary audio. This transfers support relative to baseline, avoiding repeated frequency weighting when updating the target prior. We prove that transport preserves neutral evidence and derive an exact identity for the loss of chi-square contrast. Fusion weights depend on the available target-token context, are selected on development participants, and remain fixed for confirmation. On magnetoencephalography (MEG) data from 11 held-out LibriBrain100 participants, transport reduces target-token negative log-likelihood relative to the prior and single-weight fusion in both HuBERT–Wav2Vec2 directions. It outperforms matched target-space decoders for Wav2Vec2 and retains 97% of Ridge’s improvement over the prior for HuBERT. In an independent electroencephalography (EEG) study with 14 held-out participants, transport achieves 60.6% attended-speaker accuracy over 12 seconds (95% confidence interval: 56.4–65.6%), retaining 81% of the target-space Ridge gain over chance. Timing and source-label controls link these gains to aligned neural observations and vocabulary correspondence. These results demonstrate decoder reuse without an additional participant-specific target-space neural fit.

Primary area:
applications to neuroscience & cognitive science
Keywords:
neural speech decoding, decoder reuse, cross-tokenizer transfer, density-ratio fusion, EEG, MEG

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RkjZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Neural Evidence Transport for reusing a participant-specific neural decoder when a downstream speech tokenizer changes. It converts source posterior support into frequency-normalized evidence, averages that evidence through an audio-derived soft bridge, and fuses it with a frozen target-token prior using development-selected context-dependent weights. The method is evaluated on held-out MEG participants and an independent EEG attended-speech task without fitting a new target-space neural decoder for confirmation participants.

Critical strengths
  1. The paper is generally well written and easy to follow. It addresses cross-tokenizer decoder reuse in neural decoding through a ratio-based transport perspective that is well motivated and mathematically specified. Neutral source evidence leaves the target prior unchanged, while the contraction result makes explicit what a many-to-one audio bridge necessarily loses.
  2. The empirical protocol is careful: development and confirmation participants are separated, policies are frozen before confirmation, and timing/source-label controls test whether effects depend on aligned neural evidence. The released tests and records make the reported calculations substantially more auditable than usual.
Critical weaknesses
  1. The claimed tokenizer generality is not established. All experiments use two closely compatible 256-state acoustic codebooks with frame-aligned assignments from the same audio. The count bridge assumes an estimable joint distribution over aligned source and target tokens. The paper does not explain how the method extends to tokenizers with different frame rates, multiple or residual codebooks, hierarchical streams, or context-dependent semantic/LLM-oriented tokens, for which this bridge may not be directly defined. The evidence therefore supports reuse between these two compatible codebooks rather than tokenizer-agnostic decoder reuse.
  2. The mathematical presentation remains unnecessarily abstract at key transitions. Figure 1 shows the high-level pipeline, but it does not connect Eq. 4's forward-channel equivalence, Eq. 6's exact conditional-variance loss, or Eqs. 8--9's risk/convexity argument to the computation or the selected policy. It also omits the code-vector-to-posterior construction that yields qAq_A. This makes the distinction among source posterior, evidence ratio, transported distribution, and fused target posterior difficult to follow. The practical scope is also narrow: MEG confirmation shares the same stimuli across participants, while EEG is fixed offline candidate ranking; a participant-specific source decoder remains necessary and transport is not superior to direct fitting in every reported setting.
Questions for authors
  1. Please revise Figure 1 to link each computational transition to the relevant equations, especially the relation between the reverse bridge in Eq. 3 and the forward push-forward in Eq. 4, the loss identity in Eq. 6, and the risk argument motivating the regime-specific weights in Eqs. 8--9. Please also depict how the code-vector decoder constructs qAq_A.
  2. How would the count-based bridge operate for tokenizers with different token rates, multiple/residual codebooks, or context-dependent semantic/LLM-oriented token streams? Please state the alignment assumptions explicitly and evaluate at least one materially different tokenizer family.
  3. MEG confirmation holds out participants but uses shared speech stimuli, and the masked-completion protocol can reveal ground-truth target tokens as context. How much of the reported reuse benefit remains under an unseen-stimulus MEG split and a causal or free-running target-context setting? Please also state prominently that the method is not calibration-free: a participant-specific source decoder remains necessary, and direct target fitting can be comparable or better in some settings.
2 comments

Thank you for the concrete suggestions on Figure 1 and evaluation scope.

Short answer: NET shows that neural evidence can be reused across the tested acoustic interfaces without an additional target-space neural-decoder fit. Retention of Ridge’s gain is 97% in HuBERT, at least 75% in reverse MEG-SCANS and 81% in EEG. Reference-ratio transport, context-dependent convex fusion, and explicit zero-weight prior fallback define the framework. Figure 1 and the codec/WavLM results clarify the tested alignment scope; arbitrary-tokenizer and free-running transfer remain outside it. We also state the calibration requirement plainly, as you asked: the method is not calibration-free, a participant-specific source decoder remains necessary throughout, and direct target fitting is comparable or better in several of our own settings.

Reply to Q1 (Figure 1). We retain the original artwork and add four blue callouts with equation numbers: source code-vector energy and ratio, reverse/forward bridge equivalence, the exact conditional-variance loss, and the convex risk slope motivating development-frozen weights. Callout (1) shows how unit vectors vav_a and h^(x)\widehat h(x) define LA(a;x)L_A(a;x) and the posterior qA(a∣x)=πA(a)LA(a;x)/ZAq_A(a\mid x)=\pi_A(a)L_A(a;x)/Z_A, with ZA=∑aπA(a)LA(a;x)Z_A=\sum_a \pi_A(a)L_A(a;x). Eq. 1 forms the ratio qA/πAq_A/\pi_A, which Eq. 3 transports. The caption defines unit-vector normalization and zero-weight prior fallback.

Reply to Q2 (Different tokenizer structure and alignment). Appendix D.17 evaluates HuBERT evidence at 50 Hz against two native 75-Hz EnCodec residual streams with 1,024 states each. Overlap weights wji=∣Ii∩Jj∣/∣Jj∣w_{ji}=|I_i\cap J_j|/|J_j| map source frame cells to target cells; separate bridges use 106,962 training-audio segments. The frozen native prior predicts current codes from true past codes. Prior−NET is .00169 [.00123, .00213] and wrong-time−NET .00304 [.00253, .00353] nat/code symbol. A stronger per-stream comparison gives Ridge−NET −.00036 [−.00080, .00007], so superiority over direct fitting is not claimed. This is marginal stream transport, not a joint neural likelihood, semantic alignment or free-running generation. WavLM K=256/512 supplies the complementary vocabulary-size test (.02369/.02435 prior gains; .01139/.01033 Ridge−NET). With matched task-2 calibration, direct-target−NET is +.00558/+.00658 (11/11 positive). The four WavLM N=5/20 intervals favor NET. Both routes share N task-2 calibration windows and Ridge uses N extra fit windows. The HuBERT↔Wav2Vec2 frontier (Table 14) gives Ridge 20 extra task-2 windows per confirmation participant. At 70 fit windows, Ridge−NET is .00837 [.00496,.01185] for HuBERT→Wav2Vec2 and .00242 [−.00040,.00532] in reverse.

Reply to Q3 (Unseen stimuli, context and calibration). German MEG-SCANS (D.15) changes corpus, participants, language, setting and story. The reverse direction passes all five route checks, including 75% Ridge retention (.00356 [.00246, .00479], 11/11); the forward direction fails three of five, so 8/12 registered checks pass and the joint criterion fails. The French study (D.18) uses 30 development and 17 confirmation participants, with runs 8–9 untouched during weight selection. Its global primary selects the zero-weight prior fallback and direct Ridge gains .00500 [.00334, .00684], so this primary is reported as a failure. An earlier French feasibility check stopped before fitting at 15 eligible participants, below the minimum of 16; it remains separate from this prospective study.

Source-reference centering (D.19) gives calibrated source-Ridge gain .00490 [.00374, .00603] on new 30-person runs (29/30; wrong-time−aligned .01092 [.00967, .01215]); bit-identical MLP replay gains .00392 [.00289, .00499]. A disjoint 14-person panel gains .00788 [.00541, .01073] (13/14), but both global IID-50 bounds fail. The locked shared-panel context extension adds .00516 [.00394, .00634] (28/30; 93.9% of calibrated direct-Ridge context gain). Its IID-50 interval includes zero, so it is descriptive secondary evidence, not independent replication or a replacement for the failed global test. Target labels select fusion weights, whereas source-reference means use source predictions only.

Reply to computation and reliability. Component timings (D.16) are .284/.349/.613 ms (source/transport/transport-plus-fusion) for a synthetic three-second input on one Apple CPU thread; full-pipeline latency is unmeasured. EEG is 60.6% versus target Ridge’s 63.0%; 81% denotes retention of Ridge’s gain over chance. We preserve these limitations, as well as the untested cross-participant decoder transfer, free-running and streaming settings. On the shared-stimulus LibriBrain100 panel the 50%/75% suffix-mask prior gains are .02279/.02734 forward and .03271/.02692 reverse, and every such score conditions on a supplied true prefix with no predicted token fed back.

AuthorsOpenReview ↗

Follow-up on EnCodec and Figure 1

Following the AC's note, we briefly highlight our updates addressing your remaining points.

In Appendix D.17, we align 50 Hz HuBERT frames to 75 Hz EnCodec frames by temporal overlap and use a separate bridge for each of two 1,024-code residual streams. On the 11-participant MEG panel, NET lowered NLL by 0.00169 nat/code symbol relative to the frozen prior (using ground-truth past codes) and by 0.00304 relative to time-shifted evidence. Both group-level paired 95% intervals excluded zero.

Figure 1 now shows how source code vectors enter the posterior computation and marks where Eqs. 4, 6, and 8–9 apply, and §4.6 clarifies the per-participant source calibration and target supervision boundaries.

Thank you again for your supportive review and helpful suggestions.

Reviewer_saRdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Neural Evidence Transport, which aims to enable brain decoders trained with one speech tokenizer to support another without additional participant-specific training. The proposed method preserves neutral evidence and characterizes information loss. The experimental results demonstrate that token prediction across HuBERT and Wav2Vec2 can be effectively improved using MEG recordings. Experimental results on EEG show 60.6% attended-speaker accuracy over 12 seconds, retaining 81% of the target-trained decoder’s performance. In summary, the proposed method can support efficient decoder reuse across pretrained speech models without retraining existing neural decoders.

Critical strengths

(1) The proposed method can enable encoder reuse without retraining. More specifically, an existing participant-specific brain decoder can be combined with different speech tokenizers, consequently reducing training costs and data requirements. (2) The paper develops a reference-frequency normalization scheme that can avoid repeatedly weighting common tokens. Meanwhile, the proposed framework can preserve neutral evidence and characterize information loss. (3) The experimental results on MEG and EEG data show effective transfer, including improved attended-speaker accuracy and maintaining the target-trained Ridge decoder’s improvement.

Critical weaknesses

(1) The major contribution is to use an existing brain decoder to support a different speech tokenizer, without additional participant-specific neural training. This can be used in practical applications, but the novelty of the scientific contribution seems limited. The authors may want to highlight more scientific breakthroughs of this work compared to existing methods. (2) The experiments only consider HuBERT and Wav2Vec2 with 256 states. In many speech tasks, WavLM achieves better performance than HuBERT and Wav2Vec2. The authors may want to include WavLM in the experiments. Meanwhile, experiments on larger state spaces remain untested. (3) The generalization capability of the system is critical. Therefore, the authors may want to report performance on unseen speakers, languages, and recording conditions. (4) Please disucss the computation costs and the real-time processing factor, so that the readers can understand the applicability of the proposed method in real-world application scenarios. (5) The attended-speaker result is 60.6%, which is below the target-space Ridge decoder’s 63.0%. The results suggest limited reliability for practical attention detection. Please provide some explanations for this.

Questions for authors

(1) Please use other pretrained models and larger state spaces to further validate the effectiveness of the proposed method. (2) Please report performance on unseen speakers, languages, and recording conditions to validate the generalization capability. (3) Please report the computation costs and the real-time processing factor of the proposed method. (4) In addition to implementation contributions, please highlight the major contributions of the scientific breakthroughs over existing methods.

3 comments

Thank you for the suggestions.

Short answer: NET reuses an existing participant-specific decoder’s evidence across tested acoustic vocabularies without refitting a target-space neural decoder. Retention of Ridge’s gain is 97%, at least 75% and 81% in HuBERT, reverse MEG-SCANS and EEG. Reference-ratio transport, convex context fusion and a zero-weight fallback form one risk-aware framework; the evidence supports bounded reuse, not a new general theorem. We do not claim training-free reuse across participants: the source decoder remains participant-specific, and only the target-vocabulary fit is removed.

Reply to W1 & Q4 (Scientific contribution). NET reuses an existing participant-specific decoder’s soft evidence with a frozen prior over a different acoustic vocabulary. TokenMapper translates token sequences; distillation trains a student; shared-vocabulary density-ratio fusion corrects a prior. NET keeps the source decoder and target prior fixed, obtains the bridge from ordinary audio, and transfers relative evidence rather than a raw posterior. Its scientific addition is the neutral reference-ratio operator, the conditional-variance loss identity (Eq. 6), and convex context-risk criterion (Eqs. 8–9), with tests that expose when the update should fall back to the prior. The zero-weight fallback is an allowed optimum of this same risk objective and returns the target prior exactly; it is not a post-hoc patch. The tools are standard; the contribution is the problem, the framework and the abstention tests.

Source-reference centering (D.19) makes the strong-prior mechanism testable: it is neutral at the reference vector while retaining residual evidence. On new 30-person runs, calibrated source Ridge gains .00490 [.00374, .00603] nat/token (29/30; wrong-time−aligned .01092 [.00967, .01215]); bit-identical MLP replay gains .00392 [.00289, .00499], so the effect is not solely a new Ridge fit. The same centering option is applied to direct Ridge; no superiority claim is made.

Reply to W2 & Q1 (Other models and vocabularies). WavLM K=256/512 (D.14) gives prior−NET .02369/.02435 and Ridge−NET .01139/.01033. A 50-to-75-Hz HuBERT→EnCodec test (D.17) uses two native 1,024-state residual streams and overlap-weighted bridges. Prior−NET is .00169 [.00123, .00213] and wrong-time−NET .00304 [.00253, .00353] nat/code symbol, positive in both streams and all 11 participants. With per-stream weights chosen independently, Ridge−NET is −.00036 [−.00080, .00007], so we claim no superiority over flexible direct fitting. Structural tests on the existing panel, not new-population replication. Under matched task-2 calibration, WavLM direct-target−NET is +.00558 [.00338,.00797] (K=256) and +.00658 [.00438,.00888] (K=512; 11/11 positive). All four WavLM N=5/20 intervals favor NET (+.01535, +.01361, +.01466, +.01528). Both routes share N task-2 calibration windows and Ridge uses N extra fit windows. At N=5 prior−Ridge intervals include zero. The HuBERT↔Wav2Vec2 frontier (Table 14) gives Ridge 20 extra task-2 windows per confirmation participant. At 70 fit windows, Ridge−NET is .00837 [.00496,.01185] for HuBERT→Wav2Vec2 and .00242 [−.00040,.00532] in reverse.

Reply to W3 & Q2 (Generalization). German MEG-SCANS (D.15) changes corpus, participants, language, recording setting and story. It gives .00973/.03212 prior gains. Reverse passes all five route checks, including 75% Ridge retention (.00356 [.00246,.00479], 11/11), failing only both-direct-families; forward fails three of five, so 8/12 pass and the joint criterion fails. The separate French study (D.18) uses 30 development and 17 confirmation participants on previously unaccessed runs 8–9; its frozen global policy selects the zero-weight fallback and direct Ridge gains .00500 [.00334, .00684]. The primary returns the prior by design; the registered criterion still fails.

The 14-person French secondary panel gives calibrated global NET gain .00788 [.00541, .01073], 13/14 positive, but IID-50 is −.001045, beyond the original −.001 bound. One context-policy extension (D.19), from old 30-person development data, locked before those outcomes existed, gives .00516 [.00394, .00634] on the shared 30-person panel (28/30 positive; 93.9% of calibrated direct-Ridge context gain). Its IID-50 interval includes zero, so it is descriptive secondary evidence, not independent replication; the original global criterion still fails.

Reply to W4 & Q3 (Computation). Synthetic three-second timings (D.16) are .284/.349/.613 ms for source/transport/fusion on one Apple CPU thread. Acquisition, preprocessing, tokenization, prior inference and training are excluded; end-to-end real-time performance is unmeasured.

Reply to W5 (EEG reliability). EEG is 60.6% versus Ridge’s 63.0% at 12 seconds. The reported 81% is retention of Ridge’s gain over 50% chance, not 81% accuracy. We state the offline, two-candidate limitation and do not claim practical attention-detection reliability.

AuthorsOpenReview ↗

Follow-up on WavLM, runtime, and external validation

Following the AC's reminder, we briefly summarize our new experiments addressing your questions.

With matched fusion calibration, NET lowered NLL relative to direct Ridge by 0.00558 and 0.00658 nat/token on the 256- and 512-class WavLM Base+ targets, respectively. All 11 participants favored NET at both vocabulary sizes (§4.6). Transport and fusion together took 0.613 ms per synthetic 3-second input on one CPU thread (Appendix D.16).

In the unseen German stories (Appendix D.15), NET outperformed the prior in both directions and passed 8 of 12 prespecified checks, including all 5 within-direction checks in the reverse direction. In the independent French test (Appendix D.18), the global policy selected on development data assigned zero weight to the transported evidence. This prevented negative transfer by recovering the prior exactly, while the frozen context-specific policy (Appendix D.19) retained 93.9% of the calibrated direct Ridge gain.

Thank you again for your constructive feedback. We hope these results help inform your final assessment.

AuthorsOpenReview ↗

Follow-up on cross-lingual transfer and prior strength

Following up on our earlier response, we briefly summarize our new experiments on cross-lingual transfer and prior strength (marked in red in §3.5, §4.6, and Appendix D.20).

Expanding the expected NLL in Proposition 3 around zero weight gives closed-form context weights that match exact convex optimization across all 42 French decoder-and-context settings (R2=0.9997R^2 = 0.9997; Figure 5). Because the 8-D codebooks are centered on English centroids, German and French phonetic shifts place 70–86% of unit-MLP output energy into a constant mean offset. In German, where the English prior was left unadapted, this offset adds a static frequency gain that reverses under 50% span masking (−0.00881 nat/token). In French, where the prior is adapted to French audio (and across the prior-strength sweep in Table 27), the same offset increases risk curvature 2.8-fold, so the discrete grid assigns zero weight.

Subtracting the mean prediction on development source data (without target labels) and applying these closed-form weights resolves both cases without tuning on the confirmation cohorts (Table 28). On the 11 held-out German participants (unseen story), NET improves on the prior in all 7 contexts and retains 133.4% and 96.4% of equally calibrated direct Ridge in the two directions. On the 17 held-out French confirmation participants from Appendix D.18, source MLP and source Ridge lower NLL by 0.00614 and 0.00671 nat/token (91.4% of calibrated direct Ridge). All original prespecified results remain unchanged in Appendices D.15 and D.18–D.19.

Thank you again for your constructive feedback. Please let us know if any further clarification would be helpful.

Reviewer_EPwAOpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Neural Evidence Transport lets a brain decoder trained on one speech-token vocabulary (e.g., HuBERT) supply evidence to a frozen model over another (e.g., Wav2Vec2), without target-space neural refitting. It divides source posteriors by token frequencies, transports them through a soft bridge estimated from ordinary audio, and fuses them with a frozen target prior. On MEG and EEG, it improves over the prior and roughly matches target-space decoders.

Critical strengths
  • Principled and well-controlled method. Ablations show both frequency correction and the soft bridge matter.
  • Rigorous evaluation. Disjoint splits, pre-specified criteria, matched baselines, two modalities, and honest reporting where transport only matches direct training.
Critical weaknesses
  • Unclear practical relevance. The motivation assumes participant-specific recordings are unavailable for refitting, but it is unclear how often this happens; a small calibration set can usually be collected or retained. Transport also still needs a participant-specific source-space decoder. No concrete application is given where a decoder on one representation must feed a frozen model on another.
  • Limited scope. Only two 256-state vocabularies from similar encoders are tested, and the EEG task (choosing between two known streams) is far from the motivating scenario.
Questions for authors
  1. Can you give concrete real-world scenarios where the source decoder must be reused and a small target-space calibration set is unavailable? How common are they?
  2. How does transport compare to a direct decoder trained on a reduced calibration set?
3 comments

Thank you for asking when decoder reuse is useful and how it compares with low-label direct fitting.

Short answer: NET reuses an existing participant-specific decoder across the tested acoustic vocabularies and avoids fitting a separate neural decoder in the target vocabulary. Retention of Ridge’s gain is 97% in HuBERT, at least 75% in reverse MEG-SCANS and 81% in EEG. The low-label experiments quantify what reuse offers at different target fitting budgets.

Reply to W1 & Q1 (when a source decoder must be reused). The value of reuse depends on whether the lab can obtain a target-vocabulary training set.

  • Retention limits. A lab archives participant decoders and later must delete synchronized audio at a consent or ethics expiry. The lab can keep using the decoder after losing access to the audio it would need to create new target labels.
  • Representation change on a closed cohort. A site moves from HuBERT to WavLM or to a codec interface on recordings that are still analyzed but cannot be re-opened for a new participant-specific fit.
  • Decoder handover. A decoder arrives under a data-use agreement that permits scores but not the recordings needed to fit a new head.

We have no estimate of how often these constraints arise. When the synchronized data remain accessible, re-tokenizing and fitting a direct decoder is feasible. Table 2 tests that option at low target-label budgets. What we can state is that the requirement recurs once per vocabulary change, and the revision exercises five target vocabularies across three encoder families: Wav2Vec2, WavLM at two sizes, and a two-stream EnCodec. A lab that tracks representation layers pays one target-vocabulary fit per change, per participant; reuse pays one source fit instead. Our premise is not that recordings are unavailable, but that the target-vocabulary labelled set is too small to close the gap: 5 or 20 labels leave direct fitting behind NET (Table 2).

Reply to Q2 (reduced target labels). We compare Ridge−NET in nat/token on 11 confirmation participants, equally weighting seven masks. Brackets give paired 95% participant-bootstrap intervals. In Tables 2 and 14, NET reuses a decoder built with 70 source-labelled windows and keeps its development-frozen weights.

In Table 2, Ridge fits 5 or 20 target-labelled windows per confirmation participant. A separate group of 20 development participants selects one global fusion weight per direction and budget. Ridge uses zero confirmation task-2 calibration windows. At 5 fit windows, Ridge−NET is .02008 [.01399,.02617] for HuBERT→Wav2Vec2 and .01825 [.01164,.02500] in reverse. At 20, it is .01589 [.01100,.02074] and .01046 [.00600,.01500]. With one development-frozen weight per method, all four intervals still favor NET. The smallest advantage is .00847 [.00392,.01311] (D.13, Table 13).

In Table 14, Ridge uses an extra 20 task-2 windows per confirmation participant to select one global weight per direction and budget. For HuBERT→Wav2Vec2 at 20 fit windows, Table 14 uses weight 1.0 and gives .01797 [.01428,.02152]. Table 2 uses 0.5. The other three shared cells select the same weights under both protocols and give identical results. The gap shrinks overall, with an increase in reverse at 40 fit windows. At 70 fit windows the forward advantage is .00837 [.00496,.01185]. The reverse interval includes zero (.00242 [−.00040,.00532]).

For WavLM, matching task-2 calibration with a 70-window Ridge fit gives Ridge−NET .00558 [.00338,.00797] (K=256) and .00658 [.00438,.00888] (K=512), each 11/11 positive. At N=5/20, both methods use the same N task-2 calibration windows and Ridge also fits N target windows. All four intervals favor NET (Table 16).

Reply to W2 (scope). The original tests covered two 256-state codebooks. The revision adds WavLM K=256/512 (D.14) and a 75-Hz two-stream, 1,024-state EnCodec test with overlap-weighted bridges (D.17), so the tests now include three speech encoders, two WavLM vocabulary sizes and a residual codec. NET improves over the WavLM prior by .02369/.02435 nat/token and over the EnCodec prior by .00169 [.00123,.00213] nat/code symbol. The EnCodec wrong-time contrast is .00304. With per-stream fusion weights, the Ridge−NET interval includes zero (−.00036 [−.00080,.00007]). In 12-second offline EEG ranking of two known streams, NET reaches 60.6% and direct Ridge reaches 63.0%. The German joint criterion and French global primary both fail (D.15, D.18).

Compute cost (D.16). On one CPU thread, source inference takes .284 ms and transport alone takes .349 ms for a synthetic three-second input. Transport plus fusion takes .613 ms. These timings exclude acquisition, preprocessing, tokenization, prior inference and training. End-to-end latency remains unmeasured.

AuthorsOpenReview ↗

Follow-up on limited target supervision and deployment settings

Following the AC's note, we briefly summarize our additions on limited target supervision and practical settings.

With 5 or 20 target-labeled fitting windows, direct Ridge improved on the prior in both directions. Reusing an existing source decoder for the same participant lowered NLL by a further 0.01046–0.02008 nat/token across all four comparisons (Table 2). All four paired 95% confidence intervals also favored NET when both methods used a single fusion weight fixed on development data (Appendix D.13).

We also detailed three concrete settings where direct retraining is infeasible: expired retention windows for synchronized recordings, closed-cohort representation upgrades, and privacy-restricted decoder handoffs, while noting that their field prevalence remains unmeasured.

Thank you again for suggesting this low-supervision comparison. Please let us know if any further clarification would be helpful.

I would like to thank the authors for adequately addressing my questions. Having now gained more information about the scenarios in which the source decoders must be reused, I have been able to reassess the clarity and significance of the work more positively.

In light of this, I do not see any issues that would suggest rejecting the work, and I believe my overall rating can be raised to 3. As I have already transparently disclosed, my understanding of the work is limited, and I encourage the ACs to take this into account when weighing my comments.

Comments on the paper

2 comments

Dear Reviewer,

Thank you very much for your time and thoughtful review. As the discussion deadline approaches, could you please take a look at the authors' rebuttal and share your thoughts in the discussion?

Many thanks for your continued help!

AuthorsOpenReview ↗

Summary of second-round update on cross-lingual transfer

We thank the Area Chair and all reviewers for your comments. We have uploaded a second-round update, with new text marked in red in §3.5, §4.6, and Appendix D.20 (main text within 10 pages) and new records placed in supplement folders.

This update connects our external German (Appendix D.15) and French (Appendix D.18) boundary results to a single mechanism under Proposition 3. Phonetic shift away from English-centered codebooks concentrates 70–86% of unit-MLP energy in a constant offset. That offset reverses under 50% span masking when the prior is unadapted (German), and increases risk curvature 2.8-fold to select zero grid weight when the prior is adapted (French; Table 27). Closed-form Proposition 3 context weights match exact convex optimization across 42 settings (R2=0.9997R^2 = 0.9997; Figure 5).

Subtracting the development source mean (without target labels) and applying these frozen weights requires no confirmation tuning (Table 28). NET improves on the prior in all 7 German unseen-story contexts (n=11n = 11; 133.4% and 96.4% of calibrated direct Ridge), and lowers NLL by 0.00614 and 0.00671 nat/token on the 17 held-out French confirmation participants (91.4% of direct Ridge). All original prespecified evaluation records and per-participant source-calibration requirements remain unchanged.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.