- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Neural Evidence Transport for reusing a participant-specific neural decoder when a downstream speech tokenizer changes. It converts source posterior support into frequency-normalized evidence, averages that evidence through an audio-derived soft bridge, and fuses it with a frozen target-token prior using development-selected context-dependent weights. The method is evaluated on held-out MEG participants and an independent EEG attended-speech task without fitting a new target-space neural decoder for confirmation participants.
Critical strengths
- The paper is generally well written and easy to follow. It addresses cross-tokenizer decoder reuse in neural decoding through a ratio-based transport perspective that is well motivated and mathematically specified. Neutral source evidence leaves the target prior unchanged, while the contraction result makes explicit what a many-to-one audio bridge necessarily loses.
- The empirical protocol is careful: development and confirmation participants are separated, policies are frozen before confirmation, and timing/source-label controls test whether effects depend on aligned neural evidence. The released tests and records make the reported calculations substantially more auditable than usual.
Critical weaknesses
- The claimed tokenizer generality is not established. All experiments use two closely compatible 256-state acoustic codebooks with frame-aligned assignments from the same audio. The count bridge assumes an estimable joint distribution over aligned source and target tokens. The paper does not explain how the method extends to tokenizers with different frame rates, multiple or residual codebooks, hierarchical streams, or context-dependent semantic/LLM-oriented tokens, for which this bridge may not be directly defined. The evidence therefore supports reuse between these two compatible codebooks rather than tokenizer-agnostic decoder reuse.
- The mathematical presentation remains unnecessarily abstract at key transitions. Figure 1 shows the high-level pipeline, but it does not connect Eq. 4's forward-channel equivalence, Eq. 6's exact conditional-variance loss, or Eqs. 8--9's risk/convexity argument to the computation or the selected policy. It also omits the code-vector-to-posterior construction that yields . This makes the distinction among source posterior, evidence ratio, transported distribution, and fused target posterior difficult to follow. The practical scope is also narrow: MEG confirmation shares the same stimuli across participants, while EEG is fixed offline candidate ranking; a participant-specific source decoder remains necessary and transport is not superior to direct fitting in every reported setting.
Questions for authors
- Please revise Figure 1 to link each computational transition to the relevant equations, especially the relation between the reverse bridge in Eq. 3 and the forward push-forward in Eq. 4, the loss identity in Eq. 6, and the risk argument motivating the regime-specific weights in Eqs. 8--9. Please also depict how the code-vector decoder constructs .
- How would the count-based bridge operate for tokenizers with different token rates, multiple/residual codebooks, or context-dependent semantic/LLM-oriented token streams? Please state the alignment assumptions explicitly and evaluate at least one materially different tokenizer family.
- MEG confirmation holds out participants but uses shared speech stimuli, and the masked-completion protocol can reveal ground-truth target tokens as context. How much of the reported reuse benefit remains under an unseen-stimulus MEG split and a causal or free-running target-context setting? Please also state prominently that the method is not calibration-free: a participant-specific source decoder remains necessary, and direct target fitting can be comparable or better in some settings.
2 comments
Follow-up on EnCodec and Figure 1
Following the AC's note, we briefly highlight our updates addressing your remaining points.
In Appendix D.17, we align 50 Hz HuBERT frames to 75 Hz EnCodec frames by temporal overlap and use a separate bridge for each of two 1,024-code residual streams. On the 11-participant MEG panel, NET lowered NLL by 0.00169 nat/code symbol relative to the frozen prior (using ground-truth past codes) and by 0.00304 relative to time-shifted evidence. Both group-level paired 95% intervals excluded zero.
Figure 1 now shows how source code vectors enter the posterior computation and marks where Eqs. 4, 6, and 8–9 apply, and §4.6 clarifies the per-participant source calibration and target supervision boundaries.
Thank you again for your supportive review and helpful suggestions.
Thank you for the concrete suggestions on Figure 1 and evaluation scope.
Short answer: NET shows that neural evidence can be reused across the tested acoustic interfaces without an additional target-space neural-decoder fit. Retention of Ridge’s gain is 97% in HuBERT, at least 75% in reverse MEG-SCANS and 81% in EEG. Reference-ratio transport, context-dependent convex fusion, and explicit zero-weight prior fallback define the framework. Figure 1 and the codec/WavLM results clarify the tested alignment scope; arbitrary-tokenizer and free-running transfer remain outside it. We also state the calibration requirement plainly, as you asked: the method is not calibration-free, a participant-specific source decoder remains necessary throughout, and direct target fitting is comparable or better in several of our own settings.
Reply to Q1 (Figure 1). We retain the original artwork and add four blue callouts with equation numbers: source code-vector energy and ratio, reverse/forward bridge equivalence, the exact conditional-variance loss, and the convex risk slope motivating development-frozen weights. Callout (1) shows how unit vectors va and h(x) define LA(a;x) and the posterior qA(a∣x)=πA(a)LA(a;x)/ZA, with ZA=∑aπA(a)LA(a;x). Eq. 1 forms the ratio qA/πA, which Eq. 3 transports. The caption defines unit-vector normalization and zero-weight prior fallback.
Reply to Q2 (Different tokenizer structure and alignment). Appendix D.17 evaluates HuBERT evidence at 50 Hz against two native 75-Hz EnCodec residual streams with 1,024 states each. Overlap weights wji=∣Ii∩Jj∣/∣Jj∣ map source frame cells to target cells; separate bridges use 106,962 training-audio segments. The frozen native prior predicts current codes from true past codes. Prior−NET is .00169 [.00123, .00213] and wrong-time−NET .00304 [.00253, .00353] nat/code symbol. A stronger per-stream comparison gives Ridge−NET −.00036 [−.00080, .00007], so superiority over direct fitting is not claimed. This is marginal stream transport, not a joint neural likelihood, semantic alignment or free-running generation. WavLM K=256/512 supplies the complementary vocabulary-size test (.02369/.02435 prior gains; .01139/.01033 Ridge−NET). With matched task-2 calibration, direct-target−NET is +.00558/+.00658 (11/11 positive). The four WavLM N=5/20 intervals favor NET. Both routes share N task-2 calibration windows and Ridge uses N extra fit windows. The HuBERT↔Wav2Vec2 frontier (Table 14) gives Ridge 20 extra task-2 windows per confirmation participant. At 70 fit windows, Ridge−NET is .00837 [.00496,.01185] for HuBERT→Wav2Vec2 and .00242 [−.00040,.00532] in reverse.
Reply to Q3 (Unseen stimuli, context and calibration). German MEG-SCANS (D.15) changes corpus, participants, language, setting and story. The reverse direction passes all five route checks, including 75% Ridge retention (.00356 [.00246, .00479], 11/11); the forward direction fails three of five, so 8/12 registered checks pass and the joint criterion fails. The French study (D.18) uses 30 development and 17 confirmation participants, with runs 8–9 untouched during weight selection. Its global primary selects the zero-weight prior fallback and direct Ridge gains .00500 [.00334, .00684], so this primary is reported as a failure. An earlier French feasibility check stopped before fitting at 15 eligible participants, below the minimum of 16; it remains separate from this prospective study.
Source-reference centering (D.19) gives calibrated source-Ridge gain .00490 [.00374, .00603] on new 30-person runs (29/30; wrong-time−aligned .01092 [.00967, .01215]); bit-identical MLP replay gains .00392 [.00289, .00499]. A disjoint 14-person panel gains .00788 [.00541, .01073] (13/14), but both global IID-50 bounds fail. The locked shared-panel context extension adds .00516 [.00394, .00634] (28/30; 93.9% of calibrated direct-Ridge context gain). Its IID-50 interval includes zero, so it is descriptive secondary evidence, not independent replication or a replacement for the failed global test. Target labels select fusion weights, whereas source-reference means use source predictions only.
Reply to computation and reliability. Component timings (D.16) are .284/.349/.613 ms (source/transport/transport-plus-fusion) for a synthetic three-second input on one Apple CPU thread; full-pipeline latency is unmeasured. EEG is 60.6% versus target Ridge’s 63.0%; 81% denotes retention of Ridge’s gain over chance. We preserve these limitations, as well as the untested cross-participant decoder transfer, free-running and streaming settings. On the shared-stimulus LibriBrain100 panel the 50%/75% suffix-mask prior gains are .02279/.02734 forward and .03271/.02692 reverse, and every such score conditions on a supplied true prefix with no predicted token fed back.