- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for synthesizing speech-related neural activity using reusable linguistic primitives and autoregressive noise. Using human premotor cortex recordings during imagined or attempted speech, the authors first find evidence that phoneme trajectories are reused across words. They then compare a linear model combining phoneme, biphone, syllable, and word kernels with a nonlinear TCN. Synthetic neural activity is compared with real recordings through dynamics, statistical properties, and phoneme decoding performance. Finally, synthetic trajectories for unseen sentences augment decoder training, improving performance on held-out real recordings, especially when data are scarce.
Critical strengths
- Interesting and well-motivated research question. The paper connects neural compositionality with the practical challenge of limited training data for speech BCIs. Exploring whether reusable neural primitives can support sentence-level data synthesis is conceptually appealing and potentially valuable for both understanding speech-related neural activity and improving data-efficient decoding.
- Plausible augmentation method supported by useful controls. The paper presents an interpretable linear generator and a nonlinear TCN alternative, evaluating synthetic activity through neural statistics and downstream decoding performance. Tests on unseen words, experiments restricting the generator to the decoder’s limited real-data subset, and controls disabling the decoder’s built-in augmentation strengthen the evidence for the approach’s practical utility.
Critical weaknesses
- Limited support for the mechanistic interpretation. The experiments provide evidence of reusable phoneme representations, but do not establish that sentence-level neural activity follows the proposed four-level additive mechanism. Decoding performance is an indirect measure of neural fidelity, particularly because noise amplitude is tuned to minimize decoding error. Comparisons with alternative encoding models and quantitative prediction of held-out neural activity would strengthen this interpretation.
- Ambiguity in the evaluation protocol. The main text refers to test-set checkpoint selection and reporting the best test PER, whereas Appendix A.4.3 specifies validation-based selection. If test performance guides checkpoint selection, the reported gains may be optimistic. The authors should clarify the roles of each split and confirm that final results use a test set untouched by model selection.
Questions for authors
- Could the authors clarify whether test-set performance was used for any checkpoint or hyperparameter selection? The main text and Appendix A.4.3 appear inconsistent on this point. Were the reported results evaluated on a test set untouched by model selection?
- Since adding syllable and word levels provides little benefit to the TCN, what evidence supports the proposed four-level hierarchy as a neural mechanism? Could held-out neural prediction, controlling for model capacity and temporal context, distinguish this interpretation from a more general context-dependent encoding model?
3 comments
Q2) When controlling for model capacity and temporal context, the 4-tower TCN that receives the four-level hierarchy and the TCN that receives only phonemes with a general temporal context explain the same held-out variance (current Figure 8 (a)). Held-out neural prediction therefore cannot distinguish the proposed hierarchy from a general context-dependent encoding model.
However, we see signatures that the TCN may be implicitly learning these hierarchical levels. When we train a linear classifier on the activity of each layer of the TCN that receives only the phoneme representation, syllable and word identity become progressively decodable in deeper layers, beyond what is explained by the lower level and by the same network with random weights (current Figure 8 (b)). While these results suggest that the TCN learns higher-level linguistic structure, they do not establish that the proposed four-level hierarchy is the underlying neural mechanism. We have clarified this distinction in the discussion:
Adding higher-level inputs provides little benefit to the TCN. However, we see hints that the phoneme-only TCN may implicitly learn syllable and word structure as its temporal context increases with depth. It remains unclear whether neural activity follows an explicit four-level hierarchy or whether these levels emerge from a more general context-dependent encoding.
The authors have addressed my concerns about the mechanistic interpretation, checkpoint selection, and the predictive contribution of the proposed four-level hierarchy through additional experiments and clarifications. I will therefore maintain my original positive recommendation of weak accept.
We thank the reviewer for the comments and suggestions.
W1) We agree that decoding performance is an indirect measure, and we added two analyses based on the prediction of held-out neural activity.
We computed the variance of the held-out neural activity explained by each model's predicted mean activity during speech, and compared it with an alternative latent diffusion encoding model, LDNS, and with the variance shared between repeats of the same sentence in the same day (currently in Appendix B, Table 7). Below is the percentage of explained variance and 95% CI for the different encoding models:
The explained variance of the linear model progressively increases as the higher level kernels are included, and the decoded PER decreases. To test whether the biphone, syllable and word levels capture real structure or only add parameters, we compared each level of the linear model with a control in which that level's labels are randomly permuted across tokens, keeping the number of kernels, the examples per kernel and the unit timing identical. On the Card and Willet datasets, the true labels improve held-out prediction at every level (biphone +0.28 / +0.33, syllable +0.08 / +0.16, word +0.04 / +0.05 R² points over the level below, all 95% CIs exclude zero), while the permuted labels do not (Appendix B, Table 8).
The TCN explains significantly more variance than the linear and LDNS models (7.3% for Card, 5.3% for Willet). Additionally, the TCN that receives four-level hierarchical information does not explain more variance than the TCN that receives phoneme-only information (Figure 8).
Q1) We selected the best test PER checkpoint, which we acknowledge leads to an optimistic estimate. We now report the PER at the last test epoch, leaving the test set untouched during selection in the augmentation experiments of the current Figure 4, 5, 6, 9-13. We have also corrected the wording error in Section 2.2, 2.4, 3.3, and A.4.3. Below is the comparison for real data fraction p and augmentatio fraction a, each cell is best checkpoint / last epoch:
Card
Willett
Kunz T12
Kunz T15
The results are very similar and the same qualitative conclusions hold: the temporal convolutional network (TCN) provides larger augmentation gains than the linear model, with the largest gains when real data is scarce.