acceptodds
Under review as a conference paper at ICLR 2027

From Phonemes to Sentences in the Brain: Composing Neural Primitives for Speech Data Synthesis

Abstract

Compositionality has been proposed as an organizing principle for how neural dynamical primitives are combined to produce movements in simple motor control tasks, such as reaching. Here, we extend this idea to speech by proposing a simple and interpretable mechanistic model of how premotor cortex constructs sentence level dynamics from reusable neural trajectories across multiple hierarchical linguistic levels: phonemes, biphones, syllables, and words. We compare it with a temporal convolutional network (TCN) that can capture non-linear interactions over a longer temporal context. We verify whether the neural activity generated by both models reproduces temporal and spectral properties of the recorded data. Both models preserve speech relevant information: a decoder trained on real neural recordings achieves comparable phoneme decoding performance on synthetic and held out real activity, across all four datasets tested. Finally, we demonstrate that augmenting real datasets with model generated activity improves speech decoding in low data regimes, reducing phoneme error rate by up to 27 percentage points with the TCN and 22 percentage points with the linear model relative to training on real data alone. This is particularly relevant for brain computer interfaces (BCIs), where neural recordings are scarce, while modern decoding models are increasingly data hungry.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Compositionality, Brain computer interfaces, Speech decoding, Neural data augmentation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EFxoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for synthesizing speech-related neural activity using reusable linguistic primitives and autoregressive noise. Using human premotor cortex recordings during imagined or attempted speech, the authors first find evidence that phoneme trajectories are reused across words. They then compare a linear model combining phoneme, biphone, syllable, and word kernels with a nonlinear TCN. Synthetic neural activity is compared with real recordings through dynamics, statistical properties, and phoneme decoding performance. Finally, synthetic trajectories for unseen sentences augment decoder training, improving performance on held-out real recordings, especially when data are scarce.

Critical strengths
  1. Interesting and well-motivated research question. The paper connects neural compositionality with the practical challenge of limited training data for speech BCIs. Exploring whether reusable neural primitives can support sentence-level data synthesis is conceptually appealing and potentially valuable for both understanding speech-related neural activity and improving data-efficient decoding.
  2. Plausible augmentation method supported by useful controls. The paper presents an interpretable linear generator and a nonlinear TCN alternative, evaluating synthetic activity through neural statistics and downstream decoding performance. Tests on unseen words, experiments restricting the generator to the decoder’s limited real-data subset, and controls disabling the decoder’s built-in augmentation strengthen the evidence for the approach’s practical utility.
Critical weaknesses
  1. Limited support for the mechanistic interpretation. The experiments provide evidence of reusable phoneme representations, but do not establish that sentence-level neural activity follows the proposed four-level additive mechanism. Decoding performance is an indirect measure of neural fidelity, particularly because noise amplitude is tuned to minimize decoding error. Comparisons with alternative encoding models and quantitative prediction of held-out neural activity would strengthen this interpretation.
  2. Ambiguity in the evaluation protocol. The main text refers to test-set checkpoint selection and reporting the best test PER, whereas Appendix A.4.3 specifies validation-based selection. If test performance guides checkpoint selection, the reported gains may be optimistic. The authors should clarify the roles of each split and confirm that final results use a test set untouched by model selection.
Questions for authors
  1. Could the authors clarify whether test-set performance was used for any checkpoint or hyperparameter selection? The main text and Appendix A.4.3 appear inconsistent on this point. Were the reported results evaluated on a test set untouched by model selection?
  2. Since adding syllable and word levels provides little benefit to the TCN, what evidence supports the proposed four-level hierarchy as a neural mechanism? Could held-out neural prediction, controlling for model capacity and temporal context, distinguish this interpretation from a more general context-dependent encoding model?
3 comments

We thank the reviewer for the comments and suggestions.

W1) We agree that decoding performance is an indirect measure, and we added two analyses based on the prediction of held-out neural activity.

We computed the variance of the held-out neural activity explained by each model's predicted mean activity during speech, and compared it with an alternative latent diffusion encoding model, LDNS, and with the variance shared between repeats of the same sentence in the same day (currently in Appendix B, Table 7). Below is the percentage of explained variance and 95% CI for the different encoding models:

ModelCardWillett
Linear, phoneme (L1)2.07 [2.00, 2.14]0.84 [0.80, 0.89]
Linear, + biphone (L2)2.31 [2.24, 2.39]1.15 [1.10, 1.21]
Linear, + syllable (L3)2.37 [2.29, 2.45]1.27 [1.21, 1.33]
Linear, + word (L4)2.39 [2.31, 2.46]1.30 [1.24, 1.36]
TCN, phoneme7.33 [7.21, 7.45]5.27 [5.16, 5.39]
TCN, 4 towers: phoneme + biphone + syllable + word7.19 [7.07, 7.31]5.16 [5.05, 5.28]
LDNS3.08 [3.00, 3.16]1.62 [1.53, 1.71]
Repeat ceiling10.33 [9.40, 11.22]6.30 [4.85, 7.56]

The explained variance of the linear model progressively increases as the higher level kernels are included, and the decoded PER decreases. To test whether the biphone, syllable and word levels capture real structure or only add parameters, we compared each level of the linear model with a control in which that level's labels are randomly permuted across tokens, keeping the number of kernels, the examples per kernel and the unit timing identical. On the Card and Willet datasets, the true labels improve held-out prediction at every level (biphone +0.28 / +0.33, syllable +0.08 / +0.16, word +0.04 / +0.05 R² points over the level below, all 95% CIs exclude zero), while the permuted labels do not (Appendix B, Table 8).

The TCN explains significantly more variance than the linear and LDNS models (7.3% for Card, 5.3% for Willet). Additionally, the TCN that receives four-level hierarchical information does not explain more variance than the TCN that receives phoneme-only information (Figure 8).

Q1) We selected the best test PER checkpoint, which we acknowledge leads to an optimistic estimate. We now report the PER at the last test epoch, leaving the test set untouched during selection in the augmentation experiments of the current Figure 4, 5, 6, 9-13. We have also corrected the wording error in Section 2.2, 2.4, 3.3, and A.4.3. Below is the comparison for real data fraction p and augmentatio fraction a, each cell is best checkpoint / last epoch:

Card

Linear‖TCN
p\a00.10.20.512‖0.10.20.512
0.13634/3534/3632/3230/3529/30‖32/3230/3029/3028/3029/29
0.252120/2221/2120/2120/2020/20‖19/1919/1917/1817/1716/16
0.51414/1414/1414/1413/1413/14‖13/1413/1312/1211/1111/11
0.751111/1111/1111/1110/1011/11‖11/1110/109/109/98/9

Willett

Linear‖TCN
p\a00.10.20.512‖0.10.20.512
0.14847/4746/4943/4443/4642/42‖40/4039/3938/3837/3837/37
0.253334/3433/3332/3232/3231/31‖30/3029/2927/2726/2626/26
0.52424/2525/2525/2524/2524/24‖23/2322/2221/2220/2019/19
0.752121/2121/2121/2121/2120/20‖20/2019/1919/1917/1717/17

Kunz T12

Linear‖TCN
p\a00.10.20.512‖0.10.20.512
0.17670/7269/7267/6969/7171/74‖69/7067/7266/7067/7165/69
0.258261/6362/6462/6562/6757/59‖62/6460/6558/6459/6459/62
0.56351/5452/5552/5651/5550/55‖48/5049/5348/5243/4639/41
0.754837/3842/4740/4242/4743/45‖35/3937/4031/3432/3727/30

Kunz T15

Linear‖TCN
p\a00.10.20.512‖0.10.20.512
0.17870/7364/6662/6563/6362/66‖65/6968/7067/7165/6964/69
0.257356/5753/5455/5657/5953/54‖59/6155/5953/5550/5247/49
0.57048/5048/4943/4443/4441/42‖55/5845/4642/4440/4238/38
0.755542/4337/3936/3841/4438/40‖39/4237/3935/3731/3230/32

The results are very similar and the same qualitative conclusions hold: the temporal convolutional network (TCN) provides larger augmentation gains than the linear model, with the largest gains when real data is scarce.

Q2) When controlling for model capacity and temporal context, the 4-tower TCN that receives the four-level hierarchy and the TCN that receives only phonemes with a general temporal context explain the same held-out variance (current Figure 8 (a)). Held-out neural prediction therefore cannot distinguish the proposed hierarchy from a general context-dependent encoding model.

However, we see signatures that the TCN may be implicitly learning these hierarchical levels. When we train a linear classifier on the activity of each layer of the TCN that receives only the phoneme representation, syllable and word identity become progressively decodable in deeper layers, beyond what is explained by the lower level and by the same network with random weights (current Figure 8 (b)). While these results suggest that the TCN learns higher-level linguistic structure, they do not establish that the proposed four-level hierarchy is the underlying neural mechanism. We have clarified this distinction in the discussion:

Adding higher-level inputs provides little benefit to the TCN. However, we see hints that the phoneme-only TCN may implicitly learn syllable and word structure as its temporal context increases with depth. It remains unclear whether neural activity follows an explicit four-level hierarchy or whether these levels emerge from a more general context-dependent encoding.

The authors have addressed my concerns about the mechanistic interpretation, checkpoint selection, and the predictive contribution of the proposed four-level hierarchy through additional experiments and clarifications. I will therefore maintain my original positive recommendation of weak accept.

Reviewer_pBrEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether speech-related neural activity in the human premotor cortex exhibits compositional structure and whether this structure can be exploited to generate synthetic neural data for improving speech decoding in data-scarce settings.

The authors first examine whether phoneme-specific neural trajectories are preserved across different words using leave-one-word-out phoneme templates, Procrustes alignment, and permutation testing. Building on this analysis, they propose a linear generative model that constructs sentence-level neural activity by superimposing onset-triggered kernels associated with phonemes, biphones, syllables, and words. An autoregressive model is used to generate residual noise. The authors also introduce a temporal convolutional network (TCN) to capture nonlinear interactions and longer-range temporal dependencies.

The proposed models are evaluated on four dataset configurations derived from the Card, Willett, and Kunz studies. The experiments assess phoneme decoding performance on synthetic activity, temporal and spectral statistics, generalization to unseen words, and the benefits of synthetic data augmentation for downstream decoding. The results suggest that synthetic neural activity preserves information relevant to phoneme decoding and that augmenting real training data with synthetic samples can substantially improve decoding performance, particularly when real training data are scarce. The authors also investigate an online setting in which the generative model is trained using only the real data available to the downstream decoder.

The main contribution is an interpretable, linguistically structured approach to neural data generation, together with empirical evidence of its utility for data-efficient speech decoding. Although questions remain regarding neural signal fidelity, generalization, and comparisons with existing generative approaches, the work presents a promising direction for neural speech decoding and brain–computer interfaces.

Critical strengths
  1. An interpretable approach to compositional neural activity generation.

The proposed linear model represents sentence-level neural trajectories as sums of reusable kernels associated with multiple linguistic levels, including phonemes, biphones, syllables, and words. This formulation translates the hypothesis of neural compositionality into an explicit and testable generative mechanism. The comparison with a TCN further explores how nonlinear temporal modeling complements explicit hierarchical structure. This combination of interpretability and practical generative modeling is a valuable contribution.

  1. Meaningful downstream benefits under data-scarce conditions.

The paper evaluates synthetic data not only in terms of their decodability but also through their use in training a decoder that is subsequently evaluated on held-out real neural recordings. The reported improvements across multiple dataset configurations are promising, particularly when only a small fraction of real training data is available. The online experiments further strengthen the practical relevance of the approach by restricting the generative model to the same limited real-data fraction available to the downstream decoder.

Critical weaknesses
  1. Potential confounding between neural signal fidelity and downstream decoding performance.

The residual noise scale is selected according to validation PER rather than exclusively according to the statistical fidelity of the generated neural activity. This optimization may make the synthetic signals easier to decode without necessarily making them more faithful to real neural recordings. Since the authors observe lower PER on synthetic data than on real data in some settings, the contribution of compositional modeling is not fully separated from the effects of noise calibration and signal difficulty.

A useful additional experiment would compare the current decoding-oriented calibration with a calibration based on real neural signal statistics, while measuring both downstream decoding performance and signal fidelity. A sensitivity analysis across noise scales would also help establish whether the augmentation gains are robust. This issue does not negate the practical utility of the synthetic data, but it limits the strength of the mechanistic interpretation.

  1. Limited evidence for cross-subject generalization and statistical robustness.

The four dataset configurations involve only two participants, T12 and T15. Although the experiments provide evidence across multiple datasets and task settings, they do not establish broad generalization across individuals. Some evaluation splits are also small, and uncertainty across random seeds and repeated data subsampling is not sufficiently characterized.

Additional uncertainty estimates and evaluation on more participants would strengthen the claims regarding robustness and generalizability. The authors should also clarify the extent to which the datasets and recording sessions are independent, rather than treating the four configurations as four independent participant-level validations.

  1. Lack of direct comparison with existing neural data generation methods.

The paper compares the proposed linear and TCN models and investigates several residual noise models, but it does not provide a systematic comparison with other neural data generation approaches. For example, latent diffusion models for neural activity, such as LDNS, offer an alternative approach to generating synthetic neural data by modeling their latent distributions.

Although these methods may differ in their objectives, input features, and conditioning requirements, a comparison under a unified data split, decoder, and training-data budget would help establish the relative advantages of the proposed linguistically structured approach. The absence of such a comparison limits the assessment of its competitiveness against existing generative methods, although it does not diminish the value of the proposed compositional modeling framework itself.

Questions for authors
  1. Comparison with Existing Data Augmentation Methods

The paper demonstrates that the proposed generative models can improve speech decoding, particularly when real training data are scarce. However, a direct comparison with existing data augmentation methods would help establish the practical value of the proposed approach. Could the authors compare their method against conventional augmentation strategies, such as Gaussian noise injection and temporal smoothing, under matched real-data budgets and evaluation protocols? Such a comparison would clarify the additional benefits of learning a structured neural generative model over simpler augmentation techniques.

  1. Practical Data Efficiency Gains from Synthetic Data

A central motivation of this work is to reduce the amount of real neural data required to train effective speech decoders. Although the reported results are promising, the magnitude of the improvements varies across datasets and data availability settings. Could the authors quantify how much real training data can be saved by synthetic augmentation while maintaining a given decoding performance on held-out real recordings? In particular, how do the gains in the online setting compare with those of conventional augmentation methods under the same real-data constraints? This analysis would help establish the practical value of the proposed approach for speech neuroprostheses, where neural data collection is costly and limited.

  1. The Relationship Between Noise Calibration and the Contribution of the Generative Model

The authors select the residual noise scaling coefficient by minimizing validation phoneme error rate (PER), thereby optimizing the downstream decoding performance of the synthetic data. While this is a reasonable strategy for data augmentation, it complicates the distinction between decoding utility and fidelity to real neural activity. Could the authors clarify whether the augmentation benefits on held-out real test data persist when the noise scaling coefficient is calibrated to match the residual statistics of real neural activity rather than selected based on validation PER? This comparison would help clarify how much of the observed improvement is attributable to the learned compositional structure and how much depends on task-oriented noise optimization.

5 comments

We thank the reviewer for the comments and suggestions.

W1 and Q3) We thank the reviewer for this comment. We agree that the noise gain tuned by the PER makes synthetic data easier to decode than real data. Real single trials are dominated by variability that is not tied to the sentence: two repeats of the same sentence on the same day share only 10.3% (T15, Card dataset) and 6.3% (T12, Willett dataset) of their variance during speech. The noise gain sets how much residual variability is added back. At gain 1, the added noise has the variance and the temporal and cross-channel covariance of the model's residuals on real data.

Following the reviewer's suggestion, we compared the PER based calibration with the neural signal statistics based calibration:

  • We extended the noise-gain sensitivity analysis (Appendix Fig. 5) to the TCN model and all datasets in the current Table 4. At gain 1, synthetic data from the TCN is about as decodable as real data or easier, while synthetic data from the linear model is much harder to decode.

  • We repeated the augmentation experiments considering noise gain equal to 1 (current Figure 12). The augmentation benefit persists for TCN. The linear model still improves over real data alone, but less than with the tuned gain. The test PER is below, with each cell representing PER tuned noise gain / gain 1 for different real data fractions p and augmentation factors a:

Card

Linear‖TCN
p\a00.10.20.512‖0.10.20.512
0.13635/3536/3332/3235/3230/32‖32/2930/2730/2530/2429/24
0.252122/2121/2221/2120/2120/20‖19/1819/1818/1717/1516/14
0.51414/1514/1514/1414/1414/14‖14/1313/1312/1211/1111/10
1109/99/99/109/99/9‖9/99/98/88/78/7

Willett

Linear‖TCN
p\a00.10.20.512‖0.10.20.512
0.14847/4649/4744/4546/4542/45‖40/3939/3738/3538/3437/33
0.253334/3433/3432/3432/3331/32‖30/3029/2927/2626/2426/23
0.52425/2525/2625/2625/2624/25‖23/2422/2322/2220/2019/19
11918/1919/2019/1919/1918/19‖17/1818/1716/1716/1615/15

Kunz T12

Linear‖TCN
p\a00.10.20.512‖0.10.20.512
0.17672/7372/7369/7371/7374/71‖70/6772/6770/6771/6569/63
0.258263/7164/6665/6567/6359/61‖64/7265/6264/5764/5262/47
0.56354/5255/5556/5055/4755/44‖50/4753/4652/4646/3741/31
12932/2330/2731/2831/2431/26‖31/3034/2530/2527/2426/21

Kunz T15

Linear‖TCN
p\a00.10.20.512‖0.10.20.512
0.17873/8166/7665/6363/6166/60‖69/6570/6571/6369/6169/60
0.257357/5754/6656/5259/5054/53‖61/6159/5655/5052/4449/42
0.57050/5249/4444/4044/4242/39‖58/4646/4544/3942/3638/33
13934/3437/3233/3235/3033/29‖33/3332/3131/2935/2832/26
  • We computed the held-out variance explained by the models' predicted mean activity. During speech, the TCN explains 7.3% (Card) and 5.3% (Willett) of the variance, i.e. 71% and 84% of the variance shared between same-day repeats of the same sentence (10.3% and 6.3%). The linear model explains 2.4% and 1.3% (23% and 21%). These results are now in Table 7.

W2) We have now made clear in the text that these are data for two subjects across two different tasks:

To evaluate the linear and TCN models, we compare phoneme decoding accuracy on real and synthetic data across Card, Willett, Kunz T12, and Kunz T15 datasets (two participants, T12 and T15, each recorded in two tasks), using BiT, a transformer-based neural decoder trained separately for phoneme decoding on each subject."

Unfortunately, these are the only publicly available data for speech BCI, which also demonstrates the importance of data augmentation for the field.

We follow the standard train/validation/test splits used in prior work on these datasets, such as Zhang et al, “A cross species neural fountain model for end-to-end speech decoding”, ICLR 2026.

W2-continued) The augmentation experiment for all real faction and augmentation factor in one node with 8 NVIDIA H100 80 GB GPUs take approximately 1750 hours for Card, 960 for Willet, 460 for Kunz T15 and 80 for Kunz T12, so repeating the experiments over several seeds is computationally infeasible. We therefore quantify the uncertainty with respect to the evaluation data, using 100 bootstrap resamples of the test trials. To better visualize the results, we added a figure (current Figure 5) showing the PER reduction relative to training on real data only, as a function of the augmentation factor, for the linear, TCN, and resample augmentations; values above zero indicate an improvement, and shaded bands show ± bootstrap standard error.

W3) We now compare our models with LDNS using the same data split, neural decoder, and training budget. We evaluate decoded PER, held-out explained variance, how well each model captures the temporal and spectral properties of real data, and its benefits for data augmentation. Details of the LDNS implementation are now provided in Appendix A.5.

  • The decoded PER is significantly higher for LDNS, than for the linear and TCN model (current Table 1). Below is the PER for all datasets and models: | Dataset | Real | Linear| TCN | LDNS | |---|---|---|---|---| | Card | 7.83 ± 0.38 | 9.91 ± 0.33 | 7.35 ± 0.28 | 61.41 ± 0.42 | | Willett | 15.96 ± 0.58 | 13.61 ± 0.52 | 10.78 ± 0.46 | 59.01 ± 0.56 | | Kunz T12 | 28.50 ± 2.62 | 12.76 ± 2.03 | 12.78 ± 1.66 | 62.24 ± 3.01 | | Kunz T15 | 37.68 ± 3.74 | 17.49 ± 1.94 | 33.02 ± 3.78 | 75.00 ± 2.51 |
  • In terms of held-out explained variance, the LDNS improves in comparison with the linear model, but not in comparison with the TCN (Table 7). Below is the explained variance for the Card and Willet datasets, for all models: | Dataset | Linear | TCN | LDNS | |---|---|---|---| | Card | 2.39 [2.31, 2.46] | 7.33 [7.21, 7.45] | 3.08 [3.00, 3.16] | | Willett | 1.30 [1.24, 1.36] | 5.27 [5.16, 5.39] | 1.62 [1.53, 1.71] |
  • The LDNS captures the real data autocorrelation and power spectra density similar to the linear model, but less accurately than the TCN (Figure 3).
  • LDNS improves decoding through data augmentation when real data is very scarce (Card and Willett at p=0.1, and Kunz T12 and T15), with gains similar to the linear model. However, when more real data is available, LDNS augmentation worsens decoding, while the linear model still provides benefits. The PER as a function of the real data fraction p and augmentation factor a is below:

Card

Linear‖TCN‖LDNS
p\a00.10.20.512‖0.10.20.512‖0.10.20.512
0.1363536323530‖3230303029‖3332333333
0.25212221212020‖1919181716‖2222232324
0.5141414141414‖1413121111‖1515161718
0.75111111111011‖11101099‖1212131315
11099999‖99888‖1011111213

Willett

Linear‖TCN‖LDNS
p\a00.10.20.512‖0.10.20.512‖0.10.20.512
0.1484749444642‖4039383837‖4341404140
0.25333433323231‖3029272626‖3231313132
0.5242525252524‖2322222019‖2526252626
0.75212121212120‖2019191717‖2122222223
1191819191918‖1718161615‖1919202019

Kunz T12

Linear‖TCN‖LDNS
p\a00.10.20.512‖0.10.20.512‖0.10.20.512
0.1767272697174‖7072707169‖7979737372
0.25826364656759‖6465646462‖6868666262
0.5635455565555‖5053524641‖5457584943
0.75483847424745‖3940343730‖5047433630
1293230313131‖3134302726‖3231332430

Kunz T15

Linear‖TCN‖LDNS
p\a00.10.20.512‖0.10.20.512‖0.10.20.512
0.1787366656366‖6970716969‖7774737170
0.25735754565954‖6159555249‖6868696766
0.5705049444442‖5846444238‖6366626061
0.75554339384440‖4239373232‖5060606557
1393437333533‖3332313532‖4542555155

LDNS captures overall population statistics (Figure 3, Table 7) but less trial-level speech information (Table 1). This is consistent with LDNS being designed to model general population dynamics rather than to map a given phoneme sequence onto neural activity.

Q1) We added experiments with conventional augmentation, in the online setting: the same number of additional trials, resampled from the available real trials, with the BiT's standard augmentations (Gaussian smoothing, white noise and constant offsets). These use the same real-data budget, the same number of training updates and the same evaluation protocol as our augmentation; results are shown in the current Figures 5 and 9. Below is the PER as a function of the real data fraction p and the augmentation factor a:

Card

Linear‖TCN‖Resample
p\a00.10.20.51248‖0.10.20.51248‖0.10.20.51248
0.13635363235302930‖32303030292929‖35363831303132
0.252122212120201818‖19191817161718‖22212020191918
0.51414141414141313‖14131211111212‖14141414131212
0.751111111110111010‖1110109999‖1111111010109
1109999998‖9988877‖9999888

Willett

Linear‖TCN‖Resample
p\a00.10.20.51248‖0.10.20.51248‖0.10.20.51248
0.14847494446424040‖40393838373837‖48474644434343
0.253334333232313030‖30292726262625‖33333332313130
0.52425252525242323‖23222220191919‖25252524232323
0.752121212121202020‖20191917171616‖21202120201920
11918191919181818‖17181616151414‖19181918181817

Kunz T12

Linear‖TCN‖Resample
p\a00.10.20.51248‖0.10.20.51248‖0.10.20.51248
0.17672726971747175‖70727071698779‖84787981808082
0.258263646567596863‖64656464626060‖82737476797684
0.56354555655555551‖50535246414439‖64666263636057
0.754838474247453932‖39403437303636‖46464141463533
12932303131312920‖31343027262118‖29272828282320

Kunz T15

Linear‖TCN‖Resample
p\a00.10.20.51248‖0.10.20.51248‖0.10.20.51248
0.17873666563667069‖69707169697465‖79787976757475
0.257357545659545759‖61595552495450‖60717271757372
0.57050494444424845‖58464442384240‖67667169647068
0.755543393844404036‖42393732323232‖64636062626054
13934373335333330‖33323135322830‖40536351423540

The benefit of TCN augmentation persists across all datasets and increases as less real data is available. On the other hand, the linear augmentation benefit persists when the available data is small, for p=0.1 Card and Willet, and for all p for Kunz T12 and T15.

Q2) Thank you for the great suggestion, we have added the analysis in current Figure 6. For each dataset, we plot the held out PER as a function of the amount of real training data (in hours) at augmentation factor a=8, and measure how much real data each method needs to reach the PER obtained with all real data and no augmentation. The TCN augmentation outperforms conventional augmentation under the same real-data constraints: it saves 47% of the real data for Willett (7.2 h) and 48% for Kunz T15 (84 min), compared with 13% for conventional augmentation, and it gives a lower PER when real data is scarce (e.g. at 10% of the real data: 28.7% vs 31.9% for Card, 37.3% vs 42.8% for Willett). The linear augmentation improves over conventional augmentation only when real data is very limited, i.e. at the smallest data fractions for Card and Willett and for the Kunz datasets.

We address Question 3 in our response to Weakness 1.

Reviewer_do4sOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work models sentence-level premotor activity using reusable kernels associated with phonemes and higher-level linguistic units. A linear superposition model and a temporal convolutional network generate neural trajectories, with autoregressive residual noise added to both. The authors examine phoneme-trajectory similarity across words, compare real and synthetic activity, and test whether synthetic recordings improve a speech decoder when real training data are limited.

Critical strengths

The simple linear model is a useful part of the contribution, not merely a weak baseline for the TCN. It makes the reuse of linguistic primitives explicit and allows the authors to test synthesis for words without a fitted word-specific kernel. Matching unseen words to in-vocabulary controls by length, duration, and phoneme frequency makes that comparison more informative than an unconditional decoding average.

I was also glad to see the restricted-data generator experiment in Appendix B. Fitting the generator on the full dataset would not establish an advantage under genuinely scarce recordings. Figure 6 addresses that concern by restricting the generator to the same real-data fraction as the decoder, and the reported benefit persists. That is an important control for the intended application.

Critical weaknesses

The checkpoint-selection descriptions conflict. Section 2.2 refers to a test split used for decoder checkpoint selection, and Sections 2.4 and 3.3 report the best test PER. Appendix A.4.3 and Table 5 instead specify selection by validation PER. These are materially different protocols. I cannot tell which produced the augmentation results; selecting the best test checkpoint would make those results optimistic, whereas a wording error would be straightforward to correct.

The augmentation comparison also needs an update-matched control. Training is specified as 800 epochs on pN+aNpN+aN trials. If each epoch traverses that dataset, adding synthetic trials increases the optimization budget as well as the data diversity. The restricted-generator and disabled-noise controls address different questions. I would want to know how much of the gain remains when the real-only decoder is allowed the same number of optimizer updates through resampling.

Questions for authors
  1. Which split selected the checkpoints reported in Figures 4 and 6? Please reconcile the main-text descriptions with Table 5 and confirm whether the reported test set was untouched during selection.
  2. How many optimizer updates are performed for each (p,a)(p,a) condition? Does the augmentation benefit persist against real-only resampling with the same update budget and learning-rate schedule?
3 comments

We thank the reviewer for the comments and suggestions.

Q1: We selected the best test PER checkpoint, which we acknowledge leads to an optimistic estimate. We now report the PER at the last test epoch, leaving the test set untouched during selection in the augmentation experiments of the current Figure 4, 5, 6, 9-13. We have also corrected the wording error in Section 2.2, 2.4, 3.3, and A.4.3. Below is the comparison for real data fraction p and augmentatio fraction a, each cell is best checkpoint / last epoch:

Card

Linear‖TCN
p\a00.10.20.512‖0.10.20.512
0.13634/3434/3432/3230/3029/30‖32/3230/3029/3028/3029/29
0.252120/2121/2120/2120/2020/20‖19/1919/1917/1817/1716/16
0.51414/1414/1414/1413/1413/14‖13/1413/1312/1211/1111/11
0.751111/1111/1111/1110/1011/11‖11/1110/109/109/98/9

Willett

Linear‖TCN
p\a00.10.20.512‖0.10.20.512
0.14847/4746/4643/4443/4342/42‖40/4039/3938/3837/3837/37
0.253334/3433/3332/3232/3231/31‖30/3029/2927/2726/2626/26
0.52424/2525/2525/2524/2424/24‖23/2322/2221/2220/2019/19
0.752121/2121/2121/2121/2120/20‖20/2019/1919/1917/1717/17

Kunz T12

Linear‖TCN
p\a00.10.20.512‖0.10.20.512
0.17670/7269/7267/6969/7171/74‖69/7067/7266/7067/7165/69
0.258261/6362/6462/6562/6757/59‖62/6460/6558/6459/6459/62
0.56351/5452/5552/5651/5550/55‖48/5049/5348/5243/4639/41
0.754837/3842/4740/4242/4743/45‖35/3937/4031/3432/3727/30

Kunz T15

Linear‖TCN
p\a00.10.20.512‖0.10.20.512
0.17870/7364/6662/6563/6362/66‖65/6968/7067/7165/6964/69
0.257356/5753/5455/5657/5953/54‖59/6155/5953/5550/5247/49
0.57048/5048/4943/4443/4441/42‖55/5845/4642/4440/4238/38
0.755542/4337/3936/3841/4438/40‖39/4237/3935/3731/3230/32

The results are very similar and the same qualitative conclusions hold: the temporal convolutional network (TCN) provides larger augmentation gains than the linear model, with the largest gains when real data is scarce.

Q2: We consider 800 epochs with batch size of 64, which for each (p,a) leads to 800 x ((p+a)N/64) optimizer updates, where N is the number of training trials. We have added experiments considering the same update budget and learning rate schedule, and resampled real trials, while still considering the default BiT noise augmentations (Gaussian smoothing, white noise and constant offset), for the online case. Below is the comparison between Linear, TCN and resample test phoneme error rate (the new results are currently in Figure 9 of the manuscipt):

Card

Linear‖TCN‖Resample
p\a00.10.20.51248‖0.10.20.51248‖0.10.20.51248
0.13634343230302930‖32303030292929‖35363831303132
0.252121212120201818‖19191817161718‖22212020191918
0.51414141414141313‖14131211111212‖14141414131212
0.751111111110111010‖1110109999‖1111111010109
1109999998‖9988877‖9999888

Willett

Linear‖TCN‖Resample
p\a00.10.20.51248‖0.10.20.51248‖0.10.20.51248
0.14847464443424040‖40393838373837‖48474644434343
0.253334333232313030‖30292726262625‖33333332313130
0.52425252524242323‖23222220191919‖25252524232323
0.752121212121202020‖20191917171616‖21202120201920
11918191919181818‖17181616151414‖19181918181817

Kunz T12

Linear‖TCN‖Resample
p\a00.10.20.51248‖0.10.20.51248‖0.10.20.51248
0.17672726971747175‖70727071698779‖84787981808082
0.258263646567596863‖64656464626060‖82737476797684
0.56354555655555551‖50535246414439‖64666263636057
0.754838474247453932‖39403437303636‖46464141463533
12932303131312920‖31343027262118‖29272828282320

Kunz T15

Linear‖TCN‖Resample
p\a00.10.20.51248‖0.10.20.51248‖0.10.20.51248
0.17873666563667069‖69707169697465‖79787976757475
0.257357545659545759‖61595552495450‖60717271757372
0.57050494444424845‖58464442384240‖67667169647068
0.755543393844404036‖42393732323232‖64636062626054
13934373335333330‖33323135322830‖40536351423540

The TCN augmentation benefit persists for all datasets, and increases as the available data is smaller. The linear augmentation benefit persists when the available data is small, for p=0.1 Card and Willet, for all p for Kunz T15, and for p smaller or equal to 0.5 for Kunz T12.

To better visualize these results, we have added a figure to the manuscript with the improvement in PER for the different augmentation factors, for the linear, TCN and resample augmentation techniques (current Figure 5). Values above y = 0 correspond to an improvement, and values below to a deterioration. In Figure 5, the lines show the mean and the shaded areas the standard deviation across seeds.

We have revised the Abstract, Results and Discussion to report the comparison between the linear model, the TCN and the update-matched resampling control clearly. The changes in the abstract:

Finally, we demonstrate that augmenting real datasets with model generated activity improves speech decoding in low data regimes, reducing phoneme error rate by up to 30 percentage points with the TCN and 24 percentage points with the linear model relative to conventional augmentation with the same amount of resampled real data.

The other changes are in lines 373–383 (Results) and lines 489–500 (Discussion) of the updated manuscript.

As the discussion deadline approaches, we would really appreciate hearing back from you.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary

We thank the reviewers and the area chair for their work.

The reviewers highlighted the strengths of the proposed simple and interpretable compositional model of how the premotor cortex generates speech, which allows us to test mechanistic hypotheses, together with a more complex, non-linear temporal convolutional model, that generates synthetic data for augmentation. This augmentation improves speech decoding by up to 30 percentage points with the TCN and 24 percentage points with the linear model relative to conventional augmentation when real available data is scarce and reduces the amount of neural data needed by up to 48%. This is very useful for speech brain-computer interfaces, where neural recordings remain costly and difficult to collect at scale.

We summarize the main points raised from the reviewers, and describe how they have been addressed.

  • Reviewer pBrE refers to potential confounding between neural signal fidelity and downstream decoding performance. We have done the augmentation experiments, not tuning the noise gain to decoding phoneme error rate, but towards neural signal fidelity (considering the noise gain equal to 1). The augmentation benefit persists for the TCN. The linear model still improves over real data alone, but less than with the tuned gain for the larger datasets. Reviewer pBrE confirmed this point had been addressed.
  • Reviewers EFxo and pBrE raise the point that there is a lack of direct comparison with existing neural data generation methods. We have now added comparisons to the latent diffusion LDNS model. This models captures well the overall statistics of the neural population but less well speech relevant information: a decoder trained on real data reaches a much higher phoneme error rate on LDNS trials than on trials from our models. In the augmentation experiments, LDNS improves decoding when real data is very scarce, with gains similar to the linear model. However, when more real data is available, LDNS augmentation worsens decoding, while the linear model still provides benefits. Both reviewers confirmed this point has been addressed.
  • Reviewers do4s and EFxo asked us to confirm whether the reported test set was untouched during selection. We previously reported the checkpoint with the best test phoneme error rate. We now report the final checkpoint (epoch 800) throughout, so the test set plays no role in model selection, and results are essentially unchanged. Reviewer pBrE confirmed this point had been addressed, reviewer do4s has not responded unfortunately.
  • Reviewers do4s and EFxo ask if the augmentation benefit persist against real-only resampling with the same update budge. We have done the augmentation experiments, considering the same update budget and learning rate schedule, and resampled real trials, while still considering the default BiT noise augmentations (Gaussian smoothing, white noise and constant offset). The benefit of TCN augmentation persists on all four datasets and grows as less real data is available. The linear model's benefit over resampling persists when real data is scarce. Reviewer pBrE later asked us to “to acknowledge these limitations clearly and keep the conclusions appropriately scoped”, which we have done. Reviewer do4s has not responded unfortunately.
  • Reviewer EFxo has raised the point that “the evidence is restricted to two participants”. No publicly available datasets exist from additional participants with attempted speech recordings, so it is not possible to extend the analysis to more subjects. The papers in the field use the same two subjects (e.g. Zhang et al A cross species neural foundation model for end-to-end speech decoding ICLR 2026).
  • Reviewer EFxo raised the point that "robustness of the augmentation gains across repeated training runs is not fully established". Although these experiments are computationally very demanding, we have now repeated the augmentation experiments with at least three seeds for the linear model, the TCN and the resample control, and all datasets (except for two configurations for Card linear that only have two seeds). Results are consistent across seeds.

We have uploaded the revised manuscript, with changes marked in blue.

Dear Reviewer,

Thank you very much for your time and thoughtful review. As the discussion deadline approaches, could you please take a look at the authors' rebuttal and share your thoughts in the discussion?

Many thanks for your continued help!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.