- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes NMCNet, a multimodal framework for cross-subject classification of Mandarin initials and finals using synchronized EEG and EMG. Its central architectural component is an asymmetric interaction mechanism: Cortical Drive Gating uses EEG features to modulate EMG representations, and Feedback Cross-Modal Fusion subsequently uses the gated EMG representation to update the EEG representation. The model additionally incorporates a multi-frequency branch and a supervised contrastive objective that upweights same-class pairs from different subjects. The authors evaluate the method on a self-collected dataset of ten healthy participants performing audible and silent articulation. Experiments include three-class broad-category decoding and six within-category fine-grained tasks. The reported results are promising, especially for final classification. However, important uncertainties concerning supervised channel selection, model selection, phonetic labels, and baseline comparability currently limit confidence in the main conclusions.
Critical strengths
S1. Relevant problem and motivated architecture. The paper investigates a meaningful multimodal decoding problem and explicitly considers the different roles of EEG and EMG rather than treating them as interchangeable inputs. The asymmetric fusion mechanism provides a coherent architectural hypothesis for combining these modalities. Evaluating this hypothesis in a cross-subject Mandarin articulation setting is a worthwhile research direction. S2. The ablation studies on the fusion architecture are quite comprehensive. Beyond removing individual modules, the authors also compare reverse fusion and symmetric fusion (with parameter matching). These experiments are more insightful than simple module removal for determining whether the direction of information interaction is significant, and whether performance gains stem merely from an increase in the number of parameters. Furthermore, the single-modality control experiments provide evidence for the complementary nature of EEG and EMG data. S3. Performance improvements are notably significant for some tasks, and results are provided on a per-subject basis. As shown in the main results table, the model demonstrates a much greater advantage in classifying final rhymes than in classifying initial consonants. The per-subject results in the appendix also help determine whether the average performance has been disproportionately influenced by individual subjects. The authors’ inclusion of data reflecting variations across different tasks and subjects is a positive aspect of this experimental report.
Critical weaknesses
W1. The task definitions and the scope of the decoding claims need greater precision The fine-grained experiments appear to classify individual initials or finals within a preselected phonetic group, rather than jointly classifying all target initials or finals without knowing the group. These are different evaluation settings. Consequently, the average macro-F1 across the within-group tasks should not be interpreted as the overall performance of a complete initial/final recognition system. The manuscript should clearly state whether ground-truth group membership is assumed at evaluation time and whether a practical system would require an additional group classifier. A related concern is that the initial and final labels are derived from the same finite set of syllable trials, while the model processes a signal segment covering the full articulation. LOSO evaluation tests generalization to unseen subjects, but it does not by itself distinguish between learning compositional initial/final representations and learning whole-syllable patterns associated with the target labels. The authors should provide the complete mapping between stimuli, initials, finals, and coarse categories. A held-out syllable-combination evaluation would strengthen the claim that the learned representations generalize beyond the observed combinations. If compositional generalization is outside the paper’s intended scope, the claims should instead be limited to cross-subject classification within the evaluated stimulus inventory. W2. The phonetic grouping needs clarification In Mandarin Pinyin, j is an affricate, not a sonorant. Why is it placed in the sonorant group rather than the fricative/affricate group? Please provide clear phonetic criteria for the grouping and explain this assignment. Please also clarify whether this is a mistake in the manuscript or whether the same grouping was used in the experiments, since the latter would affect the training labels and the interpretation of the results. W3. The independent contribution of asymmetric fusion is not fully separated from the training recipe NMCNet combines asymmetric fusion, multi-frequency processing, normalization, Mixup, noise augmentation, a classification objective, and supervised contrastive learning. Although the reversed and parameter-matched symmetric variants are useful, the manuscript should more clearly document whether the external baselines use comparable inputs, preprocessing, augmentation, loss functions, and hyperparameter-search budgets. A particularly informative comparison would preserve the encoders, multi-frequency branch, losses, and training procedure while replacing CDG and FCMF with direct concatenation or a conventional gating mechanism. This would provide a cleaner estimate of the benefit attributable specifically to the proposed asymmetric interaction.
Questions for authors
Q1. Can the authors provide the complete stimulus-to-label mapping and clarify the intended scope of the fine-grained results? Are unified or cascaded results without ground-truth group membership available? Q2. What does j denote in the phonetic grouping, and does the grouping shown in Figure 2 correspond to the labels actually used for training and evaluation? Q3. Were channel ranking, channel-count selection, hyperparameter tuning, and checkpoint selection performed exclusively using training subjects in every outer LOSO fold? Which reported results correspond to fold-wise training-only channel selection?
We thank the reviewer for the most detailed assessment of this submission; the critique shaped the revision, and every experiment has been re-run under a strictly leak-free protocol.
W3. We ran the requested control. Holding the encoders, the multi-frequency branch, Mixup, Gaussian noise, the contrastive loss and the optimiser settings fixed, replacing the entire CDG+FCMF block with a plain channel-wise concatenation reduces κ from 0.396 to 0.294, and the resulting model falls below DeepConvNet (0.329); the gain is not attributable to the encoder alone, and under the same training recipe this supports the contribution of the fusion block. We also re-examined the cross-subject reweighting: κ varies by less than 0.03 across γcs = 1.0–2.0 with no consistent direction (paired γcs = 1.0 vs 1.75: +1.69 points on initials, p = 0.24; −1.92 on finals, p = 0.39). We withdraw it as a contribution and report it as a sensitivity check.
Q3. All selection now happens within the training subjects. The channel ranking for outer fold k is computed from the nine source subjects only, using four condition-specific ANOVA p-values combined by Fisher's method; k = 15 is fixed a priori; the stopping epoch is chosen by an inner five-fold split of those nine subjects, after which the model is retrained on all nine and tested once. The held-out subject contributes to no stage. Ranking stability is Jaccard 0.62 ± 0.10 (4.2× chance). Fifteen channels and all 59 give similar observed results (65.27% / κ 0.396 vs 64.47% / κ 0.393); no formal equivalence test was performed, so we describe this as similar rather than equivalent. Channel selection is reproducible but not performance-critical. The revision will include the full per-fold ranking.
W2 / Q2. The reviewer is right that j ([tɕ]) is an affricate and not a sonorant. We merged fricatives and affricates into one class, giving Plosive (b, d, g, p, t), Fricative/Affricate (zh, ch, c, sh, h, f, s, j) and Sonorant (m, n, l, y, w, zero-onset). All results are recomputed; the revised figure will show the labels used for training and evaluation.
W1 / Q1. The fine-grained results are oracle-group analyses given the true coarse category, not system-level performance, and the table will be relabelled accordingly. The complete stimulus-to-label mapping is given below; each syllable has 16 trials per participant. Category rules: Plosive = b, d, g, p, t; Fricative/Affricate = zh, ch, c, sh, h, f, s, j; Sonorant = m, n, l, y, w, zero; Monophthong = a, i, u, e, o; Diphthong = ai, ia, ui, ao, ou; Nasal coda = an, ang, eng, uan.
We do not claim generalisation to unseen onset–rhyme combinations. What the present results establish is cross-subject classification on a fixed 30-syllable inventory (two articulation modes × two phonological levels over six groups, 16 trials per syllable per participant); we have not tested whether the model generalises to a syllable whose specific onset–rhyme combination was absent from training.
Such a test is feasible with this corpus and is the natural next step: holding out a syllable such as ma leaves mi, mo and mao in training (so the onset m is still observed) and ba, da, ya (so the final a is still observed), which yields an unseen-combination test at the level of both coarse categories. We have not yet run it; a leave-one-syllable-out variant of the LOSO protocol is the immediate next step, and extending this to a combinatorial stimulus set and to open-set decoding is future work.