- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
EMG-FoundBench evaluates the transfer and adaptation of nine pretrained model configurations from seven time-series and electrophysiology model families on 20 public EMG datasets comprising approximately 1.03 million segments. It covers 19 classification datasets and one continuous typing dataset. The evaluation separates frozen linear probing, full fine-tuning, and training the same architecture from random initialization, and additionally examines held-out users, LOSO evaluation, and pooled target-cohort five-shot adaptation. Across 171 classification model-dataset pairs, full fine-tuning improves macro-F1 over linear probing by 16.4 percentage points on average, whereas pretrained initialization improves over scratch by only 3.0 points on average. Held-out-user macro-F1 decreases by 14.9 points, and five-shot adaptation recovers 6.3 points on average. Model rankings are correlated across task categories. The contribution is an evaluation resource and empirical characterization of transfer behavior, rather than a new learning algorithm.
Critical strengths
The paper presents controlled initialization comparisons. The architecture-matched pretrained-versus-scratch comparisons help distinguish the contribution of pretrained initialization from that of the model architecture under the specified training protocol. Also, data set coverage is another contribution of the paper. The benchmark consolidates heterogeneous EMG datasets and includes held-out-user and LOSO evaluations, providing a useful reference for comparing model performance across datasets and users.
Critical weaknesses
First, the present manuscript has limited scientific insight beyond an expanded empirical benchmark. The manuscript addresses a relevant evaluation gap, but the scientific insight remains limited relative to the scale of the experiments. EMGBench [6] already investigates EMG generalization and adaptation, while related EEG work [2] examines the practical value of foundation models and the distinction between frozen and fine-tuned performance.
Second, there is insufficient evidence of robustness to optimization choices. The conclusions about pretraining benefits require stronger evidence that they are robust to the downstream training procedure. Figure 2 shows substantial variation across models and datasets, including negative pretraining gains for some configurations. However, models use a shared learning rate and a fixed maximum training budget for pretrained fine-tuning and scratch training, with limited reported sensitivity analysis. Although an exhaustive hyperparameter sweep across all model-dataset combinations would be costly, targeted sensitivity analyses on representative configurations, together with repeated paired runs, would help distinguish robust initialization effects from dependence on a particular training set-up.
Third, I think that RQ4 findings lacks task diversity and interpretation. Including emg2qwerty is a reasonable way to extend evaluation beyond classification, but a single typing dataset provides limited evidence for consistency across different EMG task types.
Questions for authors
I have a few questions for the authors.
- Isn't the biggest advantage of pretraining its efficiency? There is limited discussion of this point.
- What kind of EMG information does the model use? The model can use frequency, channel, and temporal information. Masking some of this information or shuffling the temporal order of the data could help identify the main information used by the model.
- The 171 model-dataset comparisons are not independent. The NinaPro databases are related, and MOMENT and Chronos-2 each include two configurations from the same model family.
- Is pretraining useful, or are its benefits lost during fine-tuning? MOMENT-B performs worse after full fine-tuning than with LP and also performs worse than training from scratch. These results alone do not distinguish whether the pretrained knowledge is unsuitable for EMG or whether useful representations are lost during fine-tuning.
- How different is performing well on multiple datasets from reusing one representation across different settings? When models are trained separately on each dataset, evaluating them on 20 datasets does not clearly show whether a single representation can be reused across devices or tasks.
2 comments
I appreciate the authors’ substantial efforts to strengthen the paper through additional experiments and clarifications. However, my core concerns regarding the robustness of the findings under the final evaluation protocol and the depth of scientific insight remain insufficiently resolved. I therefore maintain my original rating.
Response to Reviewer gpyE
We thank the reviewer for the detailed comments. The review questioned the insight beyond prior benchmarks, the robustness to optimization choices, and the task diversity behind RQ4. We have revised the paper to address each of them: we added learning-rate, seed, and training-budget analyses (Appendix A.2.1), input-perturbation experiments (Appendix A.5), and a cluster bootstrap for dependent model–dataset comparisons (Appendix A.7), and we narrowed the RQ4 conclusions. All numbers below come from the revised results (General Response G1 and G2; 162 model–dataset pairs).
W1. Insight beyond EMGBench [6] and the EEG study [2]. EMGBench studies how EMG gesture classifiers with ImageNet-pretrained backbones adapt to new users; it does not evaluate frozen representations or matched random initialization. The EEG study [2], now cited in Section 2.2, compares foundation models with decoders of different architectures. We keep each architecture fixed and change only the initialization, which yields three findings: (i) The benefit of pretraining depends on the model: seven of nine configurations benefit on most datasets, whereas MOMENT-B and PatchTST perform better from scratch. The gain is moderately associated with fine-tuned performance (ρ = 0.65; MOMENT-S and TimesFM reach 62.1% and 62.9% with gains of 3.7 and 19.8 points), and in neither MOMENT nor Chronos-2 does the larger checkpoint gain more. (ii) The best linear-probing model is also the best fine-tuned model on 4 of 18 datasets. (iii) The aggregation should preserve which electrode a signal comes from: permuting the electrode order lowers the macro-F1 of fine-tuned MOMENT-S from 80.1 to 14.2 on GRABMyo (Appendix A.5).
W2. Robustness to optimization choices. Appendix A.2.1 analyzes three backbones whose gains differ in sign and size. Selecting the learning rate separately for fine-tuning and scratch training preserves the sign of the gain in 8 of 9 cases (Table 10). With three seeds, the gain of UniTS is positive in 11 of 11 runs and that of PatchTST is negative in 10 of 12 (Table 11). Within 50 epochs, the best checkpoint precedes the final epoch in 176 of 180 runs. These runs used an earlier assignment of subjects to Pool A and Pool B; within them, the sign of the mean gain is largely stable, and it agrees with the main results for 9 of 11 model–dataset pairs.
W3. RQ4. We agree and narrowed the conclusions to the evaluated tasks (Section 4.5): performance is highly consistent between upper- and lower-limb classification and moderately correlated with EMG-to-text decoding (ρ = 0.65 and 0.61 between the CER ranking and the macro-F1 rankings).
Q1. Efficiency. Pretrained fine-tuning reaches its best checkpoint earlier than scratch training in 62 of 90 model–dataset pairs (median epoch 15.5 vs. 23), most clearly for Chronos-2.
Q2. Information used. Appendix A.5 perturbs the test windows of fine-tuned MOMENT-S on six datasets (Table 17). Permuting electrodes reduces macro-F1 on all six. Shuffling 100-ms blocks, which keeps the signal within each block, reduces macro-F1 by 4 points or less on two datasets, by 8 points on MyPredict, and by 13 to 51 points on the other three, and time reversal reduces performance on all six. The model therefore relies on a fixed mapping between electrodes and features and, on several datasets, on temporal structure beyond 100 ms.
Q3. Dependence between pairs. A two-way cluster bootstrap (Appendix A.7) that resamples model families (MOMENT and Chronos-2 variants grouped) and dataset families (NinaPro DB1–DB7 grouped) gives 95% intervals of +11.9 [+6.7, +16.6] accuracy points for fine-tuning over linear probing, −20.1 [−32.6, −12.5] for new users, and +15.4 [+7.4, +24.3] macro-F1 points for five-shot adaptation. The interval for the pretraining gain is wide, +7.4 [+0.3, +15.0] macro-F1 and +6.4 [−0.04, +13.3] accuracy points, because the gain differs between families: it is positive for five of seven families and negative for MOMENT and PatchTST, consistent with our conclusion that the benefit is model-dependent.
Q4. MOMENT-B. Its frozen representation ranks fourth under linear probing and exceeds frozen MOMENT-S by 10.6 accuracy points, so its pretraining captures EMG-relevant information. Full fine-tuning then lowers its accuracy below linear probing on 13 of 18 datasets. This points to the adaptation step rather than to unsuitable pretrained knowledge, although our experiments do not isolate the specific cause (Section 4.3).
Q5. Representation reuse. Linear probing reuses one frozen representation across all datasets with a dataset-specific linear head, and its gap to full fine-tuning (11.9 accuracy points on average) shows how much dataset-specific adaptation adds. Transferring one fine-tuned EMG representation across devices or tasks is a separate question that our protocols do not cover.