- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper characterizes the singular value spectra of task gradients in EEG models. It reports that gradient energy concentrates in a few leading modes throughout training, that these leading modes carry most of the subject related gradient signal, which the authors formalize with a theorem under a latent additive subject variation model, and that tail modes have low gradient signal to noise ratio. Based on this analysis the paper proposes AGraSS, a Muon style optimizer that selects the gradient spectral head ratio once before training, rescales the selected head using the residual spectral scale, and applies additional attenuation to leading modes exhibiting consistent subject-related conflict. Across six public EEG datasets, four expert models, and three EEG foundation models, with cross-subject splits and five seeds, AGraSS outperforms AdamW, SGD, and Muon in the large majority of settings, and ablations in two settings support adaptive head selection and subject-aware shrinkage.
Critical strengths
- The pipeline from empirical gradient spectral analysis through a stylized theory to a concrete optimizer is novel and coherent. The idea of an EEG aware optimizer is new and connects naturally to current research on matrix preconditioners such as Muon.
- The evaluation is broad and disciplined for an optimizer paper. It spans 28 model and dataset combinations over seven architectures with cross-subject splits, a shared hyperparameter search, and validation based model selection, and AGraSS is best or tied best in nearly all of them according to Tables 1 and 2.
Critical weaknesses
The superiority claim is statistically under-supported. Gains over Muon are often within one standard deviation, for example 0.702±0.022 versus 0.701±0.035 on CBraMod with TUSL. Several settings favor the baselines, for example LaBraM on MentalArithmetic with 0.595 against 0.614 for AdamW. No significance tests are reported, and in several settings the fixed test splits contain only two to five subjects, for example two subjects for BCIC-IV-2a and four for Mental Arithmetic in Appendix A.2, so the small margins may not be robust.
Questions for authors
- Please report paired significance tests across seeds, and ideally across repeated subject splits, for AGraSS against Muon and against AdamW. In how many of the 28 settings is the improvement significant?
- In Table 3, the row with head shrinkage set to None and subject-aware shrinkage disabled reproduces the AdamW entries of Tables 1 and 2 digit for digit, including the standard deviations. Did this row simply reuse the AdamW runs, and which update rule does the ablation build on? For attribution, please add a variant with momentum and RMS normalization but no spectral shaping, and a Muon-style variant combined with subject-aware shrinkage, so that the contribution of each component can be isolated.
- How sensitive is AGraSS to the one-time calibration? How do the detected head ratio, the head size K_c, and the attenuation coefficients evolve during training, and how much does performance vary when calibration is run at different epochs or seeds? What is the number of power iterations used per step? Relatedly, what microbatch size and how many microbatches are used for the SNR analysis in Section 2.3? Since the estimated fluctuation across microbatches scales inversely with the batch size, the SNR profile in Fig. 3, and therefore the noise sensitivity of the tail, depends on this choice.
- Do encoders trained with AGraSS actually become more subject invariant, for example in terms of subject decoding accuracy from frozen features, than under Muon or AdamW? This would directly validate the mechanism.
- The related work cites objective and data level methods that encourage patient invariant representations, such as patient-adversarial learning (Zhang et al., 2024) and distributionally robust decoding (Duan et al., 2023), and states that optimizer level design is potentially complementary to these strategies. Can you substantiate this with an experiment, for example AGraSS combined with a patient-adversarial objective versus each alone, and can you include a within-subject condition to show that the gains of AGraSS are specific to the cross-subject setting that motivates the method?
- In Fig. 2 the largest absolute projections reach only about 0.08 to 0.30 after normalization by the Frobenius norm of the surrogate. Can you provide a null baseline for this concentration, for example the projection profile onto random orthogonal directions of matched scale, onto the singular basis of gradients computed with shuffled subject labels, or between two independent estimates of the task gradient itself? This would show whether the concentration of subject related signal in the spectral head exceeds what high-dimensional random overlap would produce, and would substantiate the strong association claim.
2 comments
Response to Reviewer VVhT (2/2)
- Subject identity decodability
We supplement the evaluation of the linear decodability of subject identity from the learned representations. We froze the encoders trained with AdamW, Muon, and AGraSS, and fitted a separate linear subject classifier to each encoder’s representations on conformer. The resulting mean test subject-classification accuracies across seeds are:
| Dataset | AdamW | Muon | AGraSS |
|---|---|---|---|
| Mental Arithmetic | 84.71% | 75.68% | 67.65% |
| FACED | 90.80% | 84.56% | 80.60% |
These results show that subject identity is less linearly decodable from representations learned with AGraSS, validating the mechanism. We have supplemented the results in Table 4 in the revised version.
- Combination with objective level method
We thank the reviewer for suggesting this experiment. We evaluated AGraSS in combination with patient-adversarial learning (PANN) [1], and conducted experiments on the CHB-MIT dataset [2] using 4-second EEG clips and a DCRNN backbone [3] following PANN. It shows that optimizer level design of AGraSS is complementary to other strategies. We have supplemented the results in Table 13 in the revised version.
| Model (Optimizer) | AUROC | F1 score |
|---|---|---|
| DCRNN (Adam) | 0.931 ± 0.017 | 0.668 ± 0.052 |
| DCRNN (AGraSS) | 0.946 ± 0.004 | 0.725 ± 0.018 |
| DCRNN+PANN (Adam) | 0.940 ± 0.013 | 0.689 ± 0.029 |
| DCRNN+PANN (AGraSS) | 0.954 ± 0.011 | 0.739 ± 0.041 |
We also provide the result under the within-subject condition. We split each subject into training, validation, and test sets, targeting a 70%/15%/15% split. We then use the training data of all subjects to train the DCRNN model. It can be seen AGraSS can also improve the performance under the within-subject condition. We’d like to clarify that our objective is not to remove subject-related information entirely or enforce complete subject invariance, which is different from PANN. Instead, AGraSS applies additional soft shrinkage to leading subject-specific gradient spectra so that subject-specific features will not dominate gradients. Thus AGraSS can encourage learning more generalizable features without sacrificing the capacity to learn useful patient-specific information under within-subject settings.
| Model (Optimizer) | AUROC | F1 score |
|---|---|---|
| DCRNN (Adam) | 0.933 ± 0.001 | 0.742 ± 0.001 |
| DCRNN (AGraSS) | 0.963 ± 0.002 | 0.807 ± 0.008 |
[1] Cross-patient automatic epileptic seizure detection using patient-adversarial neural networks with spatio-temporal eeg augmentation. Biomedical Signal Processing and Control 2024.
[2] Application of machine learning to epileptic seizure detection, in: Proceedings of the 27th International Conference on Machine Learning (ICML-10), 2010, pp. 975–982.
[3] Self-supervised graph neural networks for improved electroencephalographic seizure analysis, in: International Conference on Learning Representations, 2021.
- Null baseline for gradient spectral-head.
Thank you for this suggestion. We added a random-direction null baseline for the settings examined in Fig. 2. As shown in the table below, the observed projections of leading direction are approximately 5.5–223 times the corresponding null means across the four settings. Thus, these projections substantially exceed the average overlap with matched random directions. This comparison provides additional evidence for subject-related alignment with the leading task-gradient direction in the examined settings. We have supplemented the results in Table 7 in the revised version.
| Model / Dataset | Observed normalized absolute projection | Random-direction | Observed / Null |
|---|---|---|---|
| CTNet / Mumtaz2016 | 0.085 | 0.015 | 5.7× |
| EEGNet / FACED | 0.235 | 0.043 | 5.5× |
| Conformer / Mental Arithmetic | 0.880 | 0.018 | 48.9× |
| REVE / Mental Arithmetic | 0.290 | 0.0013 | 223.1× |
Response to Reviewer VVhT (1/2)
We sincerely thank the reviewer for appreciating our novel idea and broad evaluation, and providing insightful suggestions. We carefully address your concerns as follows.
To evaluate the statistical evidence of broad performance benefits, we perform paired statistical significance analysis on the mean balanced accuracy across all 28 settings, and the results show that AGraSS > AdamW is statistically significant (p<0.001), AGraSS > Muon is statistically significant (p<0.001), while the difference between AdamW and Muon is not significant (p=0.84). So broadly, AGraSS show robust performance gains over other optimizers. All of our experiments for data/subject split follow the common setting in the EEG literature, and the limited number of subjects is the intrinsic property of EEG datasets. Therefore, it is not common to expect a statistical significance for every setting, and instead, we consider the broad evaluation covering all 28 settings to demonstrate the overall statistical significance of the advantage of our method over AdamW and Muon. The results indicate that the gains are relatively robust in a broad range.
We thank the reviewer for identifying this ambiguity. Yes, the first row of the original Table 3 reused the AdamW runs reported in the main comparisons. We additionally add a variant that retains momentum update and RMS normalization but removes all spectral shaping. We have updated the results in the revised version and the updated results are reported below.
Table R1. Component ablations on REVE with Mental Arithmetic and EEGNet with BCIC-IV-2a.
Regarding the Muon-style variant, as standard Muon aims to flatten their singular-value spectrum, it contradicts subject-aware shrinkage.
We supplement how the Kc, head ratio and attenuation coefficient a1 evolve during training every 5 epoch in Table R2. For Conformer on Mental Arithmetic, Kc remains close to the calibration estimate of 1, head ratio is close to the calibration estimate of 0.063, recalculated attenuation coefficient a1 during training and the calibration estimate 0.557 is at the similar level. This suggests that one-time calibration can remain relatively stable.
Table R2. Re-evaluated Kc during training for Conformer on Mental Arithmetic. The calibration estimate is Kc=1.
We further supplement the performance difference when calibration is run at different epochs, as shown in Table R3. It shows broadly comparable performance across different calibration durations. We have supplemented the results in Table 11 in the revised version.
Table R3. Effect of calibration duration on AGraSS using Conformer on Mental Arithmetic. Results are test performance at the validation-selected checkpoint.
For some details of hyperparameters, we use one cached power iteration per optimization step (T=1). For the SNR analysis in Section 2.3, models use a large batch of 512, specifically, 32 microbatches of size 16.