acceptodds
Under review as a conference paper at ICLR 2027

SPD-MetaFormer Is What You Need for Small-Data Brain Decoding

Abstract

Brain signal decoding is challenging because neural recordings are noisy and vary across individuals, while labeled data are often limited. Recent attention-based models on the symmetric positive definite (SPD) manifold have nevertheless achieved strong performance using covariance and connectivity representations, yet the contribution of learned token weighting remains unclear. We examine two representative architectures, MAtt (based on log-Euclidean geometry) and GBWAtt (based on generalized Bures–Wasserstein geometry), and find that their learned attention weights remain close to uniform after training. We trace this behavior to bounded similarity parameterizations that, under the original softmax scaling, limit attention-weight contrast. Moreover, replacing learned weights with uniform weights, either throughout training or post hoc in fully trained models without further optimization, has little effect on mean predictive performance while preserving each model’s original aggregation geometry. Motivated by these findings, we introduce SPD-MetaFormer, an attention-free architecture built on uniformly weighted Fréchet aggregation under log-Euclidean geometry. Its backbone combines geodesic residual updates and spectral feedforward transformations to refine a summary token, while retaining memory tokens for a learned weighted readout. Token states remain SPD-valued until tangent-space classification. Across three EEG benchmarks, SPD-MetaFormer matches or improves upon the performance of strong Euclidean and manifold baselines, including the original and uniform-weight variants of MAtt and GBWAtt. These results suggest that, in the short-sequence and limited-data regimes studied, carefully designed SPD architectures can provide a simpler and effective alternative to adaptive manifold attention.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Geometric deep learning, Symmetric positive definite matrices, Riemannian pooling, EEG decoding, Brain-computer interfaces

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CnkAOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SPD-MetaFormer, an attention-free architecture built on uniformly weighted Fréchet aggregation under log-Euclidean geometry. Its backbone design combines geodesic residual updates and spectral feedforward transformations while retaining memory tokens for a learned weighted readout. The motivation, based on the observation that replacing these weights with uniform weights during both training and evaluation has little effect on mean predictive performance, is clear. The model design is incremental; the proposed model largely combines MetaFormer and SPD neural networks. Results on three EEG benchmarks: SPD-MetaFormer matches or improves upon the performance of strong Euclidean and manifold baselines.

Critical strengths

S1: The paper asks a motivated question about whether adaptive weighting is actually responsible for the gains of existing SPD manifold-attention models

S2: The paper is well written and easy to follow; the experimental analysis part is complex.

Critical weaknesses

W1: The motivation is that replacing these weights with uniform weights during both training and evaluation has little effect on mean predictive performance, but SPDMeta has many architectural designs that are different from Matt and GBWAtt. Given the performance gain is marginal, it is hard to interpret whether it is the uniform mixing that provides gains.

W2: The model novelty is moderate, as the proposed model largely combines MetaFormer and SPD neural networks. The main EEG comparison against prior methods relies on published numbers rather than matched reproductions, making the marginal improvements difficult to interpret.

W3: The experiment's SOTA comparison is not solid. In the manifold literature, it does not compare recent advances in hyperbolic batch normalization for EEG decoding [1] and recent high-impact EEG foundation models [2][3].

[1] Li et al. "HEEGNet: Hyperbolic embeddings for EEG." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Jiang, Wei-Bang, Liming Zhao, and Bao-Liang Lu. "Large brain model for learning generic representations with tremendous EEG data in BCI." International Conference on Learning Representations. Vol. 2024. 2024.

[3] El Ouahidi, Yassine, et al. "REVE: A foundation model for EEG-adapting to any setup with large-scale pretraining on 25,000 subjects." Advances in Neural Information Processing Systems 38 (2026): 22541-22577.

Questions for authors

Can you justify whether the gain is from uniform mixing, given that there are differences between architectures?

What are SPDMetaFormer's results compared with SOTA models in W3?

Why do you investigate adaptive weighting only for SPD-manifold models? Would we observe a similar phenomenon for any attention-based EEG decoding models?

4 comments
AuthorsOpenReview ↗

Response to Reviewer CnkA (1/2): W1&Q1; W2

Thank you for your careful reading and constructive comments, and for recognizing the motivation of our study. Your comments prompted several new experiments, including a matched-tuning comparison and a comparison with HEEGNet and recent EEG foundation models. All results are in the revised paper, and we summarize the key findings here.

W1&Q1.

Thank you for the opportunity to clarify our claim: uniform mixing is not the primary source of the observed performance gains; rather, it serves as a motivating principle for our architectural design. We isolate uniform mixing from the architecture by testing it inside MAtt and GBWAtt, where only the weights change. SPD-MetaFormer then tests whether a model built on uniform mixing can be competitive. Uniform mixing is performance-neutral: it matches learned attention with a simpler and cheaper model, and the margin comes from the SPD path built around it. We examine the two parts separately:

  1. The learned weights are nearly uniform, so attention acts as a geometric average. With each architecture kept identical, the learned weights of MAtt and GBWAtt are nearly uniform (normalized entropy > 0.998), and locking them to uniform changes mean scores by at most 0.36 points (Tables 4–5; robustness checks in App. B and F).
  2. SPD-MetaFormer is designed from this interpretation, not assembled from uniform mixing and a MetaFormer template. The composition of our model is not trivial: with identical tokens and uniform mixing, the direct combination, a Euclidean MetaFormer, is significantly worse on all three datasets (Table 2). Each component instead follows from this interpretation and is further supported by ablation:
    • Summary and memory token for performance (Sec. 3.3). A uniform mean gives a single output shared by all positions, so it updates a separate summary while the memory tokens are kept. Reading out from the memory tokens alone, without the summary and the block, lowers the score on all three datasets (Table 2), as does reading out from the summary alone (Table 3).
    • FFN for robustness (Sec. 3.4). The FFN is a one-scalar spectral shrinkage shared by all tokens, because averages in log coordinates are sensitive to small eigenvalues and tokens from short windows are ill-conditioned. With it, MI accuracy stays within 1.2 points from 3 to 36 tokens, compared with 9.2 points without it (App. D).
    • Readout for regularization (Sec. 3.5). Since tokens can be combined with input-independent weights, the readout is also a geometric average. It outperforms a concatenation readout by 5.13 and 9.06 points on MI and SSVEP (Table 3), even when that head is regularized and tuned (App. I).

Uniform mixing also makes the model interpretable, attention-free, and linear in the number of tokens (App. K). In the revision, we therefore present the attention diagnosis as the key finding and SPD-MetaFormer as the design that follows from it.


W2.

Table 1 follows prior work on these benchmarks (e.g., Jin et al., 2026) in comparing with published scores on the same splits. To verify this comparison, App. H (Table 15) adds a matched comparison with MAtt and GBWAtt, the original SPD-attention model and the strongest baseline of Table 1. All three models share the same data splits, the same tuning budget of 10 configurations, and validation-only selection. The matched results lead to the same conclusion as Table 1, and the gains are not marginal where they matter: SPD-MetaFormer is significantly ahead of all baselines on SSVEP, by 3.8–5.7 points, and of MAtt on ERN, by 3.1–3.3 AUC points (pp < 0.001). It has the highest mean on MI and is on par with GBWAtt on ERN, and no baseline is significantly better on any dataset. Regarding novelty, please see point 2 under W1.

AuthorsOpenReview ↗

Response to Reviewer CnkA (2/2): W2; W3&Q2; Q3

W2.

Table 1 follows prior work on these benchmarks (e.g., Jin et al., 2026) in comparing with published scores on the same splits. To verify this comparison, App. H (Table 15) adds a matched comparison with MAtt and GBWAtt, the original SPD-attention model and the strongest baseline of Table 1. All three models share the same data splits, the same tuning budget of 10 configurations, and validation-only selection. The matched results lead to the same conclusion as Table 1, and the gains are not marginal where they matter: SPD-MetaFormer is significantly ahead of all baselines on SSVEP, by 3.8–5.7 points, and of MAtt on ERN, by 3.1–3.3 AUC points (pp < 0.001). It has the highest mean on MI and is on par with GBWAtt on ERN, and no baseline is significantly better on any dataset. Regarding novelty, please see point 2 under W1.


W3&Q2.

SPD-MetaFormer obtains the highest mean on all three datasets against HEEGNet [1], LaBraM [2] and REVE [3], and also against CBraMod [4], another widely used EEG foundation model.

Table R1. SPD-MetaFormer against HEEGNet and EEG foundation models (10 seeds; App. J, Table 17 in the paper). Accuracy (%) for MI/SSVEP, AUC × 100 for ERN, as mean ± SD across runs (averaged over subjects). *: significantly below SPD-MetaFormer (paired t-test and Wilcoxon test, pp < 0.05).

ModelMI (acc.)SSVEP (acc.)ERN (AUC)
HEEGNet [1], session-adaptive65.66 ± 6.98*55.15 ± 6.98*82.02 ± 2.34
HEEGNet [1], inductive61.96 ± 5.14*52.15 ± 6.73*81.73 ± 2.40
LaBraM-Base [2]49.17 ± 3.51*41.09 ± 4.77*66.62 ± 5.56*
REVE-Base [3]66.68 ± 2.58*34.09 ± 4.59*73.77 ± 2.69*
CBraMod [4]51.63 ± 2.52*39.19 ± 4.25*73.09 ± 2.65*
SPD-MetaFormer75.44 ± 1.6970.79 ± 2.1782.53 ± 1.85

All baselines are trained under the protocol of Table 1 (one model per subject), using their official code and, for the foundation models, the released pretrained weights. Each is tuned on validation data, starting from its published recipe. The low SSVEP scores of the foundation models are consistent with a linear probe of their frozen features, which reaches only 21.7–25.5% on this five-class task (chance: 20%; App. J). Kontras et al.[5] similarly report that frozen REVE and NeuroLM "remain near chance level" on other SSVEP datasets. This broader comparison further supports the paper's finding.


Q3.

Why SPD-manifold models. Recent work has proposed more sophisticated attention on the SPD manifold for EEG decoding (Pan et al., 2022; Hu et al., 2025; Jin et al., 2026) and has credited the improvements to it, whereas the contribution of the learned weights had not been examined. We therefore focus on this line of work, where the question is most directly tied to published claims.

A similar finding outside the manifold. A concurrent ICLR submission, Does Attention Matter in Motor Imagery EEG Decoders? Diagnosing and Repairing Attention Failure [6], examines eight Euclidean convolution–attention decoders on four motor-imagery datasets and finds that their predictions do not depend consistently on the attention weights. Our MI results agree with it, and our results on two further paradigms, SSVEP and ERN, give additional support (Tables 4–5).

Future direction. Whether this holds for attention-based EEG decoders in general is beyond the scope of this paper, but it is a question we would very much like to investigate in future research. Our two controls, the entropy of the weights and the locked-uniform comparison, do not depend on the geometry, so they carry over directly to other decoders, and we plan to take this up as our next step.

Thank you again for the careful and constructive review, which has made the paper clearer and its evidence much broader. We hope that these responses address your concerns, and we would be grateful if you would consider raising your score. We are glad to clarify any remaining point.

References

[4] Wang, Jiquan, et al. "CBraMod: A criss-cross brain foundation model for EEG decoding." International Conference on Learning Representations, 2025.

[5] Kontras, Konstantinos, et al. "NeuroAtlas: Benchmarking foundation models for clinical EEG and brain-computer interfaces." arXiv:2605.14698, 2026.

[6] Anonymous. "Does Attention Matter in Motor Imagery EEG Decoders? Diagnosing and Repairing Attention Failure." Under review at ICLR 2027. https://openreview.net/forum?id=VcgYbGuRnf

I thank the author for the detailed rebuttal, which has addressed most of my concerns. Looking at the results, I am wondering why recent high-impact foundation models have such a large performance drop, especially on SSVEP tasks, and cbramod is usually the state-of-the-art on motor imagery datasets, but here we see a huge drop.

So far, the authors have not submitted the revised manuscript; I would also like to see how the authors have revised the manuscript based on all reviewers comments.

AuthorsOpenReview ↗

Response to Reviewer CnkA: W3&Q2

Thank you for the quick follow-up. We are glad most of your concerns are addressed.

The gap comes from the evaluation regime, and our results agree with published ones. Foundation models are usually fine-tuned on pooled data from many subjects, whereas Table 1, like prior manifold-attention work, trains one model per subject on a few hundred trials. App. J of the revised manuscript details the settings, and we summarize the key points here because the numbers can be confusing without them.

Independent work agrees. Bdeir et al. [7], accepted at NeurIPS 2026, fine-tune CBraMod under the same per-subject protocol and report 45.22% / 40.45% / 68.49 on MI / SSVEP / ERN (their Table 9), against our 51.63% / 39.19% / 73.09. They likewise find that foundation models "underperform on these smaller EEG datasets".

MI. On BCIC-IV-2a, the CBraMod [4] (their Table 15) reports 51.38% for CBraMod and 48.69% for LaBraM across subjects, and we obtain 51.63% and 49.17% per subject, the same level despite the different protocols.

SSVEP. These models read each channel as 1-s patches and a trial here lasts 1 s, so the pretrained temporal modelling goes unused, which Bdeir et al. [7] also note for CBraMod. A linear probe on the frozen features is accordingly near chance (21.7–25.5% vs. 20%), as in Kontras et al. [5] (NeurIPS 2026).

We have double-checked that the revised PDF is uploaded, with the fine-tuning and tuning settings of each foundation model in App. J. Could you kindly check, by clicking the pdf file sign? If it is still not visible, we will ask the PCs to resolve it.

References.

[7] Bdeir, Ahmad, et al. "LAtte: Hyperbolic Lorentz attention for joint-subject EEG classification." Advances in Neural Information Processing Systems, 2026.

Reviewer_VHR9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper diagnoses near-uniform attention in MAtt and GBWAtt through matched uniform-weight interventions, entropy measurements, and temperature controls. It proposes SPD-MetaFormer, combining uniform log-Euclidean mixing, summary refinement, spectral shrinkage, and static weighted readout. Evaluation covers three EEG benchmarks and ABIDE fMRI.

Critical strengths
  1. Well-controlled attention diagnosis. Ten-seed uniform-weight interventions, entropy measurements, and temperature controls consistently support limited benefit from adaptive weighting in the tested settings.
  2. Useful spectral-shrinkage evidence. Appendix D isolates improved robustness to finer covariance tokenization, providing a concrete practical justification for this inexpensive component.
Critical weaknesses
  1. Architectural superiority remains insufficiently established. Table 1 uses published EEG scores rather than matched reproductions; the separate attention interventions do not resolve this comparison. ABIDE uses shared splits, but its 0.24-point AUC lead over SPDNet-BN lacks paired uncertainty establishing a reliable advantage.
  2. Readout gains are not fully attributed. Equations 12–13 equal weighted pooling of log-token features, which a linear concatenation head can represent. This remains geometrically meaningful, but the concatenation comparison does not separate pooling structure from regularization and optimization. The arithmetic-pooling control supports a task-dependent geometric benefit, not a general explanation of all gains.
Questions for authors
  1. Does SPD-MetaFormer outperform learned and uniform MAtt/GBWAtt under matched splits and comparable tuning budgets?
  2. Does the readout advantage persist against a suitably regularized concatenation head? Please explicitly state its weighted log-space pooling equivalence.
  3. Does learned shrinkage outperform a validation-selected fixed coefficient?
3 comments
AuthorsOpenReview ↗

Response to Reviewer VHR9 (1/2): W1 & Q1

Thank you for three insightful questions. We ran every experiment you suggested. The additional experiments addressing Q1 and Q2 are detailed in Appendices H and I of the revised manuscript; we summarize the key observations and conclusions below. Our full response to Q3 is provided in Part 2 of our reply.

W1 & Q1.

Yes: on SSVEP against all four baselines (+3.8 to +5.7 points) and on ERN against both MAtt variants (+3.1 to +3.3 AUC points). On MI and against GBWAtt on ERN, we detect no statistically significant difference (Table 15 below). Without any attention, SPD-MetaFormer wins 6 of the 12 comparisons, loses none, and its SPD block is faster than the baselines’ attention modules on all three datasets (Table 18, Appendix K).

Splits were already matched; only tuning was not. In the original submission, Table 1 compares SPD-MetaFormer with published baseline scores on the same train/test splits, but it does not control the tuning budget, and you are right that it therefore does not settle the comparison. The revised paper adds a new experiment (Appendix H, Table 15) that closes this gap: all five models share the splits and one tuning grid that contains the official setting of both baselines, with validation-only selection, fresh seeds for the reported runs, and a protocol fixed before any run.

Table 15. Δ = SPD-MetaFormer − baseline in points, paired over subjects, with 95% subject-bootstrap intervals. Bold: p-value <0.05<0.05 (uncorrected paired t-test).

BaselineMI (acc.)SSVEP (acc.)ERN (AUC)
MAtt, learned+0.66 [−0.42, +1.78]+5.48 [+2.38, +8.99]+3.33 [+1.85, +4.73]
MAtt, uniform+0.94 [−0.18, +2.09]+5.71 [+3.15, +8.48]+3.12 [+1.74, +4.35]
GBWAtt, learned+1.05 [−0.62, +2.97]+3.82 [+0.95, +7.12]+0.33 [−1.13, +1.76]
GBWAtt, uniform+1.21 [−0.43, +3.16]+4.51 [+1.62, +7.88]−0.29 [−1.23, +0.51]

We also verified that these gains do not come from tuning. When we give each baseline its best configuration by test score, while SPD-MetaFormer keeps its validation-selected one, SPD-MetaFormer still leads by 3.2–5.5 points on SSVEP and by 3.1 AUC points over MAtt on ERN, and all six differences remain statistically significant (95% bootstrap intervals; Appendix H).

ABIDE by W1. We agree, and we do not claim an advantage here. The AUC lead over SPDNet-BN is small relative to the split-to-split variation (Table 11). We have revised the text accordingly: Section 4.1 and Appendix C now state that SPD-MetaFormer is competitive with the strongest SPD baseline (SPDNetBN) and the connectome-specific BrainNetCNN. ABIDE is meant to show that the unchanged backbone transfers to fMRI connectivity, not that it is the best method there.

AuthorsOpenReview ↗

Response to Reviewer VHR9 (2/2): W2 & Q2; Q3

W2& Q2.

Answer. Yes. With ridge or dropout tuned on validation data, the pooled readout still leads the concatenation head by 3.4–3.7 points on MI and by 7.7–8.5 points on SSVEP (Table 16 below). On ERN, the two heads were statistically indistinguishable in Table 3 and remain so.

Equivalence (new Appendix I, Eq. 20). The revised paper now states this explicitly: the new Appendix I (Eq. 20) derives from Equations 12–13 that the readout is weighted log-space pooling followed by one shared linear classifier, i.e., the concatenation head with its per-token weight blocks tied to that classifier (Wj=πjWhW_j=\pi_jW_h).

We agree with both points in W2. (i) The pooled readout is a concatenation head with tied per-token weights, so its gain comes from this tying constraint, not from added capacity. (ii) Log-space pooling beats arithmetic pooling clearly on SSVEP but only negligibly on MI (Table 3), so it is a task-dependent benefit, not a general explanation of the gains.

Table 16. Concatenation head with ridge or dropout strength selected on validation trials (setup in Appendix I). Parentheses give concatenation − full model, paired over subjects, with 95% subject-bootstrap intervals.

Concatenation headMI (acc.)SSVEP (acc.)ERN (AUC)
Original (Table 3)70.31 (−5.13 [−6.31, −3.99])61.73 (−9.06 [−13.71, −4.94])81.83 (−0.69 [−1.50, +0.13])
Tuned ridge72.03 (−3.41 [−4.38, −2.35])62.30 (−8.49 [−12.80, −4.75])82.73 (+0.21 [−1.07, +1.47])
Tuned dropout71.76 (−3.68 [−4.77, −2.62])63.13 (−7.66 [−11.85, −3.85])82.35 (−0.18 [−1.46, +1.08])
Best setting by test score73.59 (−1.85 [−3.57, +0.08])63.23 (−7.56 [−10.85, −4.79])82.96 (+0.43 [−0.32, +1.16])
Full model (pooled readout)75.4470.7982.53

Tuned regularization recovers only a fraction of the gap on MI and SSVEP, and even the best setting by test score stays below the full model on both. Tuned ridge and dropout do not substitute for tying: they shrink the blocks WjW_j, whereas pooling ties them to one shared classifier.


Q3.

Answer. We find no statistically significant advantage from learning ρ over a validation-selected fixed ρ in any of the six settings (Table R3 below).

Experiment. We freeze ρ at one of seven values from 0 to 0.5, change nothing else, and keep, for each subject and seed, the fixed ρ with the lowest validation loss (the checkpoint-selection criterion). We run all subjects with ten seeds each, at both the main tokenization of Table 1 and the pre-specified one of Appendix D. The protocol was fixed before any run.

Table R3. Accuracy (%) for MI and SSVEP, AUC (%) for ERN. Difference = learned − fixed, with its 95% subject-bootstrap interval and paired t-test p-value. Learned-ρ scores are the Appendix D reruns, which are within 0.11 points of Table 1.

DatasetTokenizationLearned ρFixed ρDifference [95% CI]p-value
MImain (K = 3)75.4575.76−0.31 [−0.57, −0.02]0.07
MIApp. D (K = 18)74.2674.42−0.15 [−1.03, +0.58]0.74
SSVEPmain (K = 7)70.9070.23+0.67 [−0.25, +1.62]0.21
SSVEPApp. D (K = 4)66.6266.10+0.52 [−0.94, +1.97]0.51
ERNmain (K = 1)82.5182.72−0.21 [−0.53, +0.11]0.24
ERNApp. D (K = 6)80.9980.87+0.12 [−0.87, +1.13]0.82
  1. Shrinkage matters more than its exact strength. Setting ρ = 0 costs up to 7.76 points (see also Appendix D), whereas scores vary by at most 1.7 points for ρ between 0.025 and 0.1.
  2. A single constant performs comparably. ρ = 0.05 for all tasks, which matches the median learned ρ, scores within 0.3 points of the learned ρ or above it in all six settings. We keep the learned ρ as a convenience: it needs no grid search.

Thank you again for these questions. We would be glad to run any further analysis you would find useful, and if you find that your concerns have been addressed, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal to Reviewer VHR9

Thank you again for your constructive feedback! We just wanted to kindly follow up on our rebuttal and revised manuscript. We have added matched baseline comparisons, regularized readout experiments, and fixed-shrinkage controls in response to your questions.

We would really appreciate hearing whether these new results address your concerns. We would be happy to clarify or discuss any remaining questions!

Thank you again for your time and consideration!

Reviewer_efGUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the necessity of incorporating SPD attention mechanism into brain decoding tasks. The authors observe that the attention weights learned by MAtt and GBWAtt tend to be uniform and propose SPD-MetaFormer, an attention-free SPD backbone that separates local covariance memories from a global summary based on the LEM. Experiments on three EEG benchmark datasets and one fMRI dataset validate the effectiveness of the SPD-MetaFormer.

Critical strengths

This paper observes that the weights generated by existing SPD attention mechanisms are nearly uniform. Rather than introducing another complex attention mechanism, this work offers a new perspective on the geometric aggregation for brain decoding. Besides, this paper is well-motivated.

Critical weaknesses
  1. The primary methodological contribution of this paper appears to lie in the integration and adaptation of existing SPD operations, rather than the introduction of a novel geometric learning mechanism. Most components of SPD-MetaFormer are based on the established mathematical operations, such as Fréchet mean, geodesic interpolation, and covariance shrinkage, the overall architecture follows the existing MetaFormer design paradigm.

  2. The proposed method is based on the LEM, where Fréchet aggregation and geodesic interpolation correspond to linear operations within the matrix logarithm. The "geometric readout" operation introduced in the paper followed by a logarithmic projection is equivalent to computing a weighted average of SPD tokens after they have been mapped into the logarithmic space. It remains unclear whether the observed performance gains from the proposed manifold-processing architecture itself, or simply from the fact that the operations are performed within a Log-Euclidean coordinate system.

Questions for authors
  1. The near-uniform attention phenomenon is primarily demonstrated on MAtt and GBWAtt with relatively short token sequences. Would the same observations hold for alternative attention scoring functions or longer token sequences?

  2. Since Log-Euclidean Fréchet aggregation and geodesic interpolation correspond to linear operations in the matrix-logarithm, how does the SPD-MetaFormer perform under other SPD geometries, such as the Bures–Wasserstein or Affine-Invariant geometries?

  3. The experiments are conducted on small-scale brain decoding datasets (although the paper's title already specifies a small dataset). Authors need to evaluate SPD-MetaFormer on larger datasets to demonstrate its scalability and generalization capability.

5 comments
AuthorsOpenReview ↗

Response to Reviewer efGU (1/4): W1

Thank you for your thoughtful review and for your positive comments on our observation of near-uniform SPD attention as a new perspective on geometric aggregation. We have added new experiments to address your concerns, which we believe strengthen the paper's conclusions. All new results are included in the revised PDF; for convenience, Tables R1–R5 below reproduce the key numbers.

W1. We agree that the individual operations in our architecture are classical. Our contribution is twofold: (i) the finding that, in this regime, SPD attention effectively computes a uniform mean (Sec. 4.4), and (ii) an architecture built around this finding. The design is simple by intent, but assembling it is not trivial: simpler combinations perform worse. A Euclidean MetaFormer on the same tokens underperforms on all three datasets (Table 2), and the performance of variants that keep only the local memories or only the global summary are also worse on every dataset (Tables 2 and 3). Each component below is supported by ablations, and the role-separated tokens and the spectral FFN are new relative to the standard MetaFormer template.

  • The gains do not depend on log-linearity. Following your suggestion (Q2), we added a new experiment under Affine-Invariant geometries (AIRM), where means and geodesics are no longer log-linear, and the architecture retains its accuracy (App. E).
  • Role-separated tokens. Local covariance tokens are kept unchanged as evidence, and only a separate global summary token is updated, through a gated geodesic step. Removing either role lowers performance on every dataset (Tables 2 and 3).
  • A spectral FFN for robustness. In place of the MLP, one learned scalar shrinks each token's spectrum, which preserves positive definiteness and provably cannot increase the condition number (Sec. 3.4). This keeps accuracy stable as tokens become finer and covariance estimates noisier (App. D).
  • A structurally regularized readout. Geometric pooling passes all tokens through one shared classifier. A concatenation head with tuned ridge or dropout does not match it on MI and SSVEP (Sec. 4.3, App. I).

Together, these choices make the attention-free model more accurate and more robust in the small-data regime, as well as computationally more efficient (App. K).

AuthorsOpenReview ↗

Response to Reviewer efGU (2/4): W2 & Q2

W2 & Q2. The gains indeed come from the architectural design rather than merely from operating in the log-coordinate space. To clarify and substantiate this point, we include the following two controlled studies in the revised manuscript.

(a) Log coordinates without the architecture: accuracy drops (Section 4.2, Table 2).

  • A linear classifier on the same log-vectorized tokens ("SPD tangent readout") underperforms the full model on all three datasets.
  • A Euclidean MetaFormer on the same tokens also underperforms on all three datasets (Table 2).

(b) The architecture without log-linearity: accuracy holds (Appendix E; Table 12, reproduced below as Table R1). We retrained SPD-MetaFormer with all four metric-dependent operations (mixer mean, gated geodesic, readout mean, tangent map) replaced by their AIRM or BW counterparts.

  • AIRM matches LE on all three datasets (∣Δ∣≤0.37|\Delta| \le 0.37 points, all p≥0.099p \ge 0.099), although its Karcher mean and geodesic are not linear in the logarithm.
  • BW matches LE on MI, slightly underperforms on ERN, and clearly underperforms on SSVEP. The SSVEP gap comes from BW’s square-root tangent map, not from BW aggregation. SSVEP tokens are estimated from only 1.2 samples per dimension, and the square root compresses the small-eigenvalue directions that the logarithm resolves.

Table R1. SPD-MetaFormer under different geometries (10 seeds; Table 12 in the paper). Accuracy (%) for MI/SSVEP, AUC × 100 for ERN, as mean ± SD across runs (averaged over subjects). Δ rows: paired difference from LE [95% subject-bootstrap CI], with paired t-test pp-value.

GeometryMISSVEPERN
LE75.44 ± 1.6970.79 ± 2.1782.53 ± 1.85
AIRM75.81 ± 1.5370.44 ± 2.3182.61 ± 1.84
BW75.05 ± 1.8864.55 ± 2.4381.09 ± 2.02
AIRM − LE+0.37 [+0.02, +0.74], pp = 0.099−0.35 [−1.65, +1.00], pp = 0.62+0.09 [−0.19, +0.34], pp = 0.56
BW − LE−0.39 [−1.59, +1.03], pp = 0.60−6.25 [−9.68, −2.97], pp = 0.006−1.43 [−3.07, −0.00], pp = 0.098

Conclusion. Log coordinates alone are not sufficient, and log-linearity is not necessary. We keep LE because it reaches the same accuracy as AIRM with closed-form operations, whereas AIRM's iterative Karcher mean takes 1.6–2.7× the training time (Appendix E).

Clarification on the equivalence. You are right that under LE, the geometric readout followed by the logarithm is a weighted average of log-tokens; we now state this explicitly in Appendix I (Equation 20). The backbone as a whole, however, is not linear in log coordinates, because two of its operations act on the matrices themselves: the congruence maps with eigenvalue rectification (Section 3.3, Equation 6) and the spectral shrinkage (Section 3.4, Equation 9).

AuthorsOpenReview ↗

Response to Reviewer efGU (3/4): Q1

Q1. Longer sequences: yes, near-uniform attention is also observed in longer token sequences (Table R2; App. F). Using finer temporal windows, we retrained the released MAtt and GBWAtt with up to 36 tokens (originally 3, 7, and 3 on MI/SSVEP/ERN), with both learned and uniform weights.

  • Attention stays uniform: normalized entropy is ≥ 0.9988 in all 40 settings, at up to 12× the original length.
  • Removing the learned weights has marginal effect: substituting 1m\frac{1}{m} at test time changes scores by at most 0.27 points.
  • Training with uniform weights gives the same accuracy in 36 of 40 settings, including all 20 for GBWAtt; in three of the remaining four, uniform training is better.

Table R2. Longer token sequences (3 seeds; paper Table 13). Min H/log⁡mH/\log m: lowest normalized attention entropy (1 = uniform). Swap: largest score change when the learned weights are replaced by 1m\frac{1}{m} at test time. Last column: settings where uniform and learned training differ at p<0.05p < 0.05 (uniform − learned, points). † Rank-deficient tokens (mm = 27, 36).

Model / datasetmmMin H/log⁡mH/\log mSwapp-value < 0.05
MAtt / MI3–360.999990.042/7 (+7.65†, +9.77†)
MAtt / SSVEP2–140.999990.061/7 (+1.15)
MAtt / ERN3–240.999960.031/6 (−2.50)
GBWAtt / MI3–360.999810.240/7
GBWAtt / SSVEP2–140.999560.270/7
GBWAtt / ERN3–240.998870.030/6

Alternative scoring functions show different patterns: one keeps attention near-uniform; the other two make it selective, but then perform worse than uniform weights (Table R3; App. B.2). We replaced the scoring function in both models with the three alternatives from GBWAtt's own ablation (Jin et al., 2026), at unit temperature.

  • Bounded (Gaussian): still uniform. Normalized entropy ≥ 0.990, and no significant difference from uniform weights (at most 0.83 points).
  • Unbounded (−d2-d^2, inner product): selective, but worse. Entropy falls to 0.04–0.76, and uniform weights are better in all 12 settings (significantly in 11). The temperature sweep agrees: every setting with entropy below 0.8 scores below uniform (App. B.1).

Table R3. Alternative scores at unit temperature (10 seeds; paper Table 10). HH: mean normalized entropy (1 = uniform). Δ\Delta: uniform − learned, in points (Δ>0\Delta > 0: uniform better); * p<0.05p < 0.05.

Model / datasetGaussian HHGaussian Δ\Delta−d2-d^2 HH−d2-d^2 Δ\DeltaInner prod. HHInner prod. Δ\Delta
MAtt / MI1.0000−0.010.22+12.82*0.19+16.94*
MAtt / SSVEP1.0000−0.350.06+10.15*0.07+13.64*
MAtt / ERN1.0000−0.830.10+3.460.11+7.51*
GBWAtt / MI0.9993−0.020.76+4.50*0.08+15.83*
GBWAtt / SSVEP0.9973−0.320.55+11.70*0.04+25.13*
GBWAtt / ERN0.9905−0.570.53+4.96*0.07+7.89*

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Dear AC, SAC, and PCs:

We sincerely appreciate your efforts in making ICLR a success, and also thank Reviewers efGU, VHR9, and CnkA for their careful and constructive reviews.

Rating update after the rebuttal

  • Reviewer CnkA: 2 → 3. Reviewer CnkA wrote that our rebuttal "has addressed most of my concerns" and, after a follow-up discussion on the foundation-model comparison (App. J), raised their rating from 2 to 3.
  • Reviewers efGU and VHR9. We have not yet received a response to our rebuttal, and we would be grateful to know whether the new experiments below address their concerns.

We have responded to every concern raised in the reviews and added the corresponding experiments (App. B.2 and App. E–L). The new results consistently support both our finding, that the learned weights of manifold attention are nearly uniform, and the proposed SPD-MetaFormer. Below we list the strengths the reviewers noted, and then map each concern to the reviewers who raised it, the new result, and its location in the revision. Details are in the individual responses.

Strengths recognized by the reviewers

We are encouraged that the reviewers found the question well-motivated and the diagnosis well-supported:

  • A new perspective (efGU). "Rather than introducing another complex attention mechanism, this work offers a new perspective on the geometric aggregation for brain decoding."
  • A well-motivated question (CnkA, efGU). "The paper asks a motivated question about whether adaptive weighting is actually responsible for the gains of existing SPD manifold-attention models" (CnkA), and "this paper is well-motivated" (efGU).
  • A well-controlled diagnosis (VHR9). Ten-seed uniform-weight interventions, entropy measurements, and temperature controls "consistently support limited benefit from adaptive weighting in the tested settings."
  • A practical, inexpensive component (VHR9). App. D provides "a concrete practical justification" for the spectral shrinkage.
  • Clear presentation (CnkA). The paper is "well written and easy to follow."

New experiments

ConcernRaised byResultWhere
Longer token sequencesefGU Q1Attention stays near-uniform in all 40 settings tested, up to 36 tokens.App. F
Alternative score functionsefGU Q1The bounded Gaussian score stays near-uniform. The unbounded scores are selective but worse than uniform weights.App. B.2
Other Riemannian geometriesefGU W2/Q2AIRM matches LE on all three datasets, so the model does not rely on log-linearity. BW is 6.25 points lower on SSVEP, which a control traces to its square-root tangent map.App. E
Scalability and generalizationefGU Q3On HBN-EEG (2,859 subjects), accuracy on unseen subjects rises with every increase in training subjects. On citation graphs, the uniform mixer is competitive with learned SPD attention. The block is linear in sequence length and 2.3–8.4× faster than the attention modules.App. L, G, K
Comparison under matched tuningVHR9 W1/Q1, CnkA W2Significant gains on SSVEP and over MAtt on ERN. No baseline is significantly better on any dataset.App. H
ReadoutVHR9 W2/Q2, efGU W2The log-space pooling equivalence is now explicit (Eq. 20), and the readout still leads a regularized concatenation head on MI and SSVEP.App. I
Learned vs. fixed shrinkageVHR9 Q3No significant difference.Response to VHR9
HEEGNet and EEG foundation modelsCnkA W3/Q2SPD-MetaFormer has the highest mean on all three datasets.App. J

Clarifications

  • Contribution (efGU W1, CnkA W2). The main contribution is the controlled diagnosis of manifold attention, and SPD-MetaFormer is the model that follows from it, built from established SPD operations by design (responses to efGU and CnkA).
  • Source of the gains (CnkA W1/Q1). We clarify that the gains come from the SPD path built around the uniform mixer, not from uniform mixing itself (Sec. 1, Sec. 4.2).
  • ABIDE-I (VHR9 W1). The text now says "competitive" and claims no advantage (Sec. 4.1, App. C).
  • Scope (CnkA Q3). Whether the finding holds for attention-based EEG decoders in general is beyond the scope of this paper, and we plan to take it up as our next step (response to CnkA).

We thank the reviewers again and are happy to clarify any remaining point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.