acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Spatial Inductive Bias in EEG Transformers: A Geodesic Approach

Abstract

The generalization capacity of deep learning models depends heavily on embedding the correct inductive bias for the geometry of the underlying data. EEG electrodes are distributed over a near-spherical scalp surface whose natural metric is geodesic distance, yet existing spatial positional encoding schemes for EEG Transformers universally rely on Euclidean or coordinate-decomposed assumptions, creating a systematic mismatch with the true geometry. We provide a rigorous theoretical analysis of this mismatch, proving that several widely adopted encoding families fail to satisfy rotational invariance on the sphere, each with a quantifiable error bound. Building on this analysis, we propose a unified geometric positional encoding framework with two complementary components: one that constructs rotation encodings directly from geodesic distance and achieves exact rotational invariance by design, and one that introduces an attention bias built on a geodesic kernel whose positive-definiteness follows from classical results on spherical harmonic analysis. Across a diverse set of EEG downstream tasks, the resulting framework yields consistent gains over strong existing baselines, and our analysis further clarifies why geometric correctness in positional encoding matters for cross-subject generalization. Code is available at https://anonymous.4open.science/r/georope/.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Positional Encoding, Geometric Deep Learning, Spherical Geometry, EEG Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2d3KOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The submission investigates spatial inductive biases in EEG Transformers. The authors argue that because EEG sensors lie on an approximately spherical scalp surface, Euclidean chord distance and Cartesian coordinate encodings fail to capture the true intrinsic scalp metric. The paper makes two core contributions:

  1. Theoretical analysis of three existing spatial encoding families: proving an O(ψ3)O(\psi^3) metric distortion bound for Euclidean chord surrogates ("Theorem 1 (O(ψ 3) metric distortion)"), an irreducible 2BqBk2B_q B_k discontinuity at the poles for azimuthal spherical encodings, and frame-dependence under SO(3)SO(3) rotations for per-axis Cartesian encodings such as NeurIPT ("Proposition 4 (Frame dependence of per-axis encodings)").
  2. GeoRoPE & GDA: A geometric positional encoding framework comprising GeoRoPE (2D rotary encodings parameterized by geodesic distance) and GDA (an additive linear geodesic distance attention bias −λhdS2(p^i,p^j)-\lambda_h d_{S^2}(\hat{p}_i, \hat{p}_j), whose induced exponential/Laplacian geodesic kernel Kij=exp⁡(−λhdS2(p^i,p^j))K_{ij} = \exp(-\lambda_h d_{S^2}(\hat{p}_i, \hat{p}_j)) is proven strictly positive definite on S2S^2 for λh>0\lambda_h > 0 via "Theorem 6 (Strict positive definiteness of the GDA kernel)"). The authors evaluate this framework on four downstream EEG benchmarks (Mental Arithmetic, PhysioP300, Sleep-EDFx, BCIC-IV-2A), reporting improvements over six baselines on 11 of 12 metrics.
Critical strengths
  1. Rigorous Differential Geometry Foundation (Section 4, Theorems 1 & 6): The paper provides a principled mathematical characterization of spatial encodings on spherical manifolds. In "Theorem 1 (O(ψ 3) metric distortion)", the authors prove that Euclidean chord distance incurs an O(ψ3)O(\psi^3) metric distortion that compresses long-range scalp separations. In Section 4.3 ("Theorem 6 (Strict positive definiteness of the GDA kernel)"), the authors prove that linear geodesic distance bias induces a strictly positive-definite exponential kernel Kij=exp⁡(−λhdS2(p^i,p^j))K_{ij} = \exp(-\lambda_h d_{S^2}(\hat{p}_i, \hat{p}_j)) (λh>0\lambda_h > 0) on S2S^2, using a random-hyperplane embedding φ:S2→L2\varphi: S^2 \to L_2 with dS2(x,y)=∥φ(x)−φ(y)∥L22d_{S^2}(x, y) = \|\varphi(x) - \varphi(y)\|^2_{L_2}, which reduces KK to a Gaussian kernel matrix on distinct points (Wendland, 2005). Closely related positive-definiteness results on S2S^2 are classical (e.g., Schoenberg, 1942; Gneiting, 2013), and the paper would benefit from positioning Theorem 6 against them; the link to the attention bias is nonetheless clearly made and, on my reading of Appendix A, correct.

  2. Frame-Invariance and Topological Well-Posedness (Section 3.2 & Section 4.2): By constructing rotary positional encodings directly from scalar geodesic distance, GeoRoPE eliminates coordinate-frame sensitivity. In "Proposition 4 (Frame dependence of per-axis encodings)", the authors prove that per-axis Cartesian encodings are frame-dependent: for a fixed electrode pair and a π/4\pi/4 rotation about the zz-axis, the rotary per-axis analogue changes the logit by 2sin⁡(1/2)BqBk≈0.96BqBk2\sin(1/2) B_q B_k \approx 0.96 B_q B_k for some admissible representations, and the similarity induced by NeurIPT's absolute per-axis embedding changes by at least ≈0.075\approx 0.075. GeoRoPE avoids this frame dependence and eliminates polar singularities.

Critical weaknesses
  1. Missing Component and Metric Ablations and Non-Causal Single-Task Visualizations (Sections 5.1–5.3, Table 2, Figures 2–3): The empirical validation does not isolate the causal factors behind the headline claim that "downstream tasks, the resulting model improves over existing baselines on 11 of 12 metrics.":
  • GeoRoPE Parameterization and Computational Cost: Standard RoPE (Su et al., 2024) factorizes into per-token rotations R(mθ)R(m\theta) with O(ND)O(ND) cost. GeoRoPE parameterizes rotation via intrinsic pairwise distance dS2(p^i,p^j)d_{S^2}(\hat{p}_i, \hat{p}_j), which, as the authors note (Section 3.2), cannot be factorized into per-token rotations; the logits instead require two block Gram matrices per frequency pair, weighted entrywise by cos⁡(ωtdS2)\cos(\omega_t d_{S^2}) and sin⁡(ωtdS2)\sin(\omega_t d_{S^2}). The paper describes this as "a constant-factor overhead over standard attention" but reports no runtime or memory measurements, and it is unclear whether the computation can be fused into FlashAttention-style kernels. Furthermore, because dS2d_{S^2} is invariant under all of O(3)O(3), reflections included, the positional encoding by itself does not distinguish mirror-symmetric electrodes (e.g., C3 vs. C4 relative to Cz). The authors acknowledge this and state that lateralized information reaches the network "through channel identity and signal content rather than through the spatial encoding" (Section 4), but it is unclear which model component carries channel identity (see Question 4).
  • GeoRoPE vs. GDA: The method combines rotary modulation (Eq. 3) and additive distance bias (Eq. 4). The paper does not report an ablation isolating GeoRoPE alone versus GDA alone versus the combined model. It remains unknown whether rotary encoding contributes beyond simple distance decay.
  • Geodesic vs. Euclidean Chord: Despite Theorem 1's chord distortion bound, there is no experiment comparing geodesic distance against Euclidean chord distance on the exact same architecture. Without this control, the central thesis—that geodesic metric fidelity drives downstream gains—is not directly tested.
  • Rotational Stress Test & Non-Causal Single-Task Visualizations: Despite Proposition 4 proving NeurIPT's frame dependence under SO(3)SO(3) rotation, the paper does not report an evaluation of GeoRoPE and NeurIPT under rotated frames at test time. Furthermore, visualizations in Section 5.3 (Figures 2–3) are restricted to Mental Arithmetic, and the authors explicitly concede that Figure 2's gradient attribution "does not by itself establish a causal role for the geodesic bias." and Figure 3's UMAP separation "is not evidence for a specific geometric mechanism."
  1. Methodological Deficiencies in Benchmark Comparisons (Section 5.1, Table 2):
  • Borrowed Baseline Results & Unspecified Pretraining Regime: Table 2 caption explicitly states "Baseline results are taken from Fang et al. (2026);" rather than evaluating baselines locally. All six compared baselines (BENDR, BIOT, LaBraM, CBraMod, EEGPT, NeurIPT) are described in Section 2.1 as self-supervised or pretrained EEG models, yet Section 5.1 does not specify whether GeoRoPE was pretrained on an upstream corpus or trained purely from scratch for 50 epochs on each downstream benchmark on a single GPU ("All experiments are conducted on a single NVIDIA RTX 4090 GPU."), nor whether the Fang et al. (2026) numbers reflect pretrained checkpoints or from-scratch training.
  • Omission of Variance: Although the Table 2 caption states "our results are averaged over five seeds under the same data splits and preprocessing.", Table 2 reports only mean point estimates without standard deviations across seeds. Cross-subject EEG results can vary noticeably across seeds and test subjects, so margins such as 79.09% vs. 77.57% Cohen's kappa on Sleep-EDFx are difficult to interpret without uncertainty reporting or paired significance tests.
  • Missing Architecture Specifications: Transformer layer count, attention heads, hidden dimensions, and parameter counts are omitted from Section 5.1.

Score rationale

Presentation (3). The paper is well-written, logically structured, and features clear mathematical exposition. Proof sketches in Section 4 are readable, and Appendix A provides complete, unabridged proofs for all theorems and propositions. Minor naming inconsistencies exist between the title header (A Geodesic Perspective) and metadata (A Geodesic Approach), but they do not hinder comprehension.

Informativeness (2). The paper provides valuable theoretical insights for geometric deep learning in neural interfaces. In particular, the proof in Section 4.3 (specifically "Theorem 6 (Strict positive definiteness of the GDA kernel)") that the exponential (Laplacian) geodesic kernel Kij=exp⁡(−λhdS2(p^i,p^j))K_{ij} = \exp(-\lambda_h d_{S^2}(\hat{p}_i, \hat{p}_j)) (λh>0\lambda_h > 0) induced under the softmax exponential by the linear geodesic distance bias is strictly positive definite on S2S^2—shown via a random-hyperplane embedding into L2L_2 under which dS2d_{S^2} becomes a squared Hilbert-space distance, so that KK is a Gaussian kernel matrix on distinct points—is an elegant result, contrasted with the failure of positive definiteness of the squared form exp⁡(−λdS22)\exp(-\lambda d_{S^2}^2) for λ∈(0,0.247)\lambda \in (0, 0.247) (Remark 7(i)). However, I believe Theorem 6 follows from classical results on positive definite functions on spheres (e.g., Schoenberg, 1942; Gneiting, 2013) and on geodesic exponential kernels (Feragen et al., 2015, which the paper cites only in Remark 7), and its sign-function embedding mirrors the random-hyperplane argument of Goemans & Williamson (1995). The paper should position Theorem 6 against these results; I read it as a clean application of known tools rather than a new theorem, supporting informativeness: 2.

Soundness (2). While the theoretical derivations appear sound on my reading of the proof sketches and Appendix A, the empirical methodology has significant gaps:

  1. Borrowed baseline numbers: Baseline numbers in Table 2 are directly copied from Fang et al. (2026) rather than evaluated in a unified local codebase.
  2. Missing component ablations: There are no ablation experiments separating GeoRoPE from GDA, nor comparing geodesic distance against Euclidean chord distance on the same backbone.
  3. Missing variance reporting: Table 2 provides only point estimates without standard deviations across seeds.
  4. Missing architecture specifications: Layer counts, attention heads, and parameter counts are omitted.
Questions for authors
  1. [clarification] Component & Metric Ablations: Could you provide an ablation table on the four evaluated benchmarks reporting: (a) Backbone baseline without spatial encoding, (b) Backbone with Euclidean chord rotary encoding, (c) Backbone with GeoRoPE only, (d) Backbone with GDA bias only, and (e) Full model? Specifically, what proportion of the gain is attributable to GDA versus GeoRoPE?

  2. [clarification] Standard Deviations & Statistical Significance: Could you provide standard deviations (±std\pm \text{std}) across the 5 random seeds for all 12 metrics in Table 2, and report whether improvements over NeurIPT are statistically significant under a paired t-test or Wilcoxon signed-rank test?

  3. [clarification] Pretraining vs. From-Scratch Regime & Architecture Parity: Was GeoRoPE trained purely from scratch for 50 epochs on each downstream dataset, or pretrained on an upstream EEG corpus? Do the borrowed baseline numbers from Fang et al. (2026) in Table 2 come from pretrained foundation-model checkpoints or from-scratch training, and what are the exact architectural hyperparameters (layers, heads, hidden dimension DD, parameter count) compared to NeurIPT?

  4. [future work] Rotational Invariance vs. Anatomical Lateralization: Since dS2d_{S^2} is invariant under all O(3)O(3) transformations (including reflections), the spatial encoding by itself is unchanged when the montage is mirrored left–right or anterior–posterior, and Section 4 states that lateralized information reaches the network "through channel identity and signal content rather than through the spatial encoding." Which component supplies channel identity (e.g., a learned channel embedding), and, in motor imagery tasks (BCIC-IV-2A) that depend heavily on contralateral sensorimotor rhythms (C3 vs. C4), how does relying on channel identity rather than the spatial encoding affect cross-subject generalization?

  5. [future work] Computational Overhead & FlashAttention: Because pairwise geodesic rotations cannot be factorized into per-token transformations, GeoRoPE requires computing two full Gram matrices across D/2D/2 frequency bands (Section 3.2 describes this as "a constant-factor overhead over standard attention"). Could you report wall-clock training throughput and peak GPU memory compared to standard attention, and state whether the implementation can use fused kernels such as FlashAttention?

Reviewer_AcFWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GeoRoPE and GDA, which use spherical geodesic distance in a rotary attention term and an additive distance bias. It analyzes their geometric properties and evaluates the combined model on four EEG tasks.

Critical strengths
  1. The analysis distinguishes passive coordinate-frame changes from physical rotations of neural activity, and separates frame invariance from metric choice. The checked proofs are coherent. The paper also acknowledges that chord distance is invariant and that strict positive definiteness applies only to the spatial kernel factor.
  2. The evaluation covers distinct EEG paradigms, with promising gains on cognitive load, event-related responses, and motor imagery. The authors also report the lower sleep-staging balanced accuracy alongside higher weighted F1 and kappa. This gives a more complete picture of where the method helps.
Critical weaknesses
  1. The specific empirical benefit of geodesic geometry is not isolated. Chord distance is also frame-invariant, so the invariance proofs alone do not establish a predictive advantage over chord distance. The experiments compare complete models, without controls that replace geodesic with chord distance or separate the rotary and additive terms on the same backbone.
  2. In Table 2, The comparison with external baselines does not establish comparable learning conditions. Using the same splits and preprocessing is helpful, but the backbone, pretraining or initialization, capacity, and trainable components are not described clearly enough to separate the effect of spatial encoding from differences between the full models.
  3. The stability of the gains and their distribution across classes remain unclear. The reported means cover five random initializations, but give no dispersion or paired participant-level effects. The sleep-staging tradeoff also deserves closer examination: weighted F1 and kappa improve while balanced accuracy declines. Per-stage results would help explain which classes account for this difference.
Questions for authors
  1. On the same backbone and training protocol, how do chord distance, GeoRoPE alone, GDA alone, and the combined method compare?

  2. Are pretraining exposure, model capacity, and trainable components comparable with the cited baselines? Which differences in learning conditions remain?

  3. How variable are the results across the existing runs and test participants, and which sleep stages explain the lower balanced accuracy despite higher weighted F1 and kappa?

Reviewer_HFNNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a spatial positional encoding for EEG Transformers based on the geodesic distance between electrodes on a sphere: GeoRoPE uses the geodesic distance as the RoPE rotation angle (Eq. 3), and GDA adds a distance bias to the attention logit (Eq. 4). Their claimed contributions:

(1) they "prove" that widely adopted spatial positional encoding families either depend on the coordinate frame of the electrodes or distort the intrinsic scalp metric (Section 4), (2) propose a geometric positional encoding framework. On EEG downstream tasks, their resulting model improves over existing baselines on 11/12 metrics (Table 2).

Critical strengths
  • The general idea to insert spatial inductive bias in EEG transformers in the form of electrode spatial positional encoding is interesting and promising.
  • It is interesting that the proposed method, GeoRoPE, seems to perform well compared to baselines on various datasets (Table 2).
Critical weaknesses

The reported numerical results lack error estimates or p-values; better statistical treatment is needed. Several methods and metrics used are not defined and unmotivated. There is very minimal interpretation of results in the text, so it is hard to get intuition or understanding for the claims made. The figures are unpolished, lack descriptive labels/captions.

Detailed comments:

  • The writing is hard to parse; e.g., lines 52-53: "Encoding the azimuthal difference is invariant only under rotations about the polar axis and has a O(1) discontinuity at the poles that no branch convention removes." This sentence is packed with undefined jargon that is inappropriate for an introduction section..
  • Figure 1 looks very unpolished and low-effort: it's mostly empty white space, unreadable small font.
  • Format: Theorem 1 (line 223) is followed by Theorem 6 (line 288), because theorems, propositions, remarks and corollaries share one counter. That's bad formatting; Theorem 6 should be Theorem 2.
  • In Table 2, where you show the cross-subject results, you do not show any error bars. Baseline numbers are also taken from Fang et al. (2026) (lines 325-326), while your results are averaged over five seeds (line 355).
  • In Table 2, you use metrics like Cohen's Kappa that are unintroduced and unmotivated (lines 354-355). Therefore, those values are meaningless for readers that are not experts in the exact niche of this paper's scope. The values also lack interpretation.
  • Figure 2 lacks sufficient discussion in the main text, and its caption is too short (lines 403-404). It also is missing colorbars. It's not clear what its significance is.
  • Regarding Figure 3, lines 406-409: "such separation shows that the learned features are discriminative." This would be expected of any decent trained model, and the authors do not provide a comparison between models trained with GeoRoPE versus baseline methods. Thus, Figure 3 feels pointless and does not add to the claims of the paper. The authors acknowledge this themselves (lines 408-409), but the figure is useless as it stands.
  • There is no experiment that compares the encodings directly with the same model: no spatial encoding, the Euclidean chord, the azimuthal and per-axis encodings analyzed in Section 4, GeoRoPE alone, or GDA alone. Table 2 only compares full models from prior work.
  • The model itself is not described. Lines 351-355 give only epochs, batch size, optimizer and schedule (no backbone, size, pretraining, or how channel identity and time enter the model).
  • Frame invariance, the main theoretical motivation, is never tested: all experiments are within single datasets, where no rotation occurs (lines 201-204).
Questions for authors
  1. Abstract: "coordinate-decomposed assumptions" (line 17): what does this mean?
  2. Why is the GeoRoPE rotation applied to the query vector in Figure 1 (lines 130-132) but to the key vector in Eq. 3? This adds even more confusion. In standard RoPE settings the rotation is applied to all the tokens, which then carries over to both query and key vectors.
  3. Why are you modeling the scalp as a sphere (line 145)? The EEG signal originates from the cortical surface, which has folds (gyri and sulci), so its geodesics are not those of the sphere. Why would spherical geometry outside the scalp be relevant?
  4. The role of GDA feels unmotivated; why do you need to insert spatial structure through "both the rotation and the additive distance bias" (lines 184-185)? What's the point of the additive bias?
  5. Section 4.3 "Positive definiteness of the GDA Kernel": why does it matter that K is positive definite? What is the interpretation?
  6. Can you report results with the same backbone for each encoding listed above, with error bars?
  7. What is the full architecture, and is the model pretrained? If yes, on what data?
  8. The GeoRoPE Cohen's kappa is 49.31 for both PhysioP300 and BCIC-IV-2A in Table 2. Can you confirm both values?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.