acceptodds
Under review as a conference paper at ICLR 2027

Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Sky

Abstract

The representations that language models learn for concepts such as months, weekdays, and places display consistent geometric structure: circles and saddle-shaped "Pringle" manifolds. Recent work traced these structures to *translation symmetry* in word co-occurrence statistics, deriving the observed Fourier geometry when co-occurrence depends only on distance on an abelian lattice of concepts. We demonstrate that more general notions of symmetry lead to equally structured predictions. Considering symmetries defined by arbitrary finite groups, compact groups, and homogeneous spaces, we prove that whenever the co-occurrence statistics of a word family are invariant under a group , the learned word embeddings consist of matrix elements of the irreducible representations (irreps) of . Circles and Pringles arise when is cyclic, in which case the irreps are Fourier modes. We verify the irrep structure in three experimental settings. (i) The cyclic group : for the months of the year we recover the known circular geometry. (ii) A dihedral group acting on the major and minor triads: we unify two classical observations—that transposition and chord inversion form a group () acting on chords (music theory), which *implies* that the well-known "circle of fifths" emerges in learned chord embeddings (machine learning). (iii) We explain and reproduce a recently discovered spherical representation of celestial objects in large language models (LLMs) as a spherical-harmonic embedding derived from our theory. Our results demonstrate that the geometry of learned representations is often a consequence of the statistical symmetry of underlying data.

Primary area:
interpretability and explainable AI
Keywords:
representation geometry, symmetry, word embeddings, group representation theory, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NMGFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper hypothesizes that the geometric structures empirically found in word embeddings and model activations among e.g. months, chords, and celestial bodies, result from symmetries in word co-occurrence statistics. The paper formalizes the structures assumed in each of the three examples using one unified construct via groups and representation matrices. The proposed construct explains and predicts the geometry of embeddings from primarily only symmetry relations, which is a compact property of the word co-occurrence matrix. The predicted embeddings are then compared with previous approximations from related work and yield good alignment.

Critical strengths

Although the intuition and related theories behind each of the three examples (month, chord, sky sphere) as to why the embeddings form shapes have been well established, this paper is the first to use one simple theory to explain all three. The theory involves a minimal number of constructions while being general enough to express different symmetries and directly compute the predicted shape for embeddings.

Critical weaknesses

Karkada et al. (2025) approximates real embeddings using an idealized model. The current paper approximates that idealized model using a simple, general, low-parameter model leveraging symmetry structures. This offers nontrivial math insight into the models proposed by Karkada et al. (2025), but for the sake of explaining training phenomena, this two-hop interpretation offers limited insight. All embeddings in this paper are from eigendecomposing M* (L179). No analysis used any embeddings trained from corpus. At least some experiments comparing theory with real embeddings are necessary to tie the results to real dynamics. For the same reason, the current "Empirical" labels in figures are potentially misleading, and "learned word embeddings" in the abstract are in fact "idealized word embeddings".

Questions for authors
  • Why does no experiment compare prediction with trained embeddings? Will the authors replace the "empirical" datapoints with trained embeddings?
  • Which section explains why the theory predicts the circle of fifths instead of other permutations that follow T/I, e.g. a "circle of fourths"? If their distinction comes from non-symmetry statistics, then the claims in L029 and L080 are too strong.
  • Abstract clarity. Currently, the full text, or even the beginning of Introduction, is much more readable than the abstract. Consider a longer, more self-contained abstract that is written as friendly as the full text.
  • Previous work explains embedding geometry from co-occurrence. Intuitively, co-occurrence and embedding geometry should have a common cause in the language. This paper uses symmetry assumptions about co-occurrence to predict the embeddings. This may be claimed to be structurally a step closer to modeling the common cause. The validity of this claim depends on whether co-occurrence is a bottleneck for those word embeddings and model activations considered here.
  • ChordBERT results and figure: Since we are using Gram matrix as a proxy for M*, what exactly does the figure show? Why is it not a test of circular logic, especially when layer 3 has high isotypic energy?
  • L193: I could not understand "Abusing notation ...". It's otherwise acceptable except C is used later. The definition of C should be proper.
  • At first I could not understand why "sphere" was related to celestial bodies. Consider replacing "sphere" with "sky sphere" in "Celestial objects reside on a sphere inside a LLM’s residual stream" for clarity. A similar edit should benefit the abstract too.
  • Fig 3: Consider maybe using colored text labels for matching labels to points. Coloring might work less well for Fig 2 though.
  • L1293: "Fig 4 B" should be "Fig 3 B".
Reviewer_PCN9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how invariance of the co-ocurrence statistics of word families to specific groups informs their learned word embeddings. The authors link word embeddings to the irreducible representations of the relevant group and use this framework to explain specific arrangements of learned representations for word embeddings models and LLMs

Critical strengths

The paper is well written and provides a clean extension of existing work, providing a more general theory of word embeddings under group symmetries.

Critical weaknesses

Beyond the similarity to some existing work like Karkada etc all 2025, the main limitation of the paper is that the theory uses word embeddings so that the main results do not extend to autoregressive LLMs leaving the open question of why these models learn similar structures. Additionally, a group based approach to understanding neural representations may be limited by the fact that only a few word families have co-ocurrence statistics that are close to invariant to specific groups.

These limitations are acknowledged in the main text and are small enough that I would still recommend acceptance.

Questions for authors

Is co-ocurrence still the right statistic to look at for autoregressive models with causal masking? Could any part of this framework be extended to word families with statistics that are not invariant to specific groups?

Reviewer_v3JYOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Recent interpretability work discovered that language models represent structured concepts on low-dimensional manifolds—such as months on circles/“Pringles” (Engels et al., 2025; Karkada et al., 2026) and celestial objects on a 3D sphere (Berdnikov & Liokumovich, 2026). Building on Karkada et al. (2025), which showed that quadratic word embeddings approximate the top eigenpairs of a doubly centered normalized co-occurrence matrix PM⋆PPM^\star P, this paper asks what happens when M⋆M^\star is invariant under the action of a general group G (Mg⋅x,g⋅y⋆=Mx,y⋆M^\star_{g\cdot x, g\cdot y} = M^\star_{x,y}). Theoretical Framework (Section 4, Appendices B–D): Using non-Abelian harmonic analysis (the Peter–Weyl theorem, Schur orthogonality, and Gelfand pairs), the authors prove that if a word family forms a G-torsor (Theorem 4.1) or a homogeneous space G/K (Theorem 4.2), the eigenvectors of PM⋆PPM^\star P are given analytically by the matrix coefficients of the irreducible representations (irreps) of G, ordered by the eigenvalues of the group Fourier transform C^(ρ)\widehat{C}(\rho) of the co-occurrence kernel.

Abelian Case (ℤ₁₂, Section 5): Re-derives the circular/Pringle embedding of the 12 months in text8 (per-month cosine 0.96).

Non-Abelian Case (D₁₂, Section 6): Models the 24 major and minor musical triads as a torsor under the music-theoretic transposition/inversion group T/I≅D12T/I \cong D_{12}. Shows that chord co-occurrences in 371 Bach chorales are approximately T/I-invariant (R² = 0.822), forcing the leading 2D irrep (E₅) to form interleaved major/minor circles corresponding to the classical Circle of Fifths (cosine 0.98 in co-occurrence PCA, 0.91 in ChordBERT layer 3).

Continuous Homogeneous Space (S² = SO(3)/SO(2), Section 7): Re-analyzes the LLM “sky sphere” activations of Berdnikov & Liokumovich (2026) across 7 LLMs (e.g., Mistral Large 2 123B). Shows that the representation is not merely a 3D sphere (ℓ=1\ell=1 dipole), but exhibits degree-ordered spherical harmonic blocks (ℓ=1,2,3\ell=1, 2, 3). Reproduces a weaker version of this structure in co-occurrence statistics from four public-domain stargazing guides (131k tokens, cosine 0.55) and uses Poisson simulations to argue the gap is explained by sample size (32--64 ×).

Critical strengths

Elegant Unification Across Disparate Domains: Connecting circular month embeddings (ℤ₁₂), the musical Circle of Fifths (D₁₂), and celestial representations (SO(3)/SO(2)) under a single group-theoretic principle is intellectually compelling and provides a clean vocabulary (torsors, homogeneous spaces, isotypic energy) for studying geometric representations.

Novel & Testable Prediction Confirmed in LLMs (ℓ≥2\ell \ge 2 Spherical Harmonics): While Berdnikov & Liokumovich (2026) only looked for a 3D sphere (ℓ=1\ell=1) in LLM residual streams, this paper’s theory predicts that rotational invariance of co-occurrence statistics must also produce higher-degree spherical harmonics (ℓ=2\ell = 2 quadrupole of dimension 5, ℓ=3\ell = 3 octupole of dimension 7). Finding clean, degree-ordered harmonic blocks in 5 out of 7 frontier LLMs (Figure 4b, Table 2) is a genuine, non-trivial empirical discovery.

Bridging Mathematical Music Theory and Representation Learning: The D₁₂ analysis on Bach chorales neatly explains why the Circle of Fifths and interleaved major/minor circles emerge in chord embeddings, backed by rigorous model-comparison baselines against 7 competing symmetry hypotheses (Z₁₂, Z₂₄, PLR, conjugacy classes, etc. in Figure 8 & Table 1).

Thoughtful Null Models and Statistical Controls: The authors consistently benchmark their cosine similarities and R² metrics against permutation/relabeling nulls, bootstrap intervals, and synthetic Poisson count simulations (Figure 5).

Critical weaknesses

No Actual Word Embedding Models Are Trained in the Experiments: Throughout the paper, the authors refer to comparing theoretical predictions against “empirical word embeddings.” However, as buried in Appendix A.1 (Page 12, Line 610): "Up to an orthogonal transform and realification, the projection of W‾x∈Rd\overline{W}_x \in \mathbb{R}^d along the first r principal components of W‾\overline{W} is approximately (√λᵢvᵢ(x))_(i ≤ r). This is how we generate all empirical word embeddings in this paper."

Why this is a critical weakness: The authors never actually train Word2Vec, GloVe, or skip-gram embeddings on text8, the Bach chorales, or the astronomy books! Instead: Their “empirical word embeddings” are literally the eigenvectors of PM⋆PPM^\star P.

Their “theoretical predictions” are the eigenvectors of ΠGM⋆\Pi_G M^\star (the group-averaged version of M⋆M^\star).

Consequently, the reported high cosine similarities (0.96 for months, 0.98 for Bach chords) are not testing whether neural word embedding training dynamics learn group irreps; they are merely measuring matrix perturbation stability—i.e., if ΠGM⋆≈M⋆\Pi_G M^\star \approx M^\star (R² = 0.822), then by the Davis–Kahan theorem, the top eigenvectors of M⋆M^\star are guaranteed to be close to the eigenvectors of ΠGM⋆\Pi_G M^\star!

Questions for authors

In Appendix A.1, you state that all “empirical word embeddings” in the paper are generated directly as the scaled eigenvectors (√λᵢvᵢ) of PM⋆PPM^\star P. Since M⋆M^\star has R² = 0.87 (months) and R² = 0.822 (chords) with its group-invariant projection ΠGM⋆\Pi_G M^\star, isn't the high alignment between the eigenvectors of M⋆M^\star and ΠGM⋆\Pi_G M^\star simply a consequence of matrix perturbation theory (Davis–Kahan) rather than a test of representation learning? What are the cosine similarities and isotypic energies if you actually train standard Skip-Gram with Negative Sampling (SGNS) or GloVe embeddings on these corpora, or inspect the input token embedding matrix (W_E, layer 0) of the LLMs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.