acceptodds
Withdrawn from ICLR 2027

From Language Statistics to Semantic Control: Spectral Geometry and Group Actions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language models represent ordered concepts in structured activation patterns that can span many directions. An intervention that looks correct in a low-dimensional projection can therefore leave other coordinates inconsistent. We connect language statistics, spectral geometry and semantic control to understand how this structure arises and how to change it consistently. Starting from a latent variable data generating process, we relate latent semantic structure to word co-occurrence and, under explicit embedding assumptions, to Fourier coordinates. Group-based month and weekday steering transfers to held-out prompts, while an open-interval extension transports a fitted neural year component bidirectionally. To steer richer concepts, we must also understand which geometries arise beyond the circle and what transformations they support. Drawing on lattice normal modes, we interpret spectral modes as semantic phonons: coordinated patterns of variation across semantic positions. Sturm–Liouville theory explains how boundary conditions shape these patterns for second-order operators, yielding a broader family of curved geometries even when the interior rule is unchanged. This work connects understanding neural geometry to reliable control: for safety-relevant manifolds such as refusal, interventions must intervene across all relevant subspaces.

Primary area:
interpretability and explainable AI
Keywords:
Representation learning, mechanistic interpretability, semantic geometry, data-generating processes, spectral analysis, group actions, activation steering

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_CMtNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The submission approaches activation steering in the spectral coordinates to support the symmetries in particular concepts such as periodic concepts (weekdays, months) and ordered concepts (year). Authors argue that although one could identify and read the concept in low-dimensional projections of corresponding activations, such readable view could only represent a part of broader picture (supported by less than 50% total varianace explained by low dimensional projection). Then they propose steering in full spectral coordinates as truncating it can potentially ignore large contributions coming from high-frequency modes. The modest experiments are concentrated on "circular" weekday shifting, and extended to year-shifting.

Critical strengths
  1. A nice conceptual contrast between decoding and control: The idea of high-frequency modes contribute non-negligible error term provides an interesting motivation for decoupling concept decoding and steering
Critical weaknesses
  1. The motivation of the methodology (other than the part stated in Strength section above) is heavily rooted in existing works with minimalistic new insights. For example, Eqns (1) and (2) are introduced as authors' contribution while they directly come from taking P(i) to be a uniform distribution in Section 4 of [1], followed by the same fourier diagonalization derivations.
  2. The difference between the results in [2] is not explicitly discussed other than mentioning that the authors use the derivations in [2, Appendix G] for Fourier Fourier embeddings of circulant similarity matrices and cyclic shifts on retained invariant subspaces. This makes it vague/harder for the reader to understand the significance of this work given [2, 3]. Is it the experiments on different steering tasks, transferring to held-out prompts etc.? How important are these new empirical observations?

References:

[1] Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, and Yasaman Bahri. Symmetry in language statistics shapes the geometry of model representations. In International Conference on Machine Learning, 2026

[2] Gurnee et al., When Models Manipulate Manifolds: The Geometry of a Counting Task (2026)

[3] Sheridan Feucht, et al.. Arithmetic in the wild: Llama uses base-10 addition to reason about cyclic concepts. arXiv preprint arXiv:2605.01148, 2026

Questions for authors
  1. See Weakness 2
  2. Section 6 motivates candidate representation geometries through Sturm-Liouville boundary conditions, but its operational contribution remains unclear. The year experiment constructs coordinates by numerical kernel eigendecomposition and fits transitions from supplied year adjacencies. It does not demonstrate that selecting an operator or boundary condition improves representation prediction or behavioral control. Could authors clarify on the operational role of Sturm-Lioville framing in this work?
1 comment

We thank the reviewer for the time taken with our manuscript and for the assessment. Taking these comments seriously, we have decided to withdraw the paper and appropriately rework our manuscript. On the comments: it is true that equations (1) and (2) follow from [1] under a uniform prior. We intended them as an illustration rather than a contribution. On the relation to [2]: the difference is that the operator in our work reads the current coordinate from a live activation, rather than being supplied it. We tested this on held-out prompts against a displacement-matched rate-permutation control. We agree this distinction should have been stated more explicitly.

Reviewer_T9EeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This study discusses a potential basis of semantic information in language models (LMs). It has been inspired by the earlier study suggesting that months are encoded in the circular plane. The authors studied an optimal basis for the encoding of months and its spectral properties, and their analysis led them to develop steering algorithm to manipulate the encoding of months.

Critical strengths
  • The theoretical analysis of the potential basis of LMs is intriguing and may encourage more efforts in this direction.
  • The steering results look fairly good.
Critical weaknesses

The scope of this study is quite narrow, which leads me to question whether the analysis could be applied to more general tasks or concept-encodings. First, the theoretical analysis is limited to a single basis, and the authors do not discuss how widely this type of basis could be used in LMs. Second, they test the steering vectors (derived from the proposed bases) with 3 models, Llama-3-8B, Llama-3.1-8B and Gemma-2-2b-it. Notably, two of the models are virtually identical.

Questions for authors

Please see Weakness

1 comment

Thank you for taking the time to examine our work and providing valuable comments. We decided to withdraw the paper, and take your feedback for a revised work into account. Nonetheless, we would like to briefly mention that the restriction to a single basis may have seemed narrower in our presentation than it is. Section 6 and Appendix K derive the other mode families a one-dimensional ordered concept can carry (Neumann, Dirichlet and Robin) alongside the periodic one, and Section 5.3 constructs shifts on an open interval, where no cyclic basis exists.

Reviewer_t7LaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a method for activation steering on curved latent manifolds. The method is used on several concrete use-cases, including steering months, weekdays, and years.

Critical strengths

Activation steering is an interesting and well-researched question, and a suitable topic for ICLR.

Critical weaknesses

The paper addresses an interesting and timely question, but, in its current state, it is unreadable, and it needs to be thoroughly rewritten. The main text is full of vague, jargony assertions, and is structured in a way that does not meet standards for academic writing. This is clear even from the abstract, but I paste several examples below from the main text:

"space and time preserve meaningful coordinates"

"For a cyclic semantic variable, translation-symmetric co-occurrence is diagonalized by Fourier modes."

What a semantic variable is, is never formally defined. What about that variable makes it cyclic? Co-occurrence of what? The Fourier modes of what?

"Fourier features support learned modular arithmetic."

Fourier features on what? What learned the modular arithmetic (an LLM?)? And how?

The entire paper reads like this. Further, figures are clearly AI-generated, and contain subfigures that are never referenced in the caption nor the text. The Appendix is full of much similar content and is extremely difficult to comb through.

To the authors: my review may be rather discouraging, but I would encourage you to tailor your writing according to your own experience reading a well-written paper. For instance, we all enjoy reading papers that are clear, pedagogical, and logically presented. A very useful exercise will be to attempt to write the content all by hand, as it will help you better structure the article, e.g., introducing terms pedagogically when they first appear.

Questions for authors

N/A

1 comment

We thank the reviewer for taking the time to examine our work. We acknowledge these concerns and will rework our manuscript appropriately.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.