acceptodds
Under review as a conference paper at ICLR 2027

Circles or Orderings? A Symmetry Test for Cyclic Concepts in Language Models

10 views

Abstract

The linear representation hypothesis treats a concept as a direction. Cyclic concepts such as days of the week and months look like an exception: several studies report that language models place them on a circle (Engels et al., 2025; Karkada et al., 2026; Feucht et al., 2026). We show that the tests behind such reports cannot tell a cycle from an ordering. A fitted successor operator has its spectrum at the th roots of unity for any tokens in a fixed order, random nouns included. Concentration of variation in the fundamental Fourier mode is as high for number words, ordinals, and temperature adjectives as it is for months. A short step from the last element back to the first also appears for graded scales whose two extremes lie close together. We derive a test from the symmetry itself. Advancing a concept by one step acts as an isometry on its states exactly when their Gram matrix is circulant, so we measure the departure from circulance. Across nine models from three families (Pythia 160M to 12B, Llama-3.1-8B base and instruct, Qwen2.5-7B-Instruct), days and months are more circulant than every same-length ordered control in every model, measured at the concept token. Clock hours cannot be distinguished from a temperature scale, and compass directions are organized by cardinal versus intercardinal type rather than as an eight-cycle. Behavior does not follow automatically from the representation. With the same closed representation of months, Llama shows a wrap-around penalty consistent with the base-10 mechanism reported by Feucht et al. (2026), while Qwen does not. On the compass, Llama never answers a 45-degree step correctly, which matches its two-axis representation.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, representation geometry, linear representation hypothesis, cyclic concepts, circular features, symmetry, equivariance, circulant matrices, language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_spAaOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper concerns previous findings in interpretability literature about cyclic concepts. Unlike ordinary concepts, which can usually be represented well as a vector, certain concepts such as days of the week or months of the year are usually placed on a circle. The authors claim that the tests that underlie this judgment may not be enough to make this claim. They apply three measures of circularity and find that they do not properly distinguish linear from ordered concepts, although days and months still show some evidence of being genuinely cyclic representations.

Critical strengths

The paper conducts relatively careful experimentation and identifies a weakness in existing tests for circular representations in that an ordered list can produce several signals that appear circular. The authors experiment with several criteria and model families, and the documentation of experiments is relatively thorough.

Critical weaknesses

This paper is difficult to read and the general writing style seems heavily affected by claudisms. For instance, I have never heard anyone refer to this property as “circulance” though if the authors can point me to some recent works that use this terminology I can update my beliefs. The manuscript is extremely long-winded, yet it is never really motivated why we should care about the experiments presented here. The authors are encouraged to properly motivate their work in the text and to clean up overly-dense language.

My larger concern is the significance of this paper. Cyclic concepts are already a fairly specialized case in interpretability, and the paper does not make clear what changes if a representation passes this test or does not pass this. Does the result help explain a model’s computations, predict its behavior, or improve an interpretability method beyond making claims of circularity more precise? The two canonical examples, days and months, still pass, so the main finding is not as surprising as it seems either. The failures for other concepts are interesting, but what do we learn from this as a field? I would encourage the authors to shorten the presentation and make the broader consequences more explicit.

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Reply to Reviewer spAa (1/2)

We thank Reviewer spAa for the review. We appreciate that the reviewer found the experiments careful and the documentation thorough, that the paper identifies a weakness in the tests used for circular representations, and that the reviewer had no substantive concerns about soundness. Following the reviewer's comments, the revision rewrites the main sections in plainer language and adds a paragraph on why the question matters. Under W4 to W6 below, we explain in more detail what the results change for interpretability.

W1: Readability

We rewrote Sections 3 and 4. Each paragraph now opens with what its result means, and the numbers follow as evidence. We also rewrote the abstract, the introduction, the setup in Section 2, the behavioral results in Section 5, the related work, and the Discussion in plainer language. The introduction explains why the question matters before it turns to methods. Section 6, which two reviewers found disconnected from the rest, moved to the appendix, which shortens the main text.

W2: The term "circulance"

A circulant matrix is one in which each row is the previous row shifted by one place. Circulant matrices have long been studied in linear algebra, they are diagonalized by the discrete Fourier transform (Gray, 2005), and the circulant matrix closest to a given matrix in Frobenius norm is found by averaging each wrapped diagonal (Chan, 1988). Our ν\nu is the relative distance from the Gram matrix to that closest circulant matrix. Du et al. (2026) measure the same quantity for weekdays in four language models and call it the circulant distance (δcirc\delta_{\mathrm{circ}}). We called it non-circulance because ν\nu is zero exactly when the Gram matrix is circulant and grows as the matrix departs from that form. The revised text gives both references where ν\nu is defined, and we can adopt the name circulant distance throughout the final version if the reviewer prefers.

W3: Motivation

Circular geometry is now used in interpretability methods, and those methods assume that the curve closes. Engels et al. intervene on circular day and month features. Wurgaft et al. (2026) steer models along curves fitted through such features, and Bhalla et al. (2026) ask whether sparse autoencoders recover them. Each of these uses takes the fitted curve to be the concept, so that moving along it changes the concept and one full turn brings it back to the start.

If the states are only ordered, a closed curve through them still exists, but its last segment joins the two ends of a scale. In our data, the two extremes of the temperature scale lie close together in the representation, so the scale passes the wrap-gap test, and a ring fitted through it would treat "scorching" and "freezing" as neighbors, although the scale has no step between them. Moving along that segment changes nothing the model represents. Clock hours show the same risk from the other side: a Fourier test ranks them as the most circular concept in our data, so a researcher choosing a concept for ring steering on that evidence could pick them, yet our test finds them only borderline.

Several recent accounts derive circular geometry from the statistics of language (Karkada et al., 2026; Modell et al., 2025; Du et al., 2026), and a test of closure is what such predictions can be checked against. The test is also cheap to apply, since it needs only the Gram matrix of the states and a few ordered control sets of the same length, which any study of a cyclic concept can add. The introduction now makes this case before the paper turns to methods.

W4: What changes if a representation passes or fails

The first change is in what counts as evidence. The signals behind circle claims are produced by orderings and even by random nouns: a fitted successor map has its spectrum at the roots of unity for random nouns, and held-out successor fits reach R2R^2 of 0.75 to 0.96 for them. A claim that rests on these signals alone says nothing about closure, and our test is built so that orderings fail it.

The test also agrees with behavior in one case. It found that compass directions form two interleaved four-cycles, and in completion prompts Llama's errors match that structure, since it keeps the starting type on 97 to 100% of 45-degree steps and never answers one correctly. This link is correlational, and the paper says so.

For computation, passing the test does not mean that a model computes on the circle. The wrap-around penalty differs between models and prompt formats, which fits the base-10 mechanism that Feucht et al. (2026) found in Llama. Du et al. (2026) predict circulant geometry from symmetry in the training targets, and our controls and decision rule turn their measure into a test that can confirm or reject such predictions.

AuthorsOpenReview ↗

Reply to Reviewer spAa (2/2)

W5: Days and months pass, so the result is not surprising

What is new is that the earlier evidence could not have shown it. Before this work, the claim that days and months are circles rested on tests that an ordering also passes. A fitted successor map has its eigenvalues at the roots of unity even for random nouns, the fundamental share ranks months above ordinals in only 2 of the 9 models, and the wrap gap passes temperature scales. On that evidence the claim was untested. We test it with a measure that orderings fail, against seven matched controls in nine models, and with the noise-invariant measure all 81 Holm-corrected tests pass.

The test could have failed for days and months, and it does not give a clear pass for several other concepts that were expected to be circular. Clock hours, which most readers would expect to form a cycle, are only borderline. The day of the month inside a date behaves as an ordering, although calendars repeat. Compass directions are organized by direction type, and seasons do not form a cycle. A test that gives these answers is informative when it says that days and months are closed.

The result also says where the cycle is held. Days and months are closed at the token that names them; at the end of the sentence they remain more circulant than number words and ordinals but no longer separate from the graded scales. This agrees with the finding of Feucht et al. (2026) that Llama keeps the circle at the concept token but not where it computes the answer. The month cycle persists inside a date such as "10 January". These are new facts about the cycle, and the earlier tests could not reveal them because they also pass for orderings.

W6: What the failures teach

A recurring question in representation learning is whether a model trained on language inherits the symmetries of the world it describes (Karkada et al., 2026; Du et al., 2026). Our results show that the answer depends on the concept. Days, months, clock hours, compass directions, seasons and the day of the month all repeat in the world, but the test finds a closed ring only for days and months. Clock hours are borderline, the day of the month inside a date is an ordering, and compass directions and seasons have no more energy in the fundamental mode than random points. So a symmetry of the data does not guarantee the same symmetry in the representation, and the Gram matrix can test which symmetries a model has learned. A method that assumes a ring for every periodic concept will go wrong for some of them, and the test identifies which ones. The revised introduction makes this point.

We hope the revised text and these answers resolve the concerns about presentation and significance, and we can clarify anything further during the discussion.

Thank you again for your time and for the detailed review you provided on our paper. As the discussion period is ongoing, we wanted to check in and see if you have any follow-up questions or if there is any part of our response we could clarify further for you. We are ready to provide any additional information that might be helpful.

We thank Reviewer spAa again for the detailed review. We understand this is a very busy period, and we apologize for following up again. As the discussion period closes in less than one day, we briefly summarize again how the revision addresses the two main concerns. On presentation, Sections 3 and 4 were rewritten so that each paragraph opens with its point, the other main sections were revised in plainer language, and Section 6 moved to the appendix. On significance, the introduction now names the interpretability methods that assume a fitted curve closes (Engels et al., 2025; Wurgaft et al., 2026; Bhalla et al., 2026) and explains how they go wrong for a concept that is only ordered. It also connects the test to a recurring question in representation learning, whether a model inherits the symmetries of the world it describes, which our results answer differently for different concepts. We would be grateful for the reviewer’s feedback on whether these changes address the concerns.

Reviewer_wuRjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work analyse circular properties of representations of cyclic concepts (such as months and days of week), and distinguish it from ordering (such as numbers). The paper argues that standard circularity measures such as the Fourier test cannot distinguish cycles from ordering and therefore they propose another measure, non-circulance, that indicates whether the Gram matrix from the sequence of representations is a circulant matrix or not. They argue that the non-circulance, along with Fourier mode, can distinguish cycles from both ordering and random concepts. They conduct empirical analysis by using 9 language models from 3 families on cyclic concepts and ordered control concepts.

Critical strengths

The paper contributes some practical insights for analysing how modern language models represent cyclic properties such as months, days of the week, etc. The paper argue and empirically validate how Fourier test (fundamental share) cannot distinguish circular representation from an ordering. They propose a measure (non-circulance) and validate that it can be used to fill this gap. Their observation: Fourier mode along with their proposed non-circulance can distinguish cyclic concept from ordering and random, can be useful for other researchers working in this area.

Critical weaknesses

The readability and fluency of the paper can be improved generally. Many parts of sections 3 and 4 discuss and compare the scores; however, I would expect more explanations and interpretations of the results. These sections are overly dense, and that may affect the paper's readability. Section 6 is somewhat disconnected and seems like it can be moved to the appendix for more space.

Questions for authors

The analysis of the paper is based on two separate and somewhat independent metrics: fundamental share and Non-circulance. What are the possibilities for a generalised metric for cyclicality? Is there any way to combine or modify these two for a more generic metric?

Is your identity-invariant version of non-circulance a better measure than your regular non-circulance? Though I couldn’t critically analyse the shortcomings of the identity-invariant version of non-circulance, I got an initial impression that it can be better. It seems like the identity-invariant version of non-circulance distinguishes both random and ordering from cyclic properties. The random reference row in Table 8 with 2K, 4K and 8K principal components has very high and distinguishable values. Why did you base your experiment on the regular version?

The negative fundamental share of compass and season is an interesting observation. Can we say that the higher Fourier energy in mode j=4 in compass and j=2 in season indicates that these are still cyclic with different periodicity? In that sense, can we say the absolute value of the fundamental share is more informative?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer wuRj

We thank Reviewer wuRj for the careful reading and the helpful questions. We appreciate that the reviewer found that the paper shows why the Fourier test cannot separate a circle from an ordering, and that the combined criterion will be useful to other researchers. The reviewer's intuition about the noise-invariant measure was right, and with that measure days and months now pass all 81 Holm-corrected tests.

W1 and W2: Readability and interpretation in Sections 3 and 4

We rewrote Sections 3 and 4, so that each paragraph opens with what its result means and then gives the numbers. Section 3 now ends with its conclusion: each standard test responds to order, and a circle is ordered, so an ordering can pass each of them. Section 4 opens by saying what passing the test means for days: the step from Monday to Tuesday changes the representation in nearly the same way as the step from Thursday to Friday or from Sunday back to Monday, which is not true along a number line or a temperature scale. The paragraphs on layers, token position, clock and compass each start with their conclusion. We also rewrote the abstract, the introduction, the setup in Section 2, the behavioral results in Section 5, the related work, and the Discussion in plainer language, and the introduction has a new paragraph on why closure matters for methods that steer models along fitted curves or look for such curves with sparse autoencoders.

W3: Section 6

We agree, and Section 6 is now Appendix J. Moving it gave us the space for the explanations above.

Q1: A generalized metric

Yes. The noise-invariant ν′′\nu'' works as a single metric: on its own it rejects random points and every ordered control, and it passes 81 of 81 tests. After centering, the Gram matrix of random points is mostly a multiple of I−J/KI - J/K, and that matrix is circulant. Plain ν\nu therefore scores random points as fairly circulant, which is why the paper paired it with a Fourier condition. ν′′\nu'' removes the isotropic part, so random points score 0.96 and a perfect circle scores 0. We also tried circular RSA as a combined metric, and it passes 74 of 81.

Single measure (original frames)daysmonthsclock
ν′′\nu'' alone45/4536/3633/36
circular RSA alone43/4531/3627/36

We still report the rule in two parts, because the parts answer different questions. ν′′\nu'' says whether one step is rigid, and the mode spectrum says which harmonic carries the energy. Compass directions are very circulant under ν′′\nu'', yet their energy sits in mode 4. A single number would place them near a ring, and the two-part rule shows that they are organized by direction type.

Q2: Is the identity-invariant ν′\nu' better?

The reviewer is right that a version invariant to noise is better, and the revision now uses an improved version of ν′\nu'. As Reviewer w8MK also observed, the isotropic part after centering is c(I−J/K)c(I - J/K), so the revised ν′′\nu'' subtracts exactly that part and is fully invariant to such noise. It does what the reviewer expected of ν′\nu'. It rejects random points by itself, it passes all 81 tests, and in a stricter control where all sets share the same sentence frames, it still separates days from the temperature scale. We keep ν\nu as the main measure in the text because it is the standard quantity, the relative distance to the nearest circulant matrix (Chan, 1988), which Du et al. (2026) also use. ν′′\nu'' appears beside it in Table 9 and Appendix K.

Q3: Compass j=4j = 4 and seasons j=2j = 2

These are not cycles of a different length. For eight compass points, mode 4 takes the values +1,−1,+1,−1+1, -1, +1, -1 around the compass, so it separates cardinal from intercardinal directions. For four seasons, mode 2 does the same, pairing spring with autumn and summer with winter. The compass step is close to rigid (low ν\nu), but it acts on two interleaved four-cycles. Behavior agrees with this: in completion prompts, Llama keeps the starting type on 97 to 100% of 45-degree steps, so a cardinal start gives a cardinal answer.

The normalized fundamental share is zero for random points, so a negative value means less energy in the fundamental mode than random points. Its absolute value is therefore not a measure of cyclicity. The full mode spectrum, which we report for every set, shows where the energy goes.

We thank the reviewer again for the questions, which improved the paper, and we can clarify anything further during the discussion.

Thank you again for your time and for the detailed review you provided on our paper. As the discussion period is ongoing, we wanted to check in and see if you have any follow-up questions or if there is any part of our response we could clarify further for you. We are ready to provide any additional information that might be helpful.

Thank you for following up. I appreciate the authors' detailed responses and efforts to address my comments. I don't have any further questions and will maintain my original score.

We thank Reviewer wuRj for the reply and for supporting our paper. The suggestions to move Section 6 to the appendix and to use the noise-invariant measure both improved the revision.

Reviewer_w8MKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines whether apparent circular structure in language-model states reflects a cyclic concept or merely an ordering. It combines Gram-matrix circulance with fundamental Fourier energy and compares cyclic sets with ordered and random controls across nine models. Days and months have lower non-circulance than the ordered controls; clock, compass and seasons provide negative or qualified cases. Behavioral probes show that representation geometry alone does not determine wrap-around performance.

Critical strengths
  1. The controlled diagnostic framework is useful. Explicit counterexamples show why successor-map spectra, Fourier concentration and wrap gaps can mistake an ordering for a cycle. Proposition 1 gives the proposed symmetry criterion a clear mathematical interpretation.

  2. The study includes nine models, ordered controls, layer and projection checks, and uncertainty over sentence templates. The saved outputs regenerate all 11 tables and the numerical summary exactly, supporting the consistency and reproducibility of the primary analysis.

Critical weaknesses
  1. The criterion tests Euclidean rigidity of the measured states, which is narrower than detecting a cyclic feature in a subspace. An ellipse or a circle mixed with lexical features can fail. A circular representational similarity analysis (RSA) baseline is also missing, although the ideal target geometry is closely related. Its inclusion would clarify whether the contribution is better discrimination or diagnostic decomposition. The negative clock and compass conclusions need matched readout controls, and the critique of earlier work should distinguish geometric proxies from intervention evidence.

  2. The empirical qualifications need tightening. Table 11 gives corrected p = 0.105 in three day/temperature7 comparisons. The noise normalization does not remove the centered isotropic component, and diagonal norm differences may drive some contrasts. Section 6 also needs rechecking: protocol2.py subtracts the state mean when projecting finite-difference targets, although differences should be projected without that subtraction. Its cross-talk score is a projected-displacement proxy rather than a behavioral steering experiment. These issues limit the statistical and computational interpretations.

Questions for authors
  1. Please compare circular RSA with the two-part rule using the same template-level evaluation. Does the rule improve discrimination or explain failures more clearly?

  2. Do results survive common carrier frames, consistent span readouts, and unit-norm or off-diagonal Gram controls? Please report clock and compass tokenizations.

  3. Recompute the centered-noise control and use paired inference for matched templates. Distinguish consistent point estimates from the three nonsignificant corrected comparisons.

  4. Why does protocol2.py subtract the state mean from finite-difference targets? Please recheck Section 6 and define operator fitting, cross-talk and the matched-null construction.

  5. Check compass errors by starting type, candidate length and lexical effects, and native chat formatting. Qwen's month accuracies are 0.81 and 0.30 across templates; qualify the model-level contrast.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer w8MK (1/3)

We thank Reviewer w8MK for a careful review. We appreciate that the reviewer found the controlled framework useful and Proposition 1 a clear basis for the criterion. Every analysis the reviewer asked for is now in the revision. The central result holds in each of them, and with the corrected noise control, days and months pass all 81 corrected tests.

W1: Rigidity, ellipses and lexical mixing

We agree that the test asks for a rigid step on the whole state, so we checked both cases the reviewer raised. A circle mixed with random lexical directions of twice its norm still passes plain ν\nu: it scores 0.33, where a line scores 0.87. A strongly elongated ellipse is the harder case, since a 3:1 ellipse scores 0.62. To cover it we added a version of the test that keeps only the plane of the fundamental mode. For each family we project the states onto the real and imaginary parts of the mode-1 Fourier coefficient, whiten the two coordinates, and compute ν\nu in that plane. Any ellipse sampled at equal phases becomes a regular polygon and scores 0, while a line still scores 0.87 and a half-circle arc 0.46. With this measure, days and months pass 79 of 81 tests in the original frames and 76 of 81 in the shared frames described under W3 (Appendix K).

W2 and Q1: Circular RSA

We added circular RSA and ran it through the same frame-level permutation tests with the same Holm correction. RSA correlates the off-diagonal Gram entries with cos⁡(2π(d−e)/K)\cos(2\pi(d - e)/K), which is the Gram matrix of a circle. Used alone, against random nouns and every ordered control, it rejects 74 of 81 tests. The rule with ν\nu rejects 78, and the rule with the noise-invariant ν′′\nu'' rejects all 81.

Second condition uses (original frames)daysmonthsclock
ν\nu (submitted)42/4536/3625/36
ν′′\nu'' (noise-invariant)45/4536/3633/36
circular RSA alone43/4531/3627/36

Most of RSA's misses are in months. The rule also explains failures in a way RSA cannot. A set can fail because its step is not rigid, as an ordering does, or because its step is rigid while its energy sits in another harmonic, as the compass does in mode 4. A low correlation with a circle says only that the set is not a circle; the two conditions of the rule say which of these it is. To answer Q1 directly, the rule gives both better discrimination and a clearer account of why a set fails.

AuthorsOpenReview ↗

Reply to Reviewer w8MK (2/3)

W3 and Q2: Carrier frames, span readouts, norm controls, tokenization

We ran all four controls. For common carrier frames, we placed all 14 sets in the same 20 frames with 5 subject nouns, for example "The teacher wrote [X] on the board." Families with the same frame and subject are then matched across sets, so every comparison becomes paired, and we use the exact sign-flip test over the 20 frame differences (2202^{20} assignments). With ν′′\nu'', days pass 44 of 45 tests, months 26 of 36, and clock 35 of 36, and the whitened plane measure passes 76 of 81. With plain ν\nu, days pass 35 of 45 and months 30 of 36. A single frame can fit nouns, adjectives and clock times alike only if it mentions the word instead of using it, and in such frames the graded scales lose much of their order. Plain ν\nu is low for weakly ordered sets, so it is affected by this. Appendix K reports it next to ν′′\nu'', and Section 4 and the Discussion state the limitation in the main text.

Most of the month comparisons that do not reject in these frames come from a few models. Under plain ν\nu, all six month comparisons that do not reject are against temperature12, the near-random case described above. Under ν′′\nu'', six of the ten that do not reject come from the two Llama models, which fail against all three controls; the rest are two in Pythia-2.8B and one each in Pythia-12B and Pythia-160M. The months themselves are less cyclic when they are only mentioned. Compared with the original frames, their ν\nu rises in every model and their fundamental share falls in eight of nine, and in both Llama models their ν′′\nu'' rises from 0.65 and 0.67 to 0.90, the level of the ordered controls (0.83 to 0.86). One explanation is that the calendar structure depends on months being used as dates ("The report is due in March") and is weaker when a month is only written or read aloud ("The teacher wrote March on the board"). Days, by contrast, pass 44 of 45 tests in the same frames.

For span readouts we read the first token of the word, the mean over its tokens, its last token, and the final token of the sentence. Day and month names are single tokens in all nine models, so the readout cannot change them; it changes only the multi-token controls. In the shared frames, plain ν\nu passes 36, 35, 35 and 23 of 45 tests for days at these four readouts, and 34, 25, 30 and 22 of 36 for months (Table 14). For clock, the first token is the hour word. A causal model reads it before it sees "o'clock", so in the shared frames its state is the state of the number word itself, and clock read there cannot differ from numbers12 (14 of 36 tests pass). Read at the last token of the word, clock passes 30 of 36. The unit-norm control (the correlation matrix) and the off-diagonal control give 77 and 81 of 81, so differences in state norms do not drive the contrasts.

On tokenization, every compass word is a single token in every model. "one o'clock" is three tokens in Llama and Qwen and four in Pythia. The control words with more than one token are eleventh, twelfth, and some temperature words (frigid, balmy, sweltering, scorching, and in Pythia also chilly). Under ν′′\nu'', clock passes 33 of 36 tests in the original frames, and the revision now describes it as borderline.

W4: Geometric proxies and intervention evidence

We agree with this point. The related work now says that our critique concerns geometric evidence, and that the interventions of Engels et al. are separate evidence which we did not test.

W5 and Q3: Noise control, the three p=0.105p = 0.105 comparisons, paired inference

We thank the reviewer for pointing out the centered noise term. After per-family centering, isotropic noise adds c(I−J/K)c(I - J/K) to each Gram matrix. The revised ν′′\nu'' divides by ∥G−tr(G)K−1(I−J/K)∥F\lVert G - \frac{\mathrm{tr}(G)}{K-1}(I - J/K) \rVert_F. That subtracted matrix is the orthogonal projection of GG onto I−J/KI - J/K, so removing it leaves the numerator unchanged and makes ν′′\nu'' exactly invariant to the noise term. It also gives ν′′≥ν\nu'' \ge \nu, so the stability bound of Proposition 2 holds for ν′′\nu'' as well (Appendix C). With ν′′\nu'', all seven comparisons hold in all nine models with 2K2K, 4K4K or 8K8K components and with no projection (Table 9), and random points score 0.96 at every size.

With ν′′\nu'' in the decision rule, the three day/temperature7 comparisons that had corrected p=0.105p = 0.105 all reject, and the primary family passes 81 of 81. Appendix I keeps the two cases apart: under ν\nu those three are consistent point estimates without corrected significance, and under ν′′\nu'' they are significant. For paired inference, days and months share their frames and subjects with the random nouns, and paired tests of the first condition reject in all 18 cases. In the shared frames every comparison is paired.

AuthorsOpenReview ↗

Reply to Reviewer w8MK (3/3)

W6 and Q4: Section 6

We thank the reviewer for checking the Section 6 code. The finite-difference targets are now projected without subtracting the state mean, and the conclusion is the same. Across the 21 concept pairs, the correlation between commutator norm and cross-talk is −0.11, −0.05 and −0.13 in Pythia-410M, 2.8B and 12B, against +0.23, −0.13 and −0.03 for the matched null. The section is now Appendix J, which defines each step. Operators are fitted by ridge regression of finite differences on states, in a 256-dimensional principal subspace shared by all concepts. Cross-talk is the displacement along one concept's operator projected on the other concept's probe direction, divided by its projection on its own probe. The matched null replaces each operator by a random matrix with the same singular values, in five draws. As the reviewer suggests, the appendix calls cross-talk a projected-displacement proxy for interference and says that we did not run a steering experiment.

Q5: Compass errors and Qwen's month accuracy

The compass link holds in completion prompts and is not a length or lexical effect. Both Llama models keep the starting type on 97 to 100% of 45-degree steps, and the rate is the same for cardinal and intercardinal starts (1.00 and 1.00 for Llama-3.1-8B, 0.97 and 1.00 for the instruct model). Every compass word is a single token, so the candidates are matched in length. The errors are also not copies or a default answer: neither Llama model ever answers with the starting direction, and no single direction accounts for more than 15 of the 64 answers to 45-degree steps. These breakdowns come from the per-query records in the supplementary material. With the native chat template, Llama-3.1-8B-Instruct answers 0.25 and 0.22 of 45-degree steps correctly under the two templates and keeps the type in about half of them. Section 5 now states the link for completion prompts and reports the chat result.

On Qwen we agree with the reviewer. Its month accuracy is 0.81 and 0.30 under the two completion templates, and 0.96 and 0.97 with its chat template. Section 5 now says that the Llama and Qwen contrast rests on prompts that Qwen handles unevenly. With the chat template, both instruct models answer month queries almost perfectly and the month penalty is at most 0.05, while a day penalty remains for both (0.22 and 0.29 for Llama-3.1-8B-Instruct, 0.07 and 0.33 for Qwen).

We hope these analyses resolve the reviewer's concerns, and we can clarify anything further during the discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment to all reviewers and the AC

We thank all three reviewers for their detailed and constructive reviews. Reviewer w8MK found the diagnostic framework useful and confirmed that our saved outputs regenerate all 11 tables and the numerical summary exactly. Reviewer wuRj found that the paper shows why the Fourier test cannot separate a circle from an ordering, and that the criterion will be useful to others working on the question. Reviewer spAa found the experiments careful and the documentation thorough, and had no substantive concerns about soundness.

For the rebuttal, we ran every analysis the reviewers asked for, and the main result holds in all of them. In the revised PDF, new results and changed claims are shown in blue, and the new analyses are collected in Appendix K. We also revised the wording of the abstract, the introduction, Sections 2 to 5, the related work, and the Discussion for readability, following Reviewers wuRj and spAa; those wording changes are not highlighted. The supplementary material now includes the code and saved outputs for every new analysis.

We made two clarifications in response to Reviewer w8MK. First, after per-family centering, isotropic noise adds c(I−J/K)c(I - J/K) to each Gram matrix, so the noise control should remove that component. The corrected measure ν′′\nu'' is exactly invariant to it. With ν′′\nu'', all seven comparisons hold in all nine models at every projection size (Table 9), and the decision rule passes all 81 Holm-corrected tests, up from 78. Second, the commutator analysis of the former Section 6 now projects finite differences without subtracting the state mean. Its null result is unchanged, and the section is now Appendix J, with the definitions the reviewer asked for.

The new controls are in Appendix K. Used alone, circular RSA passes 74 of 81 tests, against 78 for our rule with ν\nu and 81 with ν′′\nu''. Restricting the measure to the plane of the fundamental mode and whitening it, which accepts a cycle drawn as any ellipse, gives 79 of 81. Unit-norm and off-diagonal versions of ν\nu give 77 and 81. We also placed every set in the same 20 sentence frames and ran paired tests at four readouts of the word. There, days pass 44 of 45 tests with ν′′\nu'', while plain ν\nu no longer separates the cycles from the temperature scales; we state this limitation in Section 4 and the Discussion. Every compass word is a single token in every model, and Appendix K also reports the behavioral probe with each model's chat template.

Three claims are revised in light of these controls. Clock hours are now described as borderline, because they pass under ν′′\nu'' (33 of 36 tests) but not against the temperature scale under ν\nu. The compass behavior result is now limited to completion-style prompts, since with its chat template Llama-3.1-8B-Instruct does answer some 45-degree steps. The contrast between Llama and Qwen on months is qualified, because Qwen's accuracy differs widely between the two completion templates.

For readability, the introduction now explains why closure matters for methods that steer along fitted curves, and Sections 3 and 4 were rewritten so that each paragraph opens with its point. The measure is linked to its standard name (the circulant distance of Du et al., 2026), and a second new paragraph in the introduction places the question in representation learning.

Our replies below follow each reviewer's numbering, with W for weaknesses and Q for questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.