acceptodds
Under review as a conference paper at ICLR 2027

Spokes, not a Ring: Revisiting the Weekday Manifold in Language Models

Abstract

Recent work has hypothesised that large language models may represent some concepts as multi-dimensional manifolds instead of single directions. Using these manifolds to interpret or steer representations often assumes that moving along them corresponds to a continuous change in the represented concept. In this work, we test this continuous manifold assumption on the prototypical weekday manifold. Despite its striking circular geometry, we find that the weekday representations behave more like seven discrete states arranged in a circle than a continuous ring. First, we show that local sensitivity has seven minima aligned with the weekdays, producing a step-like mapping between representation and behaviour under on-manifold steering. Second, we test whether any concepts are represented along the manifold outside the seven discrete states, and find neither time of day, uncertainty, nor natural text populate the intermediate space. Together, these results challenge the continuous interpretation of the weekday manifold and motivate testing whether movement along other representation manifolds correspond to meaningful changes in model representations and behaviour.

Primary area:
interpretability and explainable AI
Keywords:
representation manifolds, mechanistic interpretability, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_P9HBOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tests the circular arrangement of weekday representations in language models and explores what the spaces between the seven days mean. Moving along a curve fitted through these representations produces small changes near each weekday and larger changes between weekdays. The authors also examine whether time of day, uncertainty between days, or natural text produces representations in these intermediate regions. These findings seem to favour seven distinct weekday regions over a ring with continuous meaning.

Critical strengths
  1. The paper asks a practical question about whether a clear geometric pattern also has a meaningful interpretation. It makes a distinction between observing a circular arrangement and showing that the model uses the space between the observed points.

  2. The authors go beyond visualising representations and test how interventions affect the model.

  3. Several experiments check alternative explanations for the findings, including time of day, uncertainty, and natural text.

Critical weaknesses
  1. The contribution seems too narrow for the broader claims the paper aims to make. The study focuses on weekdays and mainly revises the interpretation of an existing finding. Even if the results hold, it remains unclear what general insight they provide beyond this particular case.

  2. Weekday names provide only a coarse description of an underlying continuous temporal variable. Learning their order and cyclic relationships does not necessarily mean that interpolating between their representations will produce meaningful intermediate states. The paper does not sufficiently separate the effects of how temporal information is expressed in the training data from the properties of the learned representation itself. Moreover, even if the observed weekday representations lie on a continuous manifold, a spline fitted through these sparse points need not follow that manifold between them. The results therefore leave open whether the intermediate responses reflect a discrete representation or a mismatch between the fitted path and the underlying geometry.

  3. The analysis remains mostly descriptive and does not explain why the observed structure emerges. The paper shows stable regions around weekdays, but does not identify the computations that produce these effects or the conditions under which they arise.

Questions for authors
  1. Could the author test concepts other than weekdays?

  2. If we sample time more irregularly and densely using continuous daytime and fit the curve through those representations, do the same sensitivity and occupancy patterns remain? Furthermore, what would be the case for other types of continuous variables that are not time-based?

  3. How sensitive are the conclusions to the choice of interpolation method used to connect the observed representations?

2 comments

we thank the reviewer for their constructive review and for engaging with the specifics of the method.

  • on using a single manifold: we agree, we chose the weekday manifold as it has been used as the canonical example of continuous manifold steering/probing in past work (Wurgaft et al, Engels et al) so it was the natural place to test that interpretation. our secondary contribution consists of a pair of diagnostic measures (sensitivity + occupancy) that can be applied to other manifolds, we are actively using these to examine color which has natural continuous structure, to see if our methods generalise.

  • on dense time sampling: figure 2 does use continuous clock time (22 hours x 7 days) and we found that clock-time variation lies almost entirely outside the weekday plane (99.8% of hour-centroid variance, the leading clock time axis is 88 degrees from the plane) so fitting a curve through time sampled points would trace a direction orthogonal to the ring rather than filling its arcs. the small in-plane component is comparable to that induced by non-temporal controls (fig 2b).

  • on sensitivity to interpolation method: we address this concern by evaluating the Jacobian field over a polar grid covering the whole weekday plane rather than along the spline alone, and the seven minima extend radially from the centroids (fig 1c, appendix b). in Llama and Mistral the sevenfold structure also appears along radial and random orthogonal perturbation directions rather than just along the tangent and the result is also reproduced on arithmetic prompts from Wurgaft et al (appendix C). hence we don't think a different spline would change the conclusion.

  • on separating training data from learned representation, and mechanism: fair point, the paper is descriptive by design and does not explain why the spoke structure emerges. that said, the diagnostics should generalise to more complex manifolds where such mechanisms might be hard to identify in the first place.

Thank you for the response. I maintain my score.

Reviewer_dCUHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this manuscript, the authors revisit the representation of the days of the week within LLM latent space, a now canonical example of features that exhibit a non-linear geometric representation. They point out that prior work has made a “continuous manifold assumption,” assuming that the days of the week sit along a smooth manifold with meaningful and naturally observed interpolations between the individual days. They provide evidence that the representations of the days do form a circular shape, but are discrete, resulting in a form resembling spokes rather than a ring. Steering and prompting towards intermediary times do not correspond to intermediary points in the representation space.

Critical strengths

I found the paper to be clear and well written. It does a good job of making the conceptual distinction between continuous manifolds and non-linear geometries with discrete representations. The analyses are appropriate, well executed, and provide good evidence for the underlying claims.

Critical weaknesses

The manuscript’s biggest weakness, in my opinion, is that the contribution is small. The representation of weekdays is not, in itself, particularly important. It has indeed become a canonical example of non-linear representation in LLM latent space, and this does motivate using it for this analysis. But ultimately distinguishing whether the days of the week are discrete or continuous is not of major practical importance. I understand the authors are using it as an exemplary case of how seemingly continuous representations may in fact be discrete, but I am still unsure what is gained from the exemplar. Is there a line of thought that assumes that all representations are continuous, and this therefore serves as an existence proof against that? Or is the idea that you are providing a methodology that can be used across instances of non-linear feature geometries to distinguish whether they are continuous or discrete? The big takeaway from the paper needs to be clearer.

Another weakness is that the results are not entirely surprising. 11:59pm on Wednesday is still 100% Wednesday, it’s not a mixture of Wednesday and Thursday. Days as conceptual categories are themselves discrete; there is no meaningful interpolation between them. It is therefore not a big surprise that these discrete concepts are represented discretely.

Questions for authors

I have two recommendations that would substantially strengthen the paper. The authors construct a sensitivity map by placing artificial activations at different locations around the weekday ring and measuring the Jacobian norm at each location. I recommend they collect a separate, held-out set of natural text examples, run the model on those examples without intervening, and record the activations at the same layer and token position where they performed the interventions. Plotting those observed activations on the sensitivity map would show whether the empirically occupied locations coincide with the low-sensitivity regions. They could then quantify whether natural activations concentrate in low-sensitivity regions and avoid high-sensitivity regions.

Second, I would like to see this same analysis also applied to a circular but continuous variable, such as color. For each case, they could measure sensitivity at points throughout the region containing the circular representation, including the spaces between observed values, and then independently examine where activations elicited by natural inputs fall within that region.

The question is whether weekday representations occupy separate stable regions, while intermediate colors occupy a continuous band, and whether the sensitivity analysis successfully distinguishes these patterns. The color inputs should span intermediate hues rather than only named color categories. If naturally occurring color representations also traverse regions of high sensitivity, this would challenge the inference that high sensitivity identifies out-of-distribution activations.

On the other hand, successful discrimination would substantially strengthen the paper. It would demonstrate a method for distinguishing categorical from continuous representations with similar geometry, rather than primarily confirming the categorical structure of weekdays.

2 comments

We thank the reviewer for their thoughtful review of the paper and constructive suggestions. The 2 recommendations in particular are very helpful and in line with our planned follow up work.

  • on takeaways: we agree, and the answer is a bit of both. recent work treats fitted manifold across a variety of concepts as continuous objects to probe/steer along and the weekday ring has been the flagship example. our main contribution is to show that on this example that this continuity does not hold under scrutiny. our secondary contribution is a pair of diagnostic methods (sensitivity + occupancy measures) that can be used to ask the same question of other manifolds. we agree the framing should lead with this and will do so in future versions

  • on results being unsurprising, we partly agree though the discrete reading was not the consensus in the literature, Engels et al explicityl interpreted temporal modifer shifts as the ring encoding time between days, and Shai et al's belief geometry suggests arcs might encode uncertainty between days. neither turn out to hold, which is not something one could have concluded from the fact that Wednesday is discrete.

  • on rec 1: section 4.3 does somrthing close to this, using held out FineWeb excerpts read at the same layer and position as the interventions, and finding that they collapse onto spokes after removing context-dependent translation. we agree that overlaying these activations directly on the sensitivity map would be a more direct test.

  • on rec 2: we flag colour as the right continuous analog in section 5.2 as well, and agree it would validate the method further. we are unable to add this in this cycle but it is the extension we are most excited about.

Thank you for these responses. I still believe that a broader set of tests across more cases than just weekdays (including some cases in which we see continuous representation and some discrete) would be necessary for me to more highly rate this paper. I therefore maintain my current rating.

Reviewer_BessOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper analyzes the circular structure of weekday representations in language models and asks whether its interpolated regions correspond to meaningful intermediate states, such as time of day. They argue that the experimental results suggest that the weekday geometry consists of seven discrete spokes rather than a continuous ring.

Critical strengths

Solid experimental analysis.

Critical weaknesses
  • The paper focuses on one manifold that has been studied quite extensively in prior literature. The study would have been stronger if broadened to other cases of circular/periodic structure beyond weekdays and showing consistency of the findings across data sets.
  • The circular structure is constructed using a periodic-spline construction, which aligns with earlier literature. It would have been nice to look at variations of the construction, e.g., interpolation within the weekday plane and paths constrained by natural activation neighborhoods.
  • No theoretical results are given.
Questions for authors

see weaknesses

2 comments

We thank the reviewer for their careful read & assessment that the experimental analysis is solid.

  • On the scope being limited to one manifold, we agree and say as much in section 5.2. We chose the weekday manifold precisely because it has been the subject of recent manifold steering work (Engels et al, Wurgaft et al) and we agree that the natural next step is to extend it to other cases (color is an obvious candidate, containing natural continuous structure) however that remains out of scope for this submission.
  • On variations of the construction, we used the periodic spline constructiion to remain consistent with past literature, but checked that the spokes are not an artefact of this by evaluating the Jacobian field (fig 1c, appendix B) over a polar grid spanning the whole weekday plane, and examining activations to natural text (section 4.3). We agree that constraining paths to natural activation neighborhoods would be a good further test.
  • on theory: fair point, the paper is empirical by design.

Thank you for the response. I maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.