acceptodds
Under review as a conference paper at ICLR 2027

What does it take to find a feature manifold?

Abstract

There is a growing interest in identifying "feature manifolds" or "multidimensional features" in neural network activations, where computational variables are encoded with some organized geometry in low-dimensional subspaces of activation space. However, the craft of discovering and characterizing these structures in practice remains nascent. In this work, we perform a deep case study of the representation of "character count" in language models, building on the work of Gurnee et al. (2025). We use this setting to explore a variety of questions about the characterization and automatic discovery of feature manifolds. In particular, we describe in detail how sparse autoencoders "tile" the character count manifold and then test whether methods that cluster SAE latents like Bhalla et al. (2026) can re-discover this structure. We find that clustering methods can correctly group together the latents that tile the character count manifold, but using these latents to then visualize raw model activations does not surface clean structure. We then apply these lessons to discover a novel representation of character count that appears on a different data distribution from Gurnee et al. (2025), finding that models represent character count modulo 8, 4, and 2 on strings of hexadecimal characters. We investigate the functional role of these representations and their universality across models. We lastly consider whether these "character count" representations are part of a higher-dimensional joint representation of other computational variables, which leads to some new ways of visualizing model computation. Overall, we hope that our work provides the community with useful lessons for the identification of feature manifolds and a calibrated picture of the difficulty of this task.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, sparse autoencoders, feature manifolds, unsupervised discovery, feature geometry

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_W6RDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This study investigates how language models (LMs) encode numbers of characters in individual lines and is built upon an earlier study proposing the character counting manifolds.

Critical strengths

The authors carefully probe the earlier observation and extend the earlier finding. They outline the potential ways to identify manifolds more systematically and present alternative manifolds for character counting.

Critical weaknesses

Major

As the proposed mechanism is tested on the character counting only, the scope of this study is extremely restricted, which leaves almost no room for generalized tasks. The authors mention the character counting manifold may be ideal for testing new methods, but they did not test their new method against other manifolds. If their method could indeed discover a completely new manifold, it would be of great interest to those who study low-dimensional manifolds. Even rediscovering a previously reported a manifold (different from character counting manifold) would make this study more compelling.

Minor

  • Lindsey 2026 is a post on X, which is not an academic reference. I recommend replacing it with a more formal literature.

  • The figures from the Appendix need to be marked accordingly.

Questions for authors
  • The authors mention only intermediate layers show interesting geometry, which is spiral shape. Why do they consider it more interesting than linear encoding?

  • Only a few manifolds have been observed under the limited contexts, indicating that the observed low-dimensional manifolds could be epiphenomenon rather than the central mechanism underlying LMs. Can the authors provide their thoughts on this possibility?

2 comments
AuthorsOpenReview ↗

Thank you, and some clarifications

Thank you for your review and your questions, and for giving us the opportunity to improve and clarify some things in the paper!

If their method could indeed discover a completely new manifold, it would be of great interest to those who study low-dimensional manifolds.

In Section 3 we indeed applied our method to discover a new manifold (perhaps more accurately a “multidimensional feature” in this case) that has not been reported yet in the literature. In particular, we found that language models represent character count modulo 8, 4, and 2 on strings of hexadecimal characters. We then investigated a variety of questions about this new structure, such as its universality across open-weights models. While this is a character count representation, it appears on a different distribution of inputs from the character count manifold of Gurnee et al. (2025) (hex instead of prose), and the character count manifold of Gurnee et al. (2025) did not exhibit this periodic structure. Do you feel that Section 3 qualifies as discovering a new manifold?

The authors mention only intermediate layers show interesting geometry, which is spiral shape. Why do they consider it more interesting than linear encoding?

In their work first documenting the character count manifold, Gurnee et al. (2025) found character count to be represented along a spiral, and described how the model used this spiral to compare the character count with the line width. They also gave a theoretical explanation for the origin of that spiral geometry. So the fact that character count is represented along a spiral had already been established, and our finding in Section 2.1 is just that there are a narrow range of layers where that spiral can be found, which informs the experiments in the rest of Section 2.

In Appendix E, we also give our own theoretical account of why representations have the curvature they have. As we mention in Section 4, we analyze the character count spiral as part of a higher-dimensional structure that jointly encodes character count and document width. We find that we can reproduce the qualitative shape of this activation geometry from some theoretical analysis that assumes that models linearly encode the probability that each future token position contains a newline (Appendix E.3).

Only a few manifolds have been observed under the limited contexts, indicating that the observed low-dimensional manifolds could be epiphenomenon rather than the central mechanism underlying LMs. Can the authors provide their thoughts on this possibility?

This is a great question but on balance we think the current evidence favors that many of these structures are causally important for model computation, and that understanding their particular geometry is important for doing intervention on model representations. The best examples of this are probably Engels et al. (2024), who show that one can steer the “days of the week” manifold along a circle to change model behavior, and also Gurnee et al. (2025), who perform causal interventions on the character count manifold and find that it affects the model’s prediction of the newline token in the way you’d expect.

Engels, J., Michaud, E., Liao, I., Gurnee, W. and Tegmark, M., 2025, May. Not all language model features are one-dimensionally linear. In International Conference on Learning Representations (Vol. 2025, pp. 84591-84622).

Gurnee, W., Ameisen, E., Kauvar, I., Tarng, J., Pearce, A., Olah, C. and Batson, J., 2026. When models manipulate manifolds: The geometry of a counting task. arXiv preprint arXiv:2601.04480.

I appreciate that the authors have revised the manuscript with more details, and I do understand that the newly proposed method discovered a new type of manifold of character counting, but my major concern is that their method has been tested for a single task of “character counting” only. This does not guarantee that their method can be applied to discover new manifolds for additional different tasks/domains. As we already have an earlier study that found a manifold of character counting, I fail to see the significant value of another type of manifold of the same task. If they could provide evidence suggesting that their newly proposed method can be generally used in other domains, I would recognize the utility of their method, but so far I have not see such evidence.

Reviewer_rMHNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The article discusses a problem that I think is interesting and relevant: how to discover a feature manifold. Overall the article reads a bit like a review article and maybe a little out of place as a conference paper… maybe.. But at least it did not read like ai-generated text (hello human). Still you did keep the reader waiting longer than expected to get to the good parts. The article performs a case study on identifying feature manifolds, provides useful lessons for fellow manifold hunters :), and illustrates well the difficulty of the task. It also contributes some novelty to the “taxonomy of multidimensional features" i.e. the paper considers the question of how character count is represented inside a language model. Eg “Hi how ar” has a character count of 9. They consider activations in certain layers of language models for given inputs and feed these through a sparse autoencoder. The SAE tries to explain the activation using few features and then reconstructs the original activations. They show that several SAE features together describe character counting, with periodic counting patterns that repeat every 2,4,8 characters in hexidecimal strings. They check the counting patterns in different language models and also look for them directly in the model activations. They remove or change the relevant parts of the activations and find that this changes where the model predicts a line break, supporting the idea that the model uses the information. And throughout they give accounts of what worked, what did not, gave impressions of how the work was done and the state that part of the field is in.

Critical strengths

The paper finds counting patterns in the activations and in the features of the SAE, with support that the model is using these patterns. This is a nice illustration of the methods and the ideas and builds on previous work that they also discuss.

The paper goes quite deep into explaining how to find these representations. Of course this is just the beginning and likely these methods will become increasingly useful well beyond the character counting example.

Critical weaknesses

The authors test the methods using a counting pattern that was previously reported. It is less clear how someone would find the right group without knowing what to look for. While fun and interesting, the methods and application are not super novel / earth shattering. It is nice as a review article but less so as a novel contribution (please correct me if I am wrong).

I am not sure the comparison in the removal test is fully fair. The authors remove parts linked to counting and compare this with removing random parts. But the random parts may carry much less activity, so removing them could affect the model less. The counting result is interesting, but a fairer comparison would make the evidence stronger.

Questions for authors

The authors remove parts linked to counting and compare this with removing random parts. But are random parts chosen as having similar levels of activity / joint activity? This is just a minor question.

I have no big questions, things were very clear.

1 comment
AuthorsOpenReview ↗

Thank you!

Hello fellow human! ;) Thank you for your comments and suggestions, we’re glad you found the paper to be “fun and interesting”!

Overall the article reads a bit like a review article and maybe a little out of place as a conference paper… maybe.. But at least it did not read like ai-generated text

We do think that many of our contributions go beyond that of a review paper. While most of the methods we test in the paper were proposed in prior work, we feel that the existing literature overall presents too rosy a picture of how easy it is to apply these methods in practice. We spent a lot of time this year trying to find feature manifolds in language models, found this to be a pretty nuanced problem, and felt compelled to try to communicate this nuance in the format of a big case study. The resulting paper is a bit structurally unusual for an ML conference paper, but we think it will be a valuable contribution nonetheless (see Reviewer N1Je’s “critical strengths”). Some of our novel contributions include:

  • Gurnee et al. (2025) study how crosscoders (similar to SAEs) tile the character count manifold, and find that they give a canonical “featurization” of that manifold. They also find that their latents activate in a very interpretable way, where each latent responds to a narrow and contiguous range of character counts. In contrast, we discover that the featurization depends strongly on the sparsity of the SAE, so the story about the tilings being “canonical” is in fact more nuanced. Perhaps more importantly, we also find that some of our latents respond to character count in a pretty messy way, activating on multiple disconnected ranges of character count. Overall, this adds extra (but nevertheless new!) evidence that individual SAE latents are often not sensible “units” to decompose model representations into.
  • Perhaps our most important methodological result is that applying feature manifold discovery pipelines automatically doesn’t yield a clean visualization of the character count manifold. We recognize that this experiment is somewhat limited because it only tests the re-discovery of a single manifold. A more systematic investigation would have been better, but also there just aren’t that many clean manifolds to use as ground truth at the moment! We think that the failure we observe is an interesting point and deserves to be studied in future work.
  • The periodic multidimensional feature we report in Section 3, to our knowledge, hadn’t been documented before, and so we think all our observations in that section are novel!
  • We also include a novel theoretical analysis that provides an explanation for why the higher dimensional manifold might show the geometry that it does. In short: we predict a particular geometry for the centroids based on a toy version of the newline prediction task. We find that the centroid arrangement in Gemma-2-9B has a striking qualitative similarity to this prediction in the first three principal components (see figures 24,25). Additionally, this analysis suggests the prediction tasks require the model to organize its activations primarily by characters remaining on the current line. We find evidence that the model increasingly adopts this organization until saturating around layer 15. Currently, these results are referred to in the main text in Section 4, but mostly contained in the appendix. We plan to use our extra page to expand the section on higher dimensional structure, which will more clearly highlight this contribution.

The authors remove parts linked to counting and compare this with removing random parts. But are random parts chosen as having similar levels of activity / joint activity? This is just a minor question.

This is a great point. In the last couple weeks, we have tried to run some experiments on this, and have found that the causal role of the periodic representations is somewhat more nuanced than we previously realized. We are continuing to run some more experiments, and will post another response if these finish in time. In any case, we will be sure to address this in the final manuscript.

Reviewer_N1JeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a rigorous, candid, and insightful empirical and theoretical investigation into feature manifold discovery in Large Language Models (LLMs). Using the character-count / line-breaking manifold (Gurnee et al., 2025) as a central case study, the authors systematically ask what it actually takes to discover and visualize a low-dimensional feature manifold using Sparse Autoencoders (SAEs). Beyond providing a crucial methodological reality check—showing that while graph clustering over SAE latents recovers the right features, projecting raw activations onto their decoder subspace captures only 1.6% of activation variance and yields an uninterpretable noise cloud without task-specific centroid averaging, the paper makes two compelling scientific discoveries: 1. Discrete Modular (mod 8, 4, 2) Manifolds, and 2. 2D (c, w) Surface & Forecast Geometry.

Critical strengths

A much-needed reality check on "automated" manifold discovery: Recent papers make it sound like clustering SAE latents automatically reveals clean geometric manifolds. Showing that L₀ sparsity (not dictionary size) drives SAE tiling, and that the recovered cluster span is still 98.4% noise without human-guessed centroids, is a very valuable takeaway for the interpretability community.

Neat discovery of modular (mod 8, 4) manifolds with causal backing: The octagon and square manifolds on hex dumps (Fig. 5) are visually striking, and the subspace projection ablation (Fig. 7—where projecting out the periodic planes drops newline probability at c=48 from 0.99 to 0.84 and causes off-by-one errors at c=47) cleanly proves the model relies on them.

Great bridge between layer-wise geometry and automata theory: Connecting the shift from c (Register View) to r = w - c (Forecast View) to formal Hankel rank bounds and Forecast Embeddings—and showing that width-conditioned steering μ(c^* ∣ w^*) shifts where the model wraps lines while unconditioned centroids μˉ(c∗)\bar{\mu}(c^*) fail—makes the mechanistic story feel complete.

Critical weaknesses

The best parts of the paper are buried in the appendix: This is my biggest complaint with the current draft. The 9-page main text reads mostly like a qualitative case study, while the actual meat—the layer-wise variance decomposition (Fig. 27), the counterfactual steering experiments (Figs. 28–30), and the Stochastic Counter Machine / Forecast Embedding theory (Appendices E & F)—are all hidden in a 42-page appendix. Section 4 is barely 1.5 pages long and feels way too thin without the quantitative results from Appendix E.

Questions for authors

Would you consider trimming parts of Sec. 2.1–2.2 to move the variance decomposition (Fig. 27), the width-conditioned steering results (Fig. 28), and a short summary of the Forecast Embedding theorem into Section 4?

1 comment
AuthorsOpenReview ↗

Thanks, glad you liked the paper!

Thank you for your comments and suggestions! We’re very happy you found the paper to be “rigorous, candid, and insightful”, and a “much-needed reality check” on the difficulty of finding feature manifolds–this was one of our central goals in writing the paper :). Our response will be brief.

The best parts of the paper are buried in the appendix… Would you consider trimming parts of Sec. 2.1–2.2 to move the variance decomposition (Fig. 27), the width-conditioned steering results (Fig. 28), and a short summary of the Forecast Embedding theorem into Section 4?

Thanks for this suggestion and the opportunity to improve the paper! We agree that Section 4 is currently too thin in the paper. In the final version, our intention is to use the extra page primarily to expand Section 4 to include additional discussion of higher-dimensional structure and our toy/theoretical analysis. We will still need to refer readers to the appendix for the full details, but expect that we will be able to give a better treatment of those topics. Most likely we will keep most of section 2.1 and 2.2, though this is a tricky decision.

As an additional note: we did feel quite constrained by the page limit when writing the paper. Over the last month, we’ve been expanding the paper into a much larger blog post that brings some appendix content of the paper into the main text, including much of our toy model/theory analysis that you mentioned. The medium of a blog post is especially helpful for this work because it allows both for interactive figures and for the main text to be longer. We intend for the paper and the blog to ultimately link to each other, but expect that most readers will want to read this work in the blog format.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.