acceptodds
Under review as a conference paper at ICLR 2027

MindCraft: A Full-Sequence Flow Theory of How Concepts Take Shape in Deep Models

Abstract

Two views describe how a language model represents a concept: as a linear direction or as a curved manifold. Both characterize a representation once it has formed; neither describes how a distinction is carried through depth or predicts which geometry it will take. We introduce MindCraft, which treats depth as time and follows a one-token counterfactual edit through the full-sequence flow operator, the linearized map from the edited embedding to every token at every layer. Each layer stretches and turns the edit, and how the turning is distributed over depth decides the geometry: one sharp turn yields a linear direction, gradual turning yields a manifold. We show that the flow starts isotropic and sharpens at a rate set by the first layer, and that, under a growth gap between answer-relevant and answer-irrelevant directions, each input's flow is driven toward the task read-out subspace. Across four domains and six models from three families, task-central distinctions branch early while temporal and surface details branch late or never. Steering at the branching layers lowers hallucination on HalluLens by 14.5 points and ambiguous-context bias on BBQ to 0.01–0.04.

Primary area:
interpretability and explainable AI
Keywords:
Representation, Linear Representation Hypothesis, Large Language Models, Representation Manifolds

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qRQTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MindCraft, a dynamic framework based on full-sequence flow operators, designed to investigate how concept representations emerge across the depth of large language models. By analyzing the propagation of counterfactual perturbations through stretching and turning operations, the authors seek to unify the perspectives of linear and manifold-based representations. Furthermore, they propose a branching layer to construct hierarchies of concepts. Experiments conducted across various models and tasks demonstrate the potential value of this approach in mitigating hallucinations and biases.

Critical strengths

1.The paper introduces a dynamic framework for understanding how concept representations evolve across model layers, offering an interesting perspective beyond static representation analysis. 2. The proposed full-sequence flow framework provides a systematic mathematical treatment of counterfactual propagation, spectral dynamics, and representation geometry. 3. The experiments cover multiple model families and task domains, demonstrating consistent correlations between branching layers and causal effects. 4. The proposed branching-layer-guided steering method shows promising improvements in reducing hallucinations and social biases, highlighting the potential practical value of the framework.

Critical weaknesses
  1. Theorem 4.3 relies on strong growth-gap assumptions, but it is unclear whether these assumptions hold in practice. The experiments show a decreasing off-target/on-target ratio, but do not directly demonstrate convergence to the task read-out subspace.
  2. The link between turning patterns and linear or manifold representations is not fully established. The current results are suggestive, but more direct evidence is needed to support this claim.
  3. The steering experiments lack comparisons with fixed-layer, random-layer, and layer-sweeping baselines. Without these comparisons, it is hard to tell how much of the improvement comes specifically from choosing the branching layers.
  4. The branching-layer definition depends on a similarity threshold, which may affect the selected layers. It would be useful to see how sensitive the results are to this threshold, especially for downstream steering.
Questions for authors
  1. Can the authors provide more direct evidence that the growth-gap assumptions in Theorem 4.3 hold in the evaluated models? How should we interpret the gap between the theoretical convergence result and the observed off-target/on-target ratios?
  2. What evidence supports the claim that different turning patterns lead to linear versus manifold representations? Could the authors provide a more controlled experiment to test this relationship?
  3. How does MindCraft compare with fixed-layer or random-layer steering under the same experimental settings? Is selecting the branching layer necessary to achieve the reported improvements?
  4. How sensitive are the branching layers to the choice of similarity threshold? Does changing this threshold significantly affect the downstream steering performance?
Reviewer_p9LxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes MindCraft, a framework that explains how concepts form across the layers of a language model. It traces how the representation difference caused by a one-token edit propagates across layers. It uses this framework to bridge two views of concept representation, LRH and RM. However, the paper does not show consistent and clear evidence that supports these claims. Overall, the experiments are not systematic. They read like a collection of separate findings under specific settings.

Critical strengths

Studying the dynamic representation of concepts in deep models is a timely and important research direction.

Critical weaknesses
  • LRH and RM are two views on how a model represents a concept. It is unclear whether they correspond to two different forms of a concept.
  • The abstract does not clearly define stretching and turning.
  • The paper treats each single-token swap as a concept, which is too coarse. One word often carries several attributes. A token swap can therefore change several attributes at once. This makes the resulting concept change hard to isolate. Each concept is also defined inside a single base prompt.
  • The evidence that the paper bridges the two views is weak. There is still a large research gap before the paper can bridge LRH and RM. At a minimum, the paper should show that the two views correspond to two geometric forms that can transform into each other. The internal logic that links the sections is also not strong. Appendix J.1 shows that the per-layer turning angle has almost the same profile for all concepts. This does not support the claim in the abstract that one sharp turn yields a linear direction and gradual turning yields a manifold. Appendix J.2 also reports that the geometry of a concept is largely inherited from the embeddings of its values. This suggests that turning is not the main factor that decides the geometry.
  • The geometric analysis in Section 6.3 adds limited new insight. Prior work has already reported geometric structures for concepts such as months, weekdays, and numbers. The coherence analysis also needs a small set of values. Each value must be a single token.
  • The experimental setup is not clean. The counterfactual pairs are generated by an LLM and restricted by tokenizer constraints. They are not derived from a clear definition of each concept. The selective-stretching experiment relies on a single pair of honesty and control edits. The shared-turning regime is shown only for the rating scale on Qwen2.5-0.5B. Prediction 5.2 fails for the size and temperature scales on all three models. In Section 6.1, the off/on ratio never reaches the range where Theorem 4.3 applies. The steering results are compared only with the unsteered model. There is no comparison with standard steering methods that choose the layer by sweeping. On the benchmarks, there is also no comparison with steering at other layers.
Questions for authors
  • What does stretching explain beyond relative measures? The edit is a difference between two residual streams. The norm of the residual stream grows with depth anyway. How much of the growth of an edit comes from this general norm growth? After normalization, the absolute size of an edit also matters little. The branching layer is defined by a cosine.
  • Do the results for a concept hold when the same token swap appears in other base prompts?
  • How does the coherence analysis extend to generic concepts such as honesty or sentiment? These concepts do not have a small set of single-token values.
Reviewer_CvcDOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduced a theory to identify how a concept was activated across layers by swapping out one word at a time. For each single word swap (e.g., honest → untruthful), the difference between the two runs is tracked layer by layer, whereas layers are viewed as time steps. Each layer's effect is described as stretching and turning that difference, and a branching layer is defined as the first layer at which the difference dominated the last token's state. The authors predict that the difference would align with the answer-relevant directions, and they frame the linear-direction and manifold views of concepts as two regimes of turning. Experiments on six open models (sizes 0.5B-14B) revealed branching layers that organize concept hierarchies and enable steering on HalluLens and BBQ.

Critical strengths
  1. The idea is simple and well motivated. Following a single-word swap through the layers is a simple, principled way to ask when a model started to treat a distinction as relevant, and it ties together the linear-direction and manifold views of concepts.
  2. Coordinates were standardized before cosines were computed, so a few high-magnitude dimensions would not dominate the comparison.
Critical weaknesses
  1. The evidence does not support the theory's claims about trained models. The guarantee of Theorem 4.3 required an off-target/on-target ratio of at most 0.5. Still, the measured ratio stayed at 3.4-6.9 on all six models, and on two of the six (Qwen2.5-7B and 14B) the predicted alignment with the answer subspace did not appear. The claimed link between the linear and manifold views is shown only once, for ratings, only on the smallest model. The main empirical result was also partly built in: the branching layer and the causal effect were both computed from the same last-token difference, so some correlation was expected by construction, and a simple naive baseline was not tested.

  2. The Concept Hierarchy is the paper’s main interpretable output, but it is shown only qualitatively for a handful of prompts. The edits are proposed by an LLM, while the distinction between "central" and "surface" concepts is left to the authors' interpretation. This deserves substantially more systematic analysis.

  3. The only downstream evaluation is a single steering experiment, compared only against the unsteered model. It does not compare against existing steering methods (e.g., RepE, CAA, or ITI, all cited), test alternative layer choices, or report error bars. Important implementation details are also missing, including the counterfactual pairs used, steering strength, layer cutoff, and number of test questions, despite the Reproducibility Statement promising that hyperparameters and data splits would be provided in Sec. M.

  4. The figures were hard to read. The font is very small for Figs. 1 and 2.

Questions for authors

Clarifications:

  1. Given that the bound of Theorem 4.3 was not met on any model, what did the theorems establish about trained models?
  2. For steering, which counterfactual pairs were used, and what were the steering strength, layer cutoff and number of test questions?

Experiments that could change the assessment: 3. A baseline for the branching layer, such as the first layer where the norm of the last-token difference crossed a threshold. Would it correlate with the causal effect as strongly? 4. Steering compared against other methods (RepE, CAA, ITI) and against other layer choices, with error bars. 5. A direct test, across concepts and models, of the claim linking one sharp turn to a line and gradual turning to a curve (shown so far only for ratings on Qwen2.5-0.5B). 6. A more systematic analysis of the Concept Hierarchies: across many prompts (or datasets), which kinds of distinctions branched early, and did the pattern hold beyond the examples shown?

1 comment

The authors did not engage in a discussion, so my concerns were not addressed and thus I am keeping my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.