acceptodds
Under review as a conference paper at ICLR 2027

Learning the Möbius Attractor: From Gradient Descent to Reasoning by Superposition

Abstract

LLM Reasoning often requires keeping several possibilities until later evidence shows which ones matter. Through superposition reasoning, a Transformer can do this by representing a set of candidate states in one hidden state and updating the whole set over several steps. Prior work shows that carefully chosen parameters can represent it, but representation alone does not explain whether gradient training can learn it. We give sufficient conditions under which training learns the full sequence of target intermediate states, rather than only the final answer, and generalizes to new tasks from the same distribution. The conditions have a simple meaning: the model must preserve the current candidate set and be able to produce the next one, while the training signal must directly correct inaccurate intermediate states. We then study a practical problem. When intermediate-state labels are limited, which steps should be supervised? For a stated model of how supervision reduces error, we derive the optimal continuous allocation of a fixed label budget. The solution gives more labels to steps where one label is expected to reduce more error at lower cost, and it motivates a practical rule based on measured improvement. Together, these results explain when reasoning by superposition is learnable and how limited supervision should be distributed across its intermediate states.

Primary area:
learning theory
Keywords:
continuous latent reasoning, superposition reasoning, Transformer learnability, gradient descent dynamics, process supervision, supervision complexity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rf2WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the learnability of reasoning by superposition, where a Transformer represents and iteratively updates a set of candidate states in a single hidden state. The main contributions are (i) sufficient architectural conditions for gradient training to recover the full target trajectory, (ii) sufficient supervision conditions ensuring that gradients correct the earliest inaccurate intermediate state, and (iii) a budget-optimal allocation rule for intermediate-state labels based on marginal error reduction per unit cost. The theory is validated on controlled graph-reachability experiments and a GPT-2-scale experiment comparing several supervision strategies.

Critical strengths
  1. A coherent theory and experiment connection.
  2. Practical supervision allocation result.
Critical weaknesses
  1. Strong assumptions limit the scope of the theoretical claims. The main learnability guarantees rely on substantial architectural and optimization assumptions, including the correction frame, attention margins, smoothness, and gradient-transfer conditions. It is not clear how frequently these assumptions hold in realistic models.
  2. Limited empirical scope. The paper primarily concerns synthetic reachability/superposition tasks. Although the GPT-2 experiment provides a useful additional validation, the results do not yet establish that the proposed principles extend to different reasoning problems or modern LLMs.
Questions for authors
  1. Which of the main theoretical conditions are actually satisfied by the models used in the GPT-2 experiment?
  2. How sensitive is the cost-aware supervision advantage to the assumed error-response model?
  3. Do the same supervision allocation principles hold on reasoning tasks that are not naturally formulated as cumulative set expansion?
Reviewer_15qSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when Transformers can learn reasoning by superposition, representing multiple candidate states in continuous hidden representations. It establishes conditional learnability guarantees under architectural, gradient-conditioning, and intermediate-supervision assumptions, and derives a budget-optimal supervision allocation rule. Controlled experiments examine architectural conditions and supervision efficiency.

Critical strengths

The paper provides a formalization over residual memory, attention retrieval, and intermediate-state supervision, supported by ablations. The cost-aware supervision experiments show potentially useful improvements within the constructed setting.

Critical weaknesses
  1. Weak motivation and limited relevance to natural Transformer learning.

The paper largely prescribes the desired superposition mechanism through orthonormal representations, specialized architectural conditions, and direct intermediate-state supervision. Demonstrating that SGD learns this explicitly supervised construction does not explain why superposition would emerge spontaneously under standard end-to-end training.

A more convincing methodology, exemplified by Nichani et al. (2024), would first identify latent structures actually learned by attention and MLPs, then develop simplified theory explaining those observations. The current experiments mainly validate imposed conditions rather than demonstrate such emergence.

  1. Strong assumptions obscure the actual nonconvex SGD dynamics.

The target-correction condition (Eq. 12) essentially yields a PL-type gradient-dominance inequality, while Theorem 4 further assumes favorable gradient alignment, prefix stability, and restrictive stochastic-gradient bounds. Thus, the central difficulty—why ordinary SGD discovers these representations despite nonconvexity—is largely assumed away.

The discussion should engage more seriously with Tian (2024) on algebraic representations and optimization, Bu et al. (2025) on learned representations in residual Transformers, and existing analyses of stochastic learning dynamics (Vivien et al) where noise could vanish exponentially. The Möbius characterization and standard convex supervision-budget allocation offer limited additional insight without a stronger connection to actual training dynamics.

Questions for authors

Can the authors demonstrate that superposition arises under ordinary final-answer supervision, without prescribed intermediate-state targets or architectural interventions?

Can the PL-type conditioning and gradient-alignment assumptions be derived or empirically verified along unconstrained SGD trajectories rather than imposed?

Which reported experiments use standard GPT-2 rather than the constructed recurrent block? Section 7.1 describes the GPT-2 experiments in future tense.

References

Nichani et al. — How Transformers Learn Causal Structure with Gradient Descent.

Tian — Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Nets.

Vivien et al —— Exponential convergence of testing error for stochastic gradient methods.

Bu et al. — Provable In-Context Vector Arithmetic via Retrieving Task Concepts.

Reviewer_WW3XOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

In this manuscript the authors study the problem: whether and when a Transformer can actually learn “reasoning by superposition” (keeping and updating a set of candidates inside one continuous hidden state). The authors use cumulative set expansion / reachability as the running example. Then the author first propose the architectural conditions, which include a residual path to preserves the current set, attention with a positive support margin that retrieves the frontier, a normalized writer that forms the union, and a target-restricted correction frame. Secondly, the authors propose the supervision conditions, i.e., a usable gradient on the earliest inaccurate state. The authors claim that gradient training recovers the full intermediate trajectory and generalizes under those two conditions and provide the theorems. Furthermore, the authors look at what to do when intermediate-state labels are expensive under a fixed budget: put more labels on steps where one extra label is expected to cut the most error for the least cost. In the experiment section, the authors ran controlled experiments on small Transformers to check whether these architectural pieces really decide whether the model can learn the full trajectory. They also compared different ways of spending a fixed intermediate-state label budget, to see whether the cost-aware rule actually gives lower trajectory error than simpler rules like uniform allocation or always supervising the earliest inaccurate state.

Critical strengths
  1. The work is well motivated and presented. Prior work showed that superposition-style continuous states can represent multi-candidate search, while this paper actually asks when gradient descent recovers the intended trajectory instead of just the final answer. The split between architectural requirements and the need for a usable gradient on the earliest bad state is impressive.
  2. The controlled experiments are well aligned with the claims.
Critical weaknesses
  1. The sufficient conditions seem specialized. It is not so clear how much of this survives once you leave cumulative set expansion or use more realistic embeddings and attention patterns. In the experiment, the scale of the evidence looks limited. Most results are on small controlled models.
Questions for authors
  1. The proposed optimal label allocation seems to rely on the assumption that the error decreases exponentially as more intermediate-state labels are provided. How reasonable is this assumption in practice? Do the experimental results support this pattern, and would the proposed allocation still work if the error decreased more slowly or irregularly?
  2. In the experiments comparing different label allocation strategies, could the authors provide more details about the model sizes, token budgets, and number of intermediate states used? I am also curious why the cost-aware allocation sometimes achieves better accuracy than the full-supervision baseline, despite using fewer resources.
  3. The theoretical results rely on several conditions, such as the attention margin etc. How sensitive are the results to these conditions in practice? Were there any experiments in which these quantities appeared to satisfy the required conditions, but the model still failed to learn the full reasoning trajectory?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.