acceptodds
Under review as a conference paper at ICLR 2027

Reasoning on the Simplex: Geometric Fixed-Point Models

Abstract

Looped reasoners spend test-time compute by iterating a weight-tied map, but a small residual does not mean the state is a fixed point when that map lives in unconstrained latent space. We propose Geometric Fixed-Point Reasoning (GFPR), in which the iterated state is the prediction itself: a field of categorical beliefs on a product of simplices, whose is the answer at every step. Because the state is a belief, task structure can be imposed on it directly by replacing the product of simplices with an instance-dependent compact convex set, such as a Sudoku consistency polytope, a unit-flow polytope, or the Birkhoff polytope; the update remains a continuous self-map of that set, so a fixed point exists for any parameters. At about 7M parameters, GFPR reaches 95.1% exact match on Sudoku-Extreme, 92.0% on Maze-Hard, and 100% sequence accuracy on length 128, above the published FPRM numbers at the same scale. The same update also trains a 201M language model on FineWeb-Edu in which each site is a distribution over the vocabulary; with 24 Picard steps it is above GPT-2 small on four zero-shot multiple-choice tasks and above GPT-2 medium on ARC-Easy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
fixed-point iteration, probability simplex, looped transformers, iterative reasoning, deep equilibrium models, test-time compute

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jzdjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Geometric Fixed-Point Reasoning (GFPR) method whose state is a field of categorical beliefs and whose answer is the argmax of that state. Experiments on Sudoku-Extreme, Maze-Hard, and S5 state tracking demonstrate that the accuracy of GFPR outperforms several baselines.

Critical strengths
  1. Better accuracy than baselines on several tasks.
  2. The limitations of the proposed method have been discussed.
Critical weaknesses
  1. This paper claims that a fixed point exists; however, what is the relationship between the fixed point and the final prediction?
  2. The author mentioned convergence many times. Does better convergence indicate better prediction for those tasks? Any risks of overfitting?
  3. In Table 2, the compared baseline is GPT-2. How about the accuracy of more recent methods, such as GTP-6?
  4. What are the effects of λaux\lambda_{aux} and λres\lambda_{res}?
  5. Many references are not complete; the journal or book title information is missing.
Questions for authors

Please refer to the weaknesses.

Reviewer_ifKxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2(+1)
Read review
Paper summary

This paper proposes Geometric Fixed-Point Reasoning (GFPR), a looped reasoner whose recurrent state is not a free latent vector but a field of categorical beliefs: one probability vector per site over the output symbols plus a few auxiliary coordinates. Each step applies a weight-tied Transformer, maps the result back to the product of simplices with a softmax, and takes a damped Picard step, so the current answer can be read off at any iteration and a fixed point exists by Brouwer's theorem. Task structure can additionally be imposed through convex relaxations. The paper reports results above published FPRM numbers at a similar scale on Sudoku-Extreme, Maze-Hard and S5 state tracking, and trains a 201M-parameter language model with the same update that is compared with GPT-2 on zero-shot multiple-choice tasks.

Critical strengths
  1. A clean and interpretable design. Making the iterated state the prediction itself is a simple idea, and it gives useful properties: the answer is readable at every step, the residual directly measures how much the answer moves, and failures can be inspected without a decoder (e.g. on Maze-Hard 99.67% of parent pointers are correct and 99.7% of settled fields trace a connected route, so most failures are valid alternative routes). The diagnostic in Sec. 2, showing that FPRM's halted states are not fixed points on the examined puzzle, is also a useful observation for this line of work.
  2. Strong results across several settings. The reported numbers on Sudoku-Extreme (95.1%), Maze-Hard (92.0%, 96.0% with the unit-flow constraint) and S5 (100.0%) are competitive at about 7M parameters, and the language-model experiment, while small, shows that the same update can be trained at a much larger vocabulary. The paper is also open about its limitations in the conclusion.
Critical weaknesses

1. The benefit of the simplex state is not isolated. All baselines in Table 1 are quoted from Movahedi et al. (2026) and Jolicoeur-Martineau (2025), not re-run, and the training recipes differ (for example, GFPR uses dihedral augmentation on Maze-Hard and FPRM does not, and the S5 model uses a causal convolution that the paper notes is not part of the original TRM). The paper acknowledges that Table 1 "does not by itself isolate the effect of the state space" (Sec. 4.2). The natural control is mentioned in the paper itself: Sec. 3.1 notes that a bounded latent z∈[−1,1]dz \in [-1, 1]^d updated by tanh⁡(fθ(z;x))\tanh(f_\theta(z; x)) also has a fixed point. A comparison with this bounded-latent variant, using the same network, training and inference budget, is the experiment needed to show that the gains come from iterating beliefs rather than from the training recipe. It is not reported.

2. The theoretical guarantee is limited. The fixed-point guarantee is existence via Brouwer's theorem, which holds for any continuous self-map of a compact convex set and says nothing about uniqueness, reachability or correctness. Whether the iteration reaches the correct fixed point depends entirely on training, and the paper states that "none of this certifies convergence at test time" (Sec. 3.2). This matters because training rollouts are about 32 steps on Sudoku while test budgets reach 35,000 steps, and some puzzles settle to a stable but incorrect grid. For Sudoku and Maze, the convex constraints of Sec. 3.4 are applied only as a terminal readout, so the existence argument for the constrained recurrent state applies only to S5.

3. The empirical validation is incomplete in several places.

  • The fixed-point claims rest on very little evidence. The contribution list states that GFPR "settles to a fixed point while FPRM typically halts at a state that is not one". For GFPR, the only evidence is the single puzzle in Fig. 2. The paper reports no population-level residual statistics for GFPR, and does not say how many puzzles end in a stable but wrong grid. For FPRM, the evidence is the same single puzzle plus a power-iteration probe on 128 puzzles that, as stated in Sec. 2, has not been cross-checked with Arnoldi.
  • The convex constraints are only partly supported. They help on Maze-Hard (92.0% to 96.0%) but not on Sudoku (94.99% vs. 94.95%), and the Sudoku number is on a 23,680-puzzle sample rather than the full test set used for the headline result.
  • The language-model experiment does not support the claims made about it. It is not compute-matched: each Picard step is a full pass over the sequence, so 24 steps cost roughly 24 times a GPT-2 forward pass. The training data also differ: GFPR-LM is trained on FineWeb-Edu, an education-filtered corpus known to raise scores on knowledge-oriented benchmarks such as ARC and SciQ, while GPT-2 was trained on WebText. The appropriate baseline, a standard Transformer of the same size trained on the same 6B tokens, is not reported. Even so, GFPR-LM trails GPT-2 medium on three of four tasks, and App. F reports that GPT-2 large is higher on all four but omits it from the main table. The evaluation uses only multiple-choice likelihood; the wavefront decoder of Sec. 3.3 is never evaluated on generation. App. F also states that these scores "do not show that the simplex state has reached a fixed point", so the paper's central property is not verified in this setting.

Beyond these points, from my own perspective, what matters most about looped models is that they can iterate a free hidden state that carries whatever intermediate computation a problem needs. In GFPR, the only quantity carried from one step to the next is the belief state pp (Eq. 2): each step recomputes the Transformer's hidden activations from Winp+Enc(x)W_{\text{in}} p + \text{Enc}(x) and then discards them after the softmax, so all intermediate computation must pass through per-site probabilities plus a few auxiliary coordinates. To me this is a step back in expressiveness rather than an improvement. The evaluated tasks, small puzzles with a fixed set of choices per position, are where this restriction matters least, and the paper does not ablate the number of auxiliary coordinates or test a task that needs richer state. For a design that deliberately narrows what the state can represent, I would expect either a fundamental argument for why this is the right restriction, or evidence beyond toy-scale settings.

Overall, I cannot lean towards acceptance in the paper's current form.

Questions for authors
  1. Across the full Sudoku-Extreme test set, what fraction of GFPR runs reach a residual below a fixed tolerance, and what fraction settle to a stable but incorrect grid? The fixed-point claim in the contributions currently rests on the single puzzle in Fig. 2.
  2. How does GFPR compare with the bounded-latent variant mentioned in Sec. 3.1 (tanh⁡\tanh state of matched width), trained with the same recipe, augmentation and inference budget, on Sudoku-Extreme and Maze-Hard? This is the comparison that would isolate the effect of the simplex state.
  3. Can you re-run at least FPRM or TRM under your training recipe (including augmentation), so that Table 1 compares models rather than published systems?
  4. How does GFPR-LM compare with a standard Transformer of the same size trained on the same 6B FineWeb-Edu tokens? How does it perform on any generation or multi-step reasoning task rather than multiple-choice likelihood?
  5. How does performance change with the number of auxiliary coordinates AA, and how does GFPR behave on a task that needs richer intermediate state than a fixed per-position choice?
Reviewer_963oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GFPR (geometric fixed point reasoning), a method for reasoning through repeated updates to probability distributions over possible answers, rather than unconstrained hidden states. This formulation aims to guarantee that a fixed point exists and allows constraints to be incorporated into the model. The method achieves good results on tasks like sudoku, maze solving, and permutation tasks, and the authors also demonstrate a small language model using the same approach.

Critical strengths

(1) The idea of performing iterative reasoning directly in a space of probability distributions is interesting and seems decently motivated. The connection to fixed-point theory provides a useful mathematical perspective.

(2) Also, the paper evaluates several different reasoning tasks and includes interesting analyses of convergence behavior. The experiments suggest that the approach can be effective across different problem structures.

Critical weaknesses

(1) In my opinion, the experimental comparisons do not seem fully controlled. In particular, differences in data augmentation, model architecture, and inference budgets between comparisons make it difficult to isolate the benefit of the proposed representation.

(2) The language modeling experiments also seem fairly limited. For example, the authors compare against GPT-2 using up to 24 full computation steps rather than a compute-matched baseline, and do not evaluate actual text generation with the proposed decoding procedure.

Questions for authors

Could the authors provide a more controlled comparison against methods like FPRM, using the same training setup and inference computation budget? This would help isolate the benefits of the simplex representation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.