acceptodds
Under review as a conference paper at ICLR 2027

Spherical Fixed-Point Reasoners: Adaptive Loop Halting on the Hypersphere

Abstract

Looped (weight-tied) transformers enable adaptive computation by allocating more iterations to harder inputs. Fixed-Point Reasoners (FPRM) achieve this with a damped fixed-point solver, but operate in an unbounded Euclidean residual space with coupled rescaling and relative-norm halting. We introduce Spherical Fixed-Point Reasoners (SFPRM), which performs the same computation on the unit hypersphere of normalized transformers (nGPT). Input injection, damping, and halting become geodesic operations, yielding boundedness by construction and eliminating coefficient coupling. We prove fixed-point existence, local convergence under an explicit step-size condition, and global convergence under strong injection. On Sudoku-Extreme and Maze-Hard, SFPRM achieves % and %, respectively, outperforming FPRM and other baselines while allocating test-time compute more effectively. Its converged geometry is also interpretable: token–input angles capture puzzle structure and predict correctness.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
looped transformers, adaptive computation, fixed-point reasoning, adaptive halting, hyperspherical representation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xirGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SFPRM transfers the idea of restricting activations and most Transformer weights to a unit-sphere to Fixed-Point Reasoners, eliminating the need of (learned) residual scaling. Constraining updates to the latent reasoning state z to the unit-hypersphere guarantees boundedness of z, which potentially improves the iterability of the recurrent reasoning model beyond its training horizon. As all updates to z_i are damped rotations on the unit-hypersphere, their spherical nGPT block proposes a new state z_i^{\tilde}, the update uses spherical linear interpolation to rotate z_{i+1} towards it by a fraction \eta_i. Every step, z is partially (coefficient \lambda ) rotated towards the embedded spherical representation of the input x. \eta_i is initialized per sample at 1 and decreases geometrically (multiplication by \gamma < 1) whenever the maximum token-wise chordal residual did not improve for P iterations. The authors provide theoretical guarantees for convergence to and the existence of a unique fixed point under sufficiently strong input-injection and evaluate convergence empirically. Instead of the relative scale of the residual (as in FPRM), in SFPRM, the maximum proposed (undamped) angular update to a token's representation is used as a stopping criterion. They find SFPRM halts earlier than FPRM on average while simultaneously achieving higher accuracy across Sudoku and Maze.

Critical strengths

The application of nGPT's spherical computations to fixed-point reasoners seems like a natural improvement to Fixed-Point reasoners and elegantly lifts the necessity of (learned) residual scaling while keeping magnitudes bounded by design - this could credibly improve stability under extended iterations of their spherical reasoning block. Instead of stability under extended iteration counts, they find their reasoner requires fewer iterations to match FPRMs performance.

Their proofs of fixed-point existence and convergence under sufficiently strong input-injection and damped updates on the hypersphere are strong theoretical contributions, even if their solver operates in regimes where these conditions might not hold.

The comparison of alternative fixed-point solvers is particularly informative, demonstrating that inference efficiency can potentially be improved considerably without retraining the model.

The paper is written very clearly and well structured.

Critical weaknesses

The main weakness is that apparently all experiments were only done for a single seed run - I hope the authors can show the significance of their findings across various training seeds, currently differences to FPRM could worst-case also be attributed to lucky training runs.

Their current experimental setup does not allow clean conclusions about relative wall-clock efficiency nor contributions of individual adaptations, several choices are changed simultaneously: Transformer->nGPT, AdamW->Muon for weights, AdamW for other params, input-injection: affine->Slerp, relative residual halting-criterion -> angular, Sudoku: 1000x1000aug->5000x200aug.. While some choices probably need to be varied jointly (spherical-repr.+halting-criterion), geodesic input injection, damping, optimizer and regularization could be isolated. In particular, I'd like to see the individual contribution of just nGPTs normalization vs full geodesic operations. While faster convergence in terms of number of #iterations is desirable, wall-clock/throughput/FLOP comparisons could clear doubts about improved efficiency. Also their mean halting costs ignore samples that failed to halt, and ignore the fact that a halted sample needs to continue until the whole batch terminates in their implementation (acknowledged in the paper). Especially defining the halting iteration as the iteration after which the (chordal) residual STAYS below its threshold differs from the criterion in Algorithm 1 and further complicates interpreting actual inference costs. Furthermore, the parameter-count matched comparison in App. C.1 tab. 4 severely fails to reproduce FPRM's results, which severely limits clear conclusions about the architectural advantage of SFPRM. Additionally it is unclear why it uses 12k epochs instead of 15k used for the reported final SFPRM accuracy of 96.6%.

The regime where their Theorem 2 holds seems to converge to fixed-points which no longer correspond to the best solutions. Figure 4 suggests a tension between fixed-point uniqueness and and high accuracy, specifically increasing input injection improves consistency across initializations, while reducing accuracy. This questions the practical relevance of theorem 2.

Questions for authors

FPRM specifically motivated its pre-normalization and residual-scaling architecture by the signal-propagation limitations of post-normalized recurrent transformers like TRM. SFPRM now uses nGPT to recover unit-norm by construction. Could you add a comparison of post-norm TRM/FPRM, pre-norm FPRM with residual scaling, and SFPRM across increasing training and inference depth (similar to FPRM paper figure 7), to test, whether SFPRM retains FPRM's improved signal propagation properties?

Do you have a hypothesis, why your reproduction of FPRM achieves only 78.4% accuracy compared to the published 94.2%? Without either a successful reproduction on 5k x 200 of FPRM or an uncollapsed 1k x 1k SFPRM, a fully controlled comparison remains difficult.

4 comments
AuthorsOpenReview ↗

Official Response (1/2)

Thank you for your detailed review and feedback. We ran every experiment you asked for that fits in the rebuttal period and provide detailed responses to your main concerns below one by one. Tables A and B are in the general response.

W1. Single seed. We ran 3 seeds per model on all three tasks (including the new Mini-Arc task we added during this rebuttal). On Sudoku the gap is far outside the seed variance: 95.5 ± 2.5 against 82.0 ± 4.0, with all SFPRM seeds above every FPRM seed (Table A in the general response). On Maze-Hard (Table C below), SFPRM best of 3 runs 87.9 (mean ± sd: 87.3 ± 0.9) outperforms both the official reported results in the FPRM paper (87.0%), as well as our reproduction using the official code release (best of 3 runs: 85.0; mean ± sd: 83.6 ± 1.6). On Mini-ARC, we report the best of three runs per method (59.1 against 57.0) next to the means, which are on par (54.8 ± 4.0 against 55.3 ± 2.5).

Table C. Maze-Hard and Mini-ARC, 3 seeds per model. FPRM is retrained by us using the official code release. FPRM's published Maze-Hard number is 87.0, whilst the best exact accuracy we managed to get by running their official code was 85%. We also added Mini-ARC during this rebuttal as a new benchmark. As the task is transductive, we we pick the best out of 3 evaluations and report the mean ± sd for both FPRM and SFPRM. We have updated Table 2 in the revised paper PDF.

taskSFPRMFPRM
Maze-Hard, exact accuracy (%), best (mean ± sd) of 3 runs87.9 (87.3 ± 0.9)85.0 (83.6 ± 1.6)
Mini-ARC, pass@1 (%), best (mean ± sd) of 3 runs59.1 (54.8 ± 4.0)57.0 (55.3 ± 2.5)

W2a. Confounds; nGPT normalisation alone against the geodesic operations. We ran the ladder you describe, 3 seeds per rung (Table A).

  • Optimizer. Muon makes FPRM erratic: against AdamW one seed loses 1.0 points, one gains 5.6, one collapses. Its best seed reaches 95.1 on FPRM's schedule, so the optimizer may explain part of the gap on a good seed, but not on average.
  • Dropout. It lowers FPRM's accuracy on every seed with AdamW, and with Muon FPRM does not train.
  • nGPT normalisation alone. Our block and recipe, with normalised affine mixing in place of Slerp and FPRM's relative-norm residual in place of the chordal one. Two seeds score 97.0 and 96.9, matching SFPRM that uses the geodesic operations on the same settings. The third seed fails at evaluation, but none of the SFPRM onse fails this way. Therefore, the geodesic operations add no accuracy but are the more robust choice.

We also add other ablations on the Mini-ARC task, which we added during this rebuttal period (Table B in the general response). Replacing Slerp or the halting rule marginally decreases performance. The highest drop comes from ablating the QK-normalisation.

W2b. Wall-clock and FLOPs. We measured both metrics and added the following table in the revised paper PDF.

Table F. Cost per step on one A100.

SFPRMFPRMratio
one solver iteration, Sudoku, batch 76812.3 ms9.6 ms1.28x slower
forward FLOPs per sequence per iteration, Sudoku1.36 G1.36 Gequal

The extra time is in the normalisations and the interpolation, which the FLOP counter does not count. Using the nGPT operations instead of the geodesic ones, as the reviewer suggested, lowers the one solver iteration time to 11.4 ms. The training and inference overhead are repaid in faster halting at test time though, as well as in improved performance.

W2c. Halting cost: non-halting samples, batch-level halting, "stays below". Thank you, all these points are fair. For each of the three seeds of each model, we ran the solver on the same test puzzles (batches of 768) for 5,000 iterations without halting, using the default tolerance (τc=0.01\tau_c = 0.01 for SFPRM, 0.1 for FPRM). At every iteration we recorded each sequence's residual. From these traces we computed, per sequence, the first iteration at which its residual is below the tolerance ("first below", the rule of Algorithm 1) and the first iteration from which it remains below until the end ("stays below"), and per batch the iteration at which the implemented rule would stop, which requires all sequences of the batch to be below the tolerance at once.

Table G. Sudoku 5k x 200: halting cost at the default tolerances (mean of 3 seeds).

SFPRMFPRM
sequences that get below the tolerance97.0%98.9%
mean step, "first below" (Algorithm 1), these sequences only31289
same, the others counted at the cap of 5,000452144
sequences that then stay below96.6%89.9%
mean step, "stays below", these sequences only449805
batch-level rule as implemented, batch 768the capthe cap
AuthorsOpenReview ↗

Official Response (2/2)

  • Sequences that never halt. For SFPRM, 2 to 4% of the test puzzles never reach the tolerance within 5,000 iterations.
  • Batch-level halting. Our implementation stops a batch only when all of its sequences are below the tolerance. On Sudoku every batch of 768 contains at least one sequence that never halts, so every batch runs to the iteration cap, for both FPRM and SFPRM.
  • "First below" and "stays below". We use one definition throughout: a sequence halts at the first iteration whose residual is below the tolerance, as in Algorithm 1. For SFPRM the choice hardly matters, because almost every sequence that gets below the tolerance stays below it. For FPRM, 7 to 10% of the sequences get below the tolerance and later rise above it again.
  • Implications to cost. At its default tolerance SFPRM halts later than FPRM (312 against 89 iterations in Table G), but that default is stricter than it needs to be. With a looser tolerance, τc=0.08\tau_c = 0.08, SFPRM halts after 78 iterations on average and reaches 93.8% accuracy, while FPRM at its default needs 120 iterations for 79.7% (Table K in our response to Reviewer EqXb). SFPRM is then 14 points more accurate and also faster in wall-clock time. On Maze-Hard SFPRM halts earlier than FPRM at the default tolerances as well.

W2d. 12k against 15k epochs. We trained both models for 12k (78,120 steps) and 15k epochs (97,656 steps). At 12k epochs, SFPRM scores 97.7 ± 0.2 against FPRM's 85.5 ± 4.8 at 15k epochs (Table A in the general responses).

W3. Theorem 2 regime. We apologize for the theorem not being clear. The theorem guarantees uniqueness of the fixed-point under a strong input injection, however it does not say anything about the quality of the solution. In Fig. 4 we show a trade-off between uniqueness of the fixed point and accuracy for a sweep of the input injection.

Q1. Depth. For inference depth, SFPRM is ahead of FPRM at every depth and improves monotonically from 16 to 5,000 iterations (Table H below).

Table H. Sudoku 5k x 200: accuracy after a fixed number of solver iterations, no halting.

iterations1632641282565121,0245,000
SFPRM59.5 ± 1.374.8 ± 1.083.2 ± 1.188.4 ± 0.992.3 ± 0.694.2 ± 1.395.1 ± 1.995.3 ± 2.1
FPRM54.7 ± 1.867.0 ± 2.073.3 ± 2.077.3 ± 2.779.7 ± 3.580.9 ± 4.081.4 ± 4.281.6 ± 4.1

Q2. Why 78.4% against 94.2% for FPRM? We refer the reviewer to Table I below for the results of our reproductibility efforts. FPRM's public code release gives 79.2 ± 1.7. Running FPRM on 5k x 200, similarly as SFPRM, scores 82.0 ± 4.0 with the public code release. We also ran FPRM using he hyperparameters reported in the paper appendix for Sudoku, which they differ from the ones in the code release, and we obtained 85.5 ± 4.8 test performance.

Table I. FPRM in its published setting, Sudoku-Extreme 1k x 1000.

FPRM modelFPRM eval repoFPRM eval paper
released checkpoint, full test set88.694.2
retrained, current public code and config79.2 ± 1.784.5 ± 2.4

Thank you again for your detailed review and we hope the new experiments resolve your concerns. We are available to engage in further discussion in case of any further concerns or questions. Otherwise, we would appreciate if you would consider increasing your score.

Reviewer_xirGOpenReview ↗

Remaining concerns following the rebuttal

I appreciate the extensive additional experiments, but two important concerns remain.

1. Best-of-three reporting, unmatched training data, and FPRM reproduction

Reporting the best of three seeds in the main results is questionable, particularly for small differences. On Mini-ARC, SFPRM's best run outperforms FPRM (59.1% vs. 57.0%), while its mean is actually lower (54.8 ± 4.0% vs. 55.3 ± 2.5%). Means and standard deviations should be the primary results.

Furthermore, the headline Sudoku comparison (97.9% vs. 94.2%) uses different training builds (5k × 200 vs. 1k × 1000). Table 9 reveals that this choice is consequential: on the original 1k × 1000 build, reproduced FPRM outperforms SFPRM (84.5% vs. 78.2%), reversing the ranking on 5k × 200. Since the authors still cannot reproduce FPRM's published 94.2% through retraining, the matched comparisons demonstrate superiority over an underperforming reproduction, while the headline comparison remains confounded.

2. Idealized halting costs and unequal iteration costs

Despite the new measurements, the contributions (p. 2, lines 78–79) still claim SFPRM "halts up to 2.7× earlier", and the RQ2 conclusion (p. 9, lines 454–455) asserts earlier halting on both Sudoku and Maze.

However, Table G reveals that every Sudoku batch of 768 runs to the iteration cap under the implemented stopping rule, so the idealized per-sequence halting savings are not realized in practice.

Additionally, SFPRM iterations are 28% slower than FPRM iterations on Sudoku (12.3 vs. 9.6 ms, Appendix E.5). Thus, fewer iterations do not necessarily imply lower computational cost. The reported Maze improvement (108 vs. 130 iterations, only 17% fewer) lacks corresponding wall-clock measurements. On Sudoku, the default SFPRM tolerance can even require substantially more iterations.

While the relaxed-tolerance experiments demonstrate promising accuracy–iteration tradeoffs, they do not establish improved end-to-end inference efficiency. I therefore suggest qualifying the efficiency claims and clearly distinguishing idealized per-sequence costs from realized computational savings.

Given the updated version, my main concerns actually strengthened.

AuthorsOpenReview ↗

Addressing remaining concerns

Thank you for the careful follow-up. Below we address your remaining concerns.

1. Seeds, unmatched training data

Means only. We understand your concern, however we chose to use the best of 3 runs in Table 2 as all the other baselines from the respective papers also report a single seed. In the revision we report the means ± sd in the appendix. For the camera-ready version we will also train every baseline with three seeds and report its mean ± sd.

Headline against the published model, not a reproduction. On the full test set, under the FPRM paper's schedule (0.997, 10) and FPRM's budget of 12k training epochs, SFPRM scores 97.7 ± 0.2%. FPRM's released checkpoint scores 94.22% under the same protocol. The remaining difference is the unique training puzzles (the size of the training set is the same). As you note, the ranking depends on the training set build. In our runs, FPRM gains at most 1 point from training on the 5k × 200 build, while SFPRM needs it to avoid over-fitting the 1k puzzles. We also note that in Table 4 in the appendix there is already evidence that SFPRM can match or outperform FPRM with the 1k x 1000 build too when using the best checkpoint (87% test accuracy), however, in our evaluation, we strived for a training hyperparameter setting that would avoid utilizing early stopping but rather the final checkpoint.

model5k × 2001k × 1000
SFPRM, learned λ\lambda97.7 ± 0.278.2 ± 2.4
FPRM, public recipe85.5 ± 4.884.5 ± 2.4

2. Halting cost and efficiency

We agree that the per-sequence halting costs in the paper are idealized, and we now report measured wall-clock time instead.

Maze-Hard: halting saves compute, measured end to end. One of FPRM's ten batches runs to the 1,000-iteration cap:

stopping ruleSFPRMFPRM
worst-token (default)87.7%, 5.3 s per batch84.2%, 6.4 s per batch
mean-token87.5%, 0.39 s per batch83.8%, 0.86 s per batch

With the mean-token residual suggested by Reviewer D6NB, SFPRM halts each batch after about 22 iterations instead of 300 (note that this is not the 108 mean halting steps per maze, which is idealized): 14× less time for 0.2 points of accuracy. Under the same rule it needs 2.2× less time than FPRM and is 3.8 points more accurate, although each of its iterations costs 1.2× (ms) an FPRM iteration.

Sudoku: halting saves no compute, for either model. Our evaluator stops a batch only when all of its sequences are below the tolerance. On Sudoku, however, every batch of 768 contains some sequence that is not, so every batch runs to the iteration cap, for both FPRM and SFPRM. The per-sequence savings in the paper would need an evaluator that replaces each converged sequence with a new one, which neither method implements.

Fixing the iteration budget. We also run a batched evaluator that instead runs every batch for a fixed number of iterations. Its cost is then that number times the measured time per iteration, with nothing idealized. Measured this way, SFPRM is much faster than FPRM despite its slower inference time per iteration:

  • Same time, higher accuracy: at every budget SFPRM is 7 to 14 points more accurate, and every SFPRM seed is above every FPRM seed.
  • Higher accuracy in far less time: after 96 iterations (1.18 s per batch) SFPRM is at 86.6%, already above the 81.6% FPRM reaches after 4,096 iterations (39.3 s). That is 33× less time for 5 more points.
  • A level FPRM never reaches: with this schedule FPRM's mean never exceeds 82%, even in the full 35,000-iteration evaluation, while SFPRM averages 90.8% after 2.4 s.

Sudoku 5k × 200, checkpoints of Table G, schedule (0.99, 5), batch 768 on an A100. FPRM gets about 4% more time than SFPRM in every row:

wall-clock per batch, SFPRM / FPRM (iterations)SFPRMFPRM
0.59 / 0.61 s (48 / 64)80.3 ± 1.173.3 ± 2.0
1.18 / 1.23 s (96 / 128)86.6 ± 1.077.3 ± 2.7
2.36 / 2.46 s (192 / 256)90.8 ± 0.879.7 ± 3.5
4.73 / 4.92 s (384 / 512)93.6 ± 0.880.9 ± 4.0
37.8 / 39.3 s (3,072 / 4,096)95.3 ± 2.181.6 ± 4.1

We hope that with these results we have addressed your remaining concerns and we want to thank you again for your valuable feedback that ultimately helped us improve our paper and claims.

Reviewer_EqXbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SFPRM, a spherical version of FPRM that combines a weight-tied nGPT backbone with geodesic input injection, angular damping, and a chordal-residual stopping criterion. The construction keeps token states at unit norm and removes FPRM’s coupled residual-scaling constraints. The paper provides conditional fixed-point existence and convergence results, evaluates Sudoku, Maze, and ARC tasks, and analyzes solver dynamics and task-related information in the geometry of hidden states.

Critical strengths
  1. The spherical formulation is coherent and well motivated. It provides bounded states by construction and gives a clear angular interpretation of input injection, damping, and stopping tolerances.
  2. The paper investigates the behavior of the recurrent solver beyond final accuracy, including initialization dependence, alternative solvers, and hidden-state trajectories. These complementary analyses provide useful empirical insight into convergence, halting, and the geometry of the learned states.
Critical weaknesses
  1. Fairness of the Sudoku comparison. The strongest result in Table 2 uses 5,000 distinct training puzzles with 200$\times$ augmentations, whereas the cited FPRM result uses 1,000 puzzles with 1,000$\times$. Despite equal total sample counts, the diversity of underlying puzzles is not controlled. Optimization settings also differ. The matched-data experiments in Table 4 are useful, but the reproduced FPRM scores of 78.4–78.5% are far below the cited 94.2%, leaving uncertainty about the strength of the controlled baseline.
  2. Attribution of the practical benefits remains incomplete. Normalization, geodesic updates, and angular halting are introduced together, making their individual contributions difficult to assess. The accuracy and efficiency advantages also depend on the configuration, as illustrated by the fixed (λ=0.1\lambda=0.1) and learned (λ=0.47\lambda=0.47) settings in Figure 9. Further analysis would help distinguish architectural benefits from the effects of solver configuration and stopping tolerance.
Questions for authors
  1. Could you explain why the FPRM runs in Table 4 substantially underperform the published result, and clarify whether the Sudoku advantage persists with a competitive FPRM baseline under matched training-data and optimization settings?
  2. Using the default adaptive-damping solver, could you vary the stopping tolerance on a small shared Sudoku subset and compare accuracy versus mean model evaluations for FPRM and the two SFPRM checkpoints in Figure 9 (fixed λ=0.1\lambda=0.1 and learned λ=0.47\lambda=0.47)? Other targeted experiments that help clarify the attribution concerns in Weakness 2 would also be welcome.
3 comments
AuthorsOpenReview ↗

Official Response

Thank you for the review. We ran the matched comparison and the tolerance sweep you asked for, and both strengthen the case. Tables A and B are in the general response.

W1. Fairness: 5k x 200 against 1k x 1000. You are right that equal sample counts are not equal data. Table J below has both builds for both models, 3 seeds each. On 5k x 200, with data and steps matched, SFPRM is circa 11 points better than FPRM. On 1k x 1000, SFPRM reaches 87% at a third of FPRM's steps and similar to our reproduced FPRM performance from the official code release, but collapses later on.

Table J. Both Sudoku builds: FPRM eval repo / paper.

buildSFPRMFPRM, public recipeFPRM, restored settings
5k x 20095.5 ± 2.5 / 96.6 ± 2.982.0 ± 4.0 / 85.5 ± 4.880.3 ± 6.4 / 84.3 ± 7.6
1k x 100074.4 ± 1.6 / 78.2 ± 2.479.2 ± 1.7 / 84.5 ± 2.483.3 ± 1.7 / 89.6 ± 2.7

W2. Attribution; fixed against learned λ\lambda. Table A in the general response separates the changes, 3 seeds per row.

  • Optimizer and dropout. Neither helps FPRM reliably: Muon makes it erratic (one seed collapses), dropout lowers it on every seed, and with both it does not train.
  • λ\lambda. Learned λ\lambda scores 96.5 ± 1.3 against 95.5 ± 2.5 for λ=0.1\lambda = 0.1, and learns 0.34 to 0.39. In the revised version of the paper, we use the learned λ\lambda for Sudoku-Extreme as well.
  • One component at a time. We run these ablations on the Mini-ARC benchmark (added during this rebuttal period). In Table B of the general response and the revised paper PDF (Table 6) we show these results. Replacing Slerp or the angular halting rule stays inside the range of the three runs. The highest cost is disabling the QK normalization.

Q1. Why does FPRM underperform, and does the advantage persist? The baseline is explained in Table I below: it is what the public recipe gives under our schedule, and we now also report FPRM's own schedule, under which it scores highest. The advantage persists under every matched control on 5k x 200:

controlSFPRM, FPRM eval repo (0.99, 5)FPRM, FPRM eval repo (0.99, 5)SFPRM, FPRM eval paper (0.997, 10)FPRM, FPRM eval paper (0.997, 10)
matched data and training steps95.5 ± 2.582.0 ± 4.096.6 ± 2.985.5 ± 4.8
FPRM with our optimizer95.5 ± 2.582.5 / 17.0 / 90.796.6 ± 2.982.4 / 17.3 / 95.1
FPRM with our dropout95.5 ± 2.564.4 ± 19.296.6 ± 2.967.8 ± 19.8
FPRM with the released checkpoint's settings95.5 ± 2.580.3 ± 6.496.6 ± 2.984.3 ± 7.6
SFPRM with 20% fewer steps than FPRM95.8 ± 0.782.0 ± 4.097.7 ± 0.285.5 ± 4.8

The one exception is a single seed: under the FPRM paper protocol the best FPRM + Muon seed reaches 95.1, above SFPRM's worst seed there (93.3).

Table I. FPRM in its published setting, Sudoku-Extreme 1k x 1000.

FPRM modelFPRM eval repoFPRM eval paper
released checkpoint, full test set88.694.2
retrained, current public code and config79.2 ± 1.784.5 ± 2.4

Q2. Tolerance sweep. Thank you for this suggestion. In Table K below we provide the results of running the evaluation using different tolerance thresholds on both Sudoku. SFPRM's accuracy moves by at most 0.7 points for τc\tau_c from 10−510^{-5} and 0.01 (default) to 0.04, while the cost falls from 616 and 389 to 153 evaluations to halt.

Table K. Sudoku 5k x 200: stopping-tolerance sweep. Each sequence halts at the first iteration whose residual is below the tolerance (Algorithm 1) and is decoded there. Non-halting sequences are counted, at the step where the step-size floor stops them.

modeltoleranceaccuracy at haltmean model evaluations
SFPRM, λ=0.1\lambda = 0.10.1690.1 ± 1.341
0.0893.8 ± 0.578
0.0495.1 ± 0.3153
0.01 (default)95.8 ± 0.5389
10−510^{-5}95.7 ± 1.4616
SFPRM, learned λ\lambda0.0894.3 ± 2.289
0.01 (default)95.9 ± 1.6328
10−510^{-5}96.1 ± 1.4548
FPRM0.269.2 ± 10.839
0.1 (default)79.7 ± 4.4120
0.0581.3 ± 4.4297
0.012581.6 ± 4.1558

Thank you again for your review and we hope that your concerns have been addressed with our new experiments, and we are glad to run further and engage during the rest of the discussion phase. We would also appreciate if you would consider raising your score.

Thank you for the rebuttal. Table J shows that SFPRM has a substantial advantage on the 5k × 200 training set, but the ranking reverses on the original 1k × 1000 setup. The ablation also suggests that the geodesic updates may help with stability. However, two of the three runs without them also reached comparable accuracy. The tolerance sweep shows that SFPRM can retain high accuracy with fewer evaluations, but Tables F and G show no speedup under the default batch setting. These results give me a clearer picture of where SFPRM helps, but I will maintain my rating.

Thank you very much for reading the rebuttal closely. Below we add the results that address your remaining concerns.

1. Fairness in comparison to FPRM. We agree that the ranking depends on the training build. Exact accuracy (%) on Sudoku-Extreme, mean ± sd over three seeds, under the FPRM paper's evaluation schedule (0.997, 10):

model5k × 2001k × 1000
SFPRM, learned λ\lambda97.7 ± 0.278.2 ± 2.4
FPRM, public recipe85.5 ± 4.884.5 ± 2.4

Both builds contain one million training examples, they differ in the number of unique puzzles. FPRM gains at most 1 point from the more diverse 5k × 200 build, while SFPRM needs it to avoid over-fitting the 1k puzzles. With early stopping, SFPRM matches or outperforms FPRM on the 1k × 1000 build as well (Table 4 in the appendix), but our main comparison uses the final checkpoint. On the full test set, with the same protocol and the same number of training steps as FPRM's published model, SFPRM scores 97.7 ± 0.2%, against 94.22% for FPRM's released checkpoint.

Maze-Hard has a single training set, so the build question does not arise there. With both models trained in the same setup, every SFPRM run is above every FPRM run: 87.3 ± 0.9 for SFPRM vs. 83.6 ± 1.6 for FPRM.

2. Ablation. We agree with you regarding the geodesic update, and we think it supports the paper's central claim. Without the geodesic operations, but with the nGPT approximations, the state still lives on the unit sphere. The two successful runs without the geodesic operations are level with SFPRM and about 15 points above FPRM, so the gain over FPRM comes from the spherical state itself, not from the specific update formula. The geodesic updates are though the more robust choice: 1/3 runs failed w/o them, against 0/3 w/ them.

3. Speed. On Sudoku every batch contains a sequence that has not converged, for both FPRM and SFPRM, so it runs to the iteration cap for both models. However for a fixed iteration budget, as shown below, SFPRM is faster.

Fixed iteration budget. A batched evaluator runs a fixed number of iterations, with nothing idealized in its cost. On Sudoku, checkpoints of Table G, and batch 768 on an A100, FPRM gets about 4% more time than SFPRM in every row, so they are roughly within the same time allocation:

wall-clock per batch, SFPRM / FPRM (iterations)SFPRMFPRM
0.59 / 0.61 s (48 / 64)80.3 ± 1.173.3 ± 2.0
1.18 / 1.23 s (96 / 128)86.6 ± 1.077.3 ± 2.7
2.36 / 2.46 s (192 / 256)90.8 ± 0.879.7 ± 3.5
4.73 / 4.92 s (384 / 512)93.6 ± 0.880.9 ± 4.0
37.8 / 39.3 s (3,072 / 4,096)95.3 ± 2.181.6 ± 4.1

At the same wall-clock time SFPRM is 7 to 14 points more accurate, and every SFPRM seed is above every FPRM seed. After 1.18 s it is at 86.6%, above the 81.6% FPRM reaches after 39.3 s: 33× less time.

Batch-level halting on Maze-Hard, measured end to end. One checkpoint per model, 1,000 mazes, batch 100, A100; one of FPRM's ten batches runs to the 1,000-iteration cap:

stopping ruleSFPRMFPRM
worst-token (default)300 iterations, 5.3 s per batch440 iterations, 6.4 s per batch
mean-token22 iterations, 0.39 s per batch59 iterations, 0.86 s per batch

With the mean-token rule suggested by Reviewer D6NB, SFPRM's batches halt 14× sooner for 0.2 points of accuracy. They also halt 2.2× sooner than FPRM's under the same rule, while SFPRM stays about 4 points more accurate.

We hope these results make concrete where SFPRM helps, and we would be grateful if you would take them into account in your final assessment.

Reviewer_D6NBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper puts looped transformer fixed-point solving onto the unit hypersphere using an ngpt backbone (sfprm). instead of doing input injection, damping, and halting in unbounded euclidean space like fprm, they do everything with geodesic slerp updates and stop based on the worst-token chordal residual. that keeps token norms bounded at 1 by default without messy scaling tricks, gives nice fixed-point convergence guarantees, and beats fprm on sudoku-extreme, maze-hard, and mini-arc while stopping faster and giving final token angles that actually predict if the answer is right.

Critical strengths

this paper explores a way of doing loop on hypersphere of normalized transformers (nGPT).

since nGPT uses hyperball, it applies slerp instead of linear (convex) combinations. current experiment results look good, the background theory looks solid.

Critical weaknesses

this paper has lots of figures, i like them, but they look very claudish!

The idea is simple and kinda overlapped with existing looped transformer works, just hyper-ball on top of it, and here it uses slerp instead of linear damping.

the experiments are very toyish, not big scale jobs or scaling up.

did not use multi random seeds for experiments, for looped model training, the result variance is larger compared with non-looped models

Questions for authors

has the author done any exp with loop with linear attentions or hybrid models?

missing references: related works on recurrent and looped transformers: The Topological Trouble With Transformers (Mozer et al., 2026, https://arxiv.org/abs/2604.17121), Recirculation (Mozer et al., 2026, https://arxiv.org/abs/2608.17981), and Training-Free Looped Transformers (Chen et al., 2026, https://arxiv.org/abs/2605.23872).

accuracy on maze and sudoku basically plateaus around 20-35 steps, why does the solver still take like 108 steps on average to actually halt? would using mean-token residual instead of worst-token let it stop way earlier without hurting acc?

most tasks learn a pretty big input injection weight (like lambda = 0.47 on maze), why did you have to fix lambda = 0.1 on sudoku? also curious what happens if you unroll more than k = 6 steps during training.

4 comments
AuthorsOpenReview ↗

Official Response (1/2)

Thank you for the review, and for the feedback that improved our paper and strengthened its contrinbutions. We provide below detailed responses to your concerns.

W1. This paper has lots of figures, i like them, but they look very claudish! We appreciate the fact that you like the figures in our paper. The fact that we used Claude to help creating them using our instructions was disclaimed in the LLM Usage section.

W2. The idea is simple; "just hyper-ball on top", with Slerp instead of linear damping. We respectfully disagree and we see that as our idea as a strength rather than a weakness. A unit-norm state removes what FPRM needs to stay stable (four coupled scaling coefficients, a boundedness theorem, a residual with an ϵ\epsilon offset), and at the same time improving the performance or keeping it on par with FPRM. To cite reviewer xirG: "The application of nGPT's spherical computations to fixed-point reasoners seems like a natural improvement to Fixed-Point reasoners and elegantly lifts the necessity of (learned) residual scaling while keeping magnitudes bounded by design - this could credibly improve stability under extended iterations of their spherical reasoning block. Instead of stability under extended iteration counts, they find their reasoner requires fewer iterations to match FPRMs performance." and EqXb: "The spherical formulation is coherent and well motivated. It provides bounded states by construction and gives a clear angular interpretation of input injection, damping, and stopping tolerances."

W3. Toyish, no scaling up. We agree with you on this point and we have added this point to the limitations too. The benchmarks we chose are the standard ones of this line of work (HRM, TRM, FPRM, PTRM) at its scale of about 7M parameters. Nevertheless, we would like to point out a recent work (https://arxiv.org/html/2611.03998v1) applies similar ideas for language modeling.

W4. No multiple seeds; looped models have higher variance. We ran 3 seeds per model on all three tasks (including the new Mini-Arc task we added during this rebuttal). On Sudoku the gap is far outside the seed variance: 95.5 ± 2.5 against 82.0 ± 4.0, with all SFPRM seeds above every FPRM seed (Table A in the general response). On Maze-Hard (Table C below), SFPRM best of 3 runs 87.9 (mean ± sd: 87.3 ± 0.9) outperforms both the official reported results in the FPRM paper (87.0%), as well as our reproduction using the official code release (best of 3 runs: 85.0; mean ± sd: 83.6 ± 1.6). On Mini-ARC, we report the best of three runs per method (59.1 against 57.0) next to the means, which are on par (54.8 ± 4.0 against 55.3 ± 2.5).

Table C. Maze-Hard and Mini-ARC, 3 seeds per model. FPRM is retrained by us using the official code release. FPRM's published Maze-Hard number is 87.0, whilst the best exact accuracy we managed to get by running their official code was 85%. We also added Mini-ARC during this rebuttal as a new benchmark. As the task is transductive, we we pick the best out of 3 evaluations and report the mean ± sd for both FPRM and SFPRM. We have updated Table 2 in the revised paper PDF.

taskSFPRMFPRM
Maze-Hard, exact accuracy (%), best (mean ± sd) of 3 runs87.9 (87.3 ± 0.9)85.0 (83.6 ± 1.6)
Mini-ARC, pass@1 (%), best (mean ± sd) of 3 runs59.1 (54.8 ± 4.0)57.0 (55.3 ± 2.5)

Q1. Linear attention or hybrid models? Thank you for this suggestion. As Mini-ARC is cheap to train on, we replaced the softmax attention in the nGPT block of SFPRM by DeltaNet and by Kimi Delta Attention.

Table D. Mini-ARC: delta-rule mixers in place of softmax attention.

mixerpass@1pass@2
Softmax59.159.7
DeltaNet + short conv + output gate + RoPE51.756.4
Kimi Delta Attention + short conv + output gate + RoPE49.758.4

The solver runs unchanged with every mixer, with no changes in the hyperparameters. The DeltaNet and KDA layers underperform the Softmax attention, even though all were still improving when training ended.

Q2. Missing references. Thank you for pointing out these missing related work. In the revised PDF we cite and discuss all three.

Q3. Why halt at about 108 steps when accuracy plateaus at 20 to 35? Would a mean-token residual stop earlier? Thank you for checking our paper in detail. Your observation is right, and your suggestion works very well. The halting threshold can be tuned post-hoc and a higher value allows faster halting with marginal or no drop in accuracy. The mean-token residual halts at 17, three times earlier, with similar accuracy (see Table E below). On Sudoku there is no early plateau (74.8% at 32 iterations, 94.2% at 512).

AuthorsOpenReview ↗

Official Response (2/2)

Table E. Accuracy at each sequence's own halt, and mean solver iterations.

taskstopping ruleaccuracymean iterations
Maze-Hardworst-token, τc=0.01\tau_c = 0.01 (default)87.7108
Maze-Hardworst-token, τc=0.04\tau_c = 0.0487.729
Maze-Hardmean-token, τc=0.01\tau_c = 0.0187.517
Sudokuworst-token, τc=0.01\tau_c = 0.01 (default)95.8 ± 0.5389
Sudokuworst-token, τc=0.04\tau_c = 0.0495.1 ± 0.3153
Sudokumean-token, τc=0.01\tau_c = 0.0195.1 ± 0.3198
Sudokumean-token, τc=0.0025\tau_c = 0.002595.8 ± 0.5348

Q4. Why fix λ=0.1\lambda = 0.1? What about k>6k > 6? We optionaly set λ\lambda as a learnable parameter. A learned λ\lambda scores 98.2 ± 0.7 against 96.6 ± 2.9 on FPRM's schedule (see Table A in the general response). The final learned λ\lambda values are between 0.34 and 0.39, so no manual tuning is needed, and the revised PDF uses the learned λ\lambda on every task. Regarding the concern os using k>6k > 6 we ran the training using k=12k = 12. Against k=6k = 6 at the same step count it scores 96.1 against 95.1 using the FPRM repo protocol (decay, patience: 0.99, 5) and 90.7 against 97.4 on FPRM's paper protocol (decay, patience: 0.997, 10), at twice the training memory. Therefore, there is no clear evidence that k>6k > 6 helps.

We hope this resolves the concerns behind your score, and we are happy to run further checks during the discussion phase. Otherwise, we would appreciate if you would consider increasing your score. Thank you very much.

thanks so much for the response. for the results, they look good, but in practice, it would be great to see CE gains at different model scales. currently, the model scale is quite limited. but good work, the idea is nice.

Thank you, we are happy that you like the idea and the results. We would like to note that Sudoku-Extreme, Maze-Hard and ARC at the 5–27M scale are the standard evaluation setting for looped reasoning models: HRM, TRM, FPRM, Equilibrium Reasoners and Attractor Models. Scaling to language modeling, where cross-entropy is outside the scope of this paper and would be a great next step, which we also state in the paper.

Since your remaining concern is scale, and you find the idea and results convincing, could you let us know which concern you consider grounds for a clear rejection?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for taking the time to review our paper and for their feedback, which ultimately strengthened the clarity and results in our paper. Overall, we ran about 50 new training runs to answer them, and all results are in our responses and the revised paper.

Table A. Sudoku-Extreme 5k x 200. 97,656 training steps (15k epochs), except the second row (12k epochs).

Protocol. Our experiments in the paper were conducted using the official code release from the FPRM authors, which uses different hyperparameters for the evaluation step-size schedules (decay, patience) as compared to the ones reported in the Table 4 of the paper's appendix. From FPRM code: (0.99, 5), and FPRM's paper: (0.997, 10).

modeldecay, patience: 0.99, 5 (FPRM repo)decay, patience: 0.997, 10 (FPRM repo)
SFPRM, λ=0.1\lambda = 0.1, 97,656 steps95.5 ± 2.5 (92.6 / 97.3 / 96.5)96.6 ± 2.9 (93.3 / 98.4 / 98.2)
SFPRM, learned λ\lambda, 78,120 steps95.8 ± 0.7 (95.1 / 96.5 / 95.7)97.7 ± 0.2 (97.4 / 97.7 / 97.9)
SFPRM, learned λ\lambda, 97,656 steps96.5 ± 1.3 (97.3 / 94.9 / 97.2)98.2 ± 0.7 (98.7 / 97.4 / 98.4)
FPRM, its own recipe82.0 ± 4.0 (83.5 / 77.5 / 85.0)85.5 ± 4.8 (86.9 / 80.2 / 89.4)
FPRM + our optimizer (Muon)82.5 / 17.0 / 90.782.4 / 17.3 / 95.1
FPRM + our dropout64.4 ± 19.2 (75.8 / 42.2 / 75.2)67.8 ± 19.8 (79.6 / 45.0 / 78.8)
FPRM + our optimizer + our dropout2.0 / 2.0 / 1.42.1 / 2.0 / 1.5
SFPRM without the geodesic operations (normalised affine mixing, relative-norm residual)97.0 / 96.9 / 0.898.1 / 98.8 / 0.7

Summary of changes in the revised draft (changes are highlighted in red in the PDF)

1. New benchmark: Mini-ARC. We added Mini-ARC [Kim et al., 2022], a 5×5-grid ARC variant with 149 tasks. We train on the 528 demonstration pairs, each augmented 100×, and report task-level Pass@1 on the 149 test pairs, voting over augmentations.

  • SFPRM reaches 59.1% Pass@1. This is above the best published result in its size group (EqR, 55.3%).
  • It is also above all baselines we retrained under the same protocol: FPRM 55.0%, HRM 51.7%, EqR 40.3% and TRM 36.9%.
  • Mini-ARC is now reported in the abstract, the main results table and the RQ1 discussion.

2. New ablation study on Mini-ARC (Appendix, new table). We retrain the model with one component changed at a time (Table B):

Table B. Mini-ARC: one change to SFPRM at a time, pass@1 (%). One seed per row.

VariantPass@1Change vs. full model
SFPRM (full)59.1—
Without QK normalization26.2−32.9
Without damping (η = 1)47.7−11.4
Fixed injection fraction (λ = 0.1)49.7−9.4
ACT halting instead of the chordal residual54.4−4.7
Fixed eigen learning rates55.0−4.0
No logit scale55.7−3.4
No weight normalization56.4−2.7

Every component contributes. The geometric parts matter most: QK normalization, angular damping and the learned injection fraction.

3. Convergence and halting on Mini-ARC (Appendix, halting figure, new panel b). These are open-loop rollouts of FPRM and SFPRM on all Mini-ARC test rows over 1000 iterations, showing Pass@1 and the halting residual.

4. Interpretability on Mini-ARC (Sec. RQ3, new figure and two appendix galleries). Under SFPRM, the cells a task must rewrite end farthest from their input anchors. We added a main-text rollout figure on a puzzle that only SFPRM solves, plus appendix galleries of 13 Mini-ARC tasks.

5. Appendix E. We added 3 more pages containing all the experiments we conducted throughout this discussion phase.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.