- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SFPRM transfers the idea of restricting activations and most Transformer weights to a unit-sphere to Fixed-Point Reasoners, eliminating the need of (learned) residual scaling. Constraining updates to the latent reasoning state z to the unit-hypersphere guarantees boundedness of z, which potentially improves the iterability of the recurrent reasoning model beyond its training horizon. As all updates to z_i are damped rotations on the unit-hypersphere, their spherical nGPT block proposes a new state z_i^{\tilde}, the update uses spherical linear interpolation to rotate z_{i+1} towards it by a fraction \eta_i. Every step, z is partially (coefficient \lambda ) rotated towards the embedded spherical representation of the input x. \eta_i is initialized per sample at 1 and decreases geometrically (multiplication by \gamma < 1) whenever the maximum token-wise chordal residual did not improve for P iterations. The authors provide theoretical guarantees for convergence to and the existence of a unique fixed point under sufficiently strong input-injection and evaluate convergence empirically. Instead of the relative scale of the residual (as in FPRM), in SFPRM, the maximum proposed (undamped) angular update to a token's representation is used as a stopping criterion. They find SFPRM halts earlier than FPRM on average while simultaneously achieving higher accuracy across Sudoku and Maze.
Critical strengths
The application of nGPT's spherical computations to fixed-point reasoners seems like a natural improvement to Fixed-Point reasoners and elegantly lifts the necessity of (learned) residual scaling while keeping magnitudes bounded by design - this could credibly improve stability under extended iterations of their spherical reasoning block. Instead of stability under extended iteration counts, they find their reasoner requires fewer iterations to match FPRMs performance.
Their proofs of fixed-point existence and convergence under sufficiently strong input-injection and damped updates on the hypersphere are strong theoretical contributions, even if their solver operates in regimes where these conditions might not hold.
The comparison of alternative fixed-point solvers is particularly informative, demonstrating that inference efficiency can potentially be improved considerably without retraining the model.
The paper is written very clearly and well structured.
Critical weaknesses
The main weakness is that apparently all experiments were only done for a single seed run - I hope the authors can show the significance of their findings across various training seeds, currently differences to FPRM could worst-case also be attributed to lucky training runs.
Their current experimental setup does not allow clean conclusions about relative wall-clock efficiency nor contributions of individual adaptations, several choices are changed simultaneously: Transformer->nGPT, AdamW->Muon for weights, AdamW for other params, input-injection: affine->Slerp, relative residual halting-criterion -> angular, Sudoku: 1000x1000aug->5000x200aug.. While some choices probably need to be varied jointly (spherical-repr.+halting-criterion), geodesic input injection, damping, optimizer and regularization could be isolated. In particular, I'd like to see the individual contribution of just nGPTs normalization vs full geodesic operations. While faster convergence in terms of number of #iterations is desirable, wall-clock/throughput/FLOP comparisons could clear doubts about improved efficiency. Also their mean halting costs ignore samples that failed to halt, and ignore the fact that a halted sample needs to continue until the whole batch terminates in their implementation (acknowledged in the paper). Especially defining the halting iteration as the iteration after which the (chordal) residual STAYS below its threshold differs from the criterion in Algorithm 1 and further complicates interpreting actual inference costs. Furthermore, the parameter-count matched comparison in App. C.1 tab. 4 severely fails to reproduce FPRM's results, which severely limits clear conclusions about the architectural advantage of SFPRM. Additionally it is unclear why it uses 12k epochs instead of 15k used for the reported final SFPRM accuracy of 96.6%.
The regime where their Theorem 2 holds seems to converge to fixed-points which no longer correspond to the best solutions. Figure 4 suggests a tension between fixed-point uniqueness and and high accuracy, specifically increasing input injection improves consistency across initializations, while reducing accuracy. This questions the practical relevance of theorem 2.
Questions for authors
FPRM specifically motivated its pre-normalization and residual-scaling architecture by the signal-propagation limitations of post-normalized recurrent transformers like TRM. SFPRM now uses nGPT to recover unit-norm by construction. Could you add a comparison of post-norm TRM/FPRM, pre-norm FPRM with residual scaling, and SFPRM across increasing training and inference depth (similar to FPRM paper figure 7), to test, whether SFPRM retains FPRM's improved signal propagation properties?
Do you have a hypothesis, why your reproduction of FPRM achieves only 78.4% accuracy compared to the published 94.2%? Without either a successful reproduction on 5k x 200 of FPRM or an uncollapsed 1k x 1k SFPRM, a fully controlled comparison remains difficult.
4 comments
Official Response (2/2)
- Sequences that never halt. For SFPRM, 2 to 4% of the test puzzles never reach the tolerance within 5,000 iterations.
- Batch-level halting. Our implementation stops a batch only when all of its sequences are below the tolerance. On Sudoku every batch of 768 contains at least one sequence that never halts, so every batch runs to the iteration cap, for both FPRM and SFPRM.
- "First below" and "stays below". We use one definition throughout: a sequence halts at the first iteration whose residual is below the tolerance, as in Algorithm 1. For SFPRM the choice hardly matters, because almost every sequence that gets below the tolerance stays below it. For FPRM, 7 to 10% of the sequences get below the tolerance and later rise above it again.
- Implications to cost. At its default tolerance SFPRM halts later than FPRM (312 against 89 iterations in Table G), but that default is stricter than it needs to be. With a looser tolerance, , SFPRM halts after 78 iterations on average and reaches 93.8% accuracy, while FPRM at its default needs 120 iterations for 79.7% (Table K in our response to Reviewer EqXb). SFPRM is then 14 points more accurate and also faster in wall-clock time. On Maze-Hard SFPRM halts earlier than FPRM at the default tolerances as well.
W2d. 12k against 15k epochs. We trained both models for 12k (78,120 steps) and 15k epochs (97,656 steps). At 12k epochs, SFPRM scores 97.7 ± 0.2 against FPRM's 85.5 ± 4.8 at 15k epochs (Table A in the general responses).
W3. Theorem 2 regime. We apologize for the theorem not being clear. The theorem guarantees uniqueness of the fixed-point under a strong input injection, however it does not say anything about the quality of the solution. In Fig. 4 we show a trade-off between uniqueness of the fixed point and accuracy for a sweep of the input injection.
Q1. Depth. For inference depth, SFPRM is ahead of FPRM at every depth and improves monotonically from 16 to 5,000 iterations (Table H below).
Table H. Sudoku 5k x 200: accuracy after a fixed number of solver iterations, no halting.
| iterations | 16 | 32 | 64 | 128 | 256 | 512 | 1,024 | 5,000 |
|---|---|---|---|---|---|---|---|---|
| SFPRM | 59.5 ± 1.3 | 74.8 ± 1.0 | 83.2 ± 1.1 | 88.4 ± 0.9 | 92.3 ± 0.6 | 94.2 ± 1.3 | 95.1 ± 1.9 | 95.3 ± 2.1 |
| FPRM | 54.7 ± 1.8 | 67.0 ± 2.0 | 73.3 ± 2.0 | 77.3 ± 2.7 | 79.7 ± 3.5 | 80.9 ± 4.0 | 81.4 ± 4.2 | 81.6 ± 4.1 |
Q2. Why 78.4% against 94.2% for FPRM? We refer the reviewer to Table I below for the results of our reproductibility efforts. FPRM's public code release gives 79.2 ± 1.7. Running FPRM on 5k x 200, similarly as SFPRM, scores 82.0 ± 4.0 with the public code release. We also ran FPRM using he hyperparameters reported in the paper appendix for Sudoku, which they differ from the ones in the code release, and we obtained 85.5 ± 4.8 test performance.
Table I. FPRM in its published setting, Sudoku-Extreme 1k x 1000.
| FPRM model | FPRM eval repo | FPRM eval paper |
|---|---|---|
| released checkpoint, full test set | 88.6 | 94.2 |
| retrained, current public code and config | 79.2 ± 1.7 | 84.5 ± 2.4 |
Thank you again for your detailed review and we hope the new experiments resolve your concerns. We are available to engage in further discussion in case of any further concerns or questions. Otherwise, we would appreciate if you would consider increasing your score.
Remaining concerns following the rebuttal
I appreciate the extensive additional experiments, but two important concerns remain.
1. Best-of-three reporting, unmatched training data, and FPRM reproduction
Reporting the best of three seeds in the main results is questionable, particularly for small differences. On Mini-ARC, SFPRM's best run outperforms FPRM (59.1% vs. 57.0%), while its mean is actually lower (54.8 ± 4.0% vs. 55.3 ± 2.5%). Means and standard deviations should be the primary results.
Furthermore, the headline Sudoku comparison (97.9% vs. 94.2%) uses different training builds (5k × 200 vs. 1k × 1000). Table 9 reveals that this choice is consequential: on the original 1k × 1000 build, reproduced FPRM outperforms SFPRM (84.5% vs. 78.2%), reversing the ranking on 5k × 200. Since the authors still cannot reproduce FPRM's published 94.2% through retraining, the matched comparisons demonstrate superiority over an underperforming reproduction, while the headline comparison remains confounded.
2. Idealized halting costs and unequal iteration costs
Despite the new measurements, the contributions (p. 2, lines 78–79) still claim SFPRM "halts up to 2.7× earlier", and the RQ2 conclusion (p. 9, lines 454–455) asserts earlier halting on both Sudoku and Maze.
However, Table G reveals that every Sudoku batch of 768 runs to the iteration cap under the implemented stopping rule, so the idealized per-sequence halting savings are not realized in practice.
Additionally, SFPRM iterations are 28% slower than FPRM iterations on Sudoku (12.3 vs. 9.6 ms, Appendix E.5). Thus, fewer iterations do not necessarily imply lower computational cost. The reported Maze improvement (108 vs. 130 iterations, only 17% fewer) lacks corresponding wall-clock measurements. On Sudoku, the default SFPRM tolerance can even require substantially more iterations.
While the relaxed-tolerance experiments demonstrate promising accuracy–iteration tradeoffs, they do not establish improved end-to-end inference efficiency. I therefore suggest qualifying the efficiency claims and clearly distinguishing idealized per-sequence costs from realized computational savings.
Given the updated version, my main concerns actually strengthened.
Addressing remaining concerns
Thank you for the careful follow-up. Below we address your remaining concerns.
1. Seeds, unmatched training data
Means only. We understand your concern, however we chose to use the best of 3 runs in Table 2 as all the other baselines from the respective papers also report a single seed. In the revision we report the means ± sd in the appendix. For the camera-ready version we will also train every baseline with three seeds and report its mean ± sd.
Headline against the published model, not a reproduction. On the full test set, under the FPRM paper's schedule (0.997, 10) and FPRM's budget of 12k training epochs, SFPRM scores 97.7 ± 0.2%. FPRM's released checkpoint scores 94.22% under the same protocol. The remaining difference is the unique training puzzles (the size of the training set is the same). As you note, the ranking depends on the training set build. In our runs, FPRM gains at most 1 point from training on the 5k × 200 build, while SFPRM needs it to avoid over-fitting the 1k puzzles. We also note that in Table 4 in the appendix there is already evidence that SFPRM can match or outperform FPRM with the 1k x 1000 build too when using the best checkpoint (87% test accuracy), however, in our evaluation, we strived for a training hyperparameter setting that would avoid utilizing early stopping but rather the final checkpoint.
| model | 5k × 200 | 1k × 1000 |
|---|---|---|
| SFPRM, learned | 97.7 ± 0.2 | 78.2 ± 2.4 |
| FPRM, public recipe | 85.5 ± 4.8 | 84.5 ± 2.4 |
2. Halting cost and efficiency
We agree that the per-sequence halting costs in the paper are idealized, and we now report measured wall-clock time instead.
Maze-Hard: halting saves compute, measured end to end. One of FPRM's ten batches runs to the 1,000-iteration cap:
| stopping rule | SFPRM | FPRM |
|---|---|---|
| worst-token (default) | 87.7%, 5.3 s per batch | 84.2%, 6.4 s per batch |
| mean-token | 87.5%, 0.39 s per batch | 83.8%, 0.86 s per batch |
With the mean-token residual suggested by Reviewer D6NB, SFPRM halts each batch after about 22 iterations instead of 300 (note that this is not the 108 mean halting steps per maze, which is idealized): 14× less time for 0.2 points of accuracy. Under the same rule it needs 2.2× less time than FPRM and is 3.8 points more accurate, although each of its iterations costs 1.2× (ms) an FPRM iteration.
Sudoku: halting saves no compute, for either model. Our evaluator stops a batch only when all of its sequences are below the tolerance. On Sudoku, however, every batch of 768 contains some sequence that is not, so every batch runs to the iteration cap, for both FPRM and SFPRM. The per-sequence savings in the paper would need an evaluator that replaces each converged sequence with a new one, which neither method implements.
Fixing the iteration budget. We also run a batched evaluator that instead runs every batch for a fixed number of iterations. Its cost is then that number times the measured time per iteration, with nothing idealized. Measured this way, SFPRM is much faster than FPRM despite its slower inference time per iteration:
- Same time, higher accuracy: at every budget SFPRM is 7 to 14 points more accurate, and every SFPRM seed is above every FPRM seed.
- Higher accuracy in far less time: after 96 iterations (1.18 s per batch) SFPRM is at 86.6%, already above the 81.6% FPRM reaches after 4,096 iterations (39.3 s). That is 33× less time for 5 more points.
- A level FPRM never reaches: with this schedule FPRM's mean never exceeds 82%, even in the full 35,000-iteration evaluation, while SFPRM averages 90.8% after 2.4 s.
Sudoku 5k × 200, checkpoints of Table G, schedule (0.99, 5), batch 768 on an A100. FPRM gets about 4% more time than SFPRM in every row:
| wall-clock per batch, SFPRM / FPRM (iterations) | SFPRM | FPRM |
|---|---|---|
| 0.59 / 0.61 s (48 / 64) | 80.3 ± 1.1 | 73.3 ± 2.0 |
| 1.18 / 1.23 s (96 / 128) | 86.6 ± 1.0 | 77.3 ± 2.7 |
| 2.36 / 2.46 s (192 / 256) | 90.8 ± 0.8 | 79.7 ± 3.5 |
| 4.73 / 4.92 s (384 / 512) | 93.6 ± 0.8 | 80.9 ± 4.0 |
| 37.8 / 39.3 s (3,072 / 4,096) | 95.3 ± 2.1 | 81.6 ± 4.1 |
We hope that with these results we have addressed your remaining concerns and we want to thank you again for your valuable feedback that ultimately helped us improve our paper and claims.
Official Response (1/2)
Thank you for your detailed review and feedback. We ran every experiment you asked for that fits in the rebuttal period and provide detailed responses to your main concerns below one by one. Tables A and B are in the general response.
W1. Single seed. We ran 3 seeds per model on all three tasks (including the new Mini-Arc task we added during this rebuttal). On Sudoku the gap is far outside the seed variance: 95.5 ± 2.5 against 82.0 ± 4.0, with all SFPRM seeds above every FPRM seed (Table A in the general response). On Maze-Hard (Table C below), SFPRM best of 3 runs 87.9 (mean ± sd: 87.3 ± 0.9) outperforms both the official reported results in the FPRM paper (87.0%), as well as our reproduction using the official code release (best of 3 runs: 85.0; mean ± sd: 83.6 ± 1.6). On Mini-ARC, we report the best of three runs per method (59.1 against 57.0) next to the means, which are on par (54.8 ± 4.0 against 55.3 ± 2.5).
Table C. Maze-Hard and Mini-ARC, 3 seeds per model. FPRM is retrained by us using the official code release. FPRM's published Maze-Hard number is 87.0, whilst the best exact accuracy we managed to get by running their official code was 85%. We also added Mini-ARC during this rebuttal as a new benchmark. As the task is transductive, we we pick the best out of 3 evaluations and report the mean ± sd for both FPRM and SFPRM. We have updated Table 2 in the revised paper PDF.
W2a. Confounds; nGPT normalisation alone against the geodesic operations. We ran the ladder you describe, 3 seeds per rung (Table A).
We also add other ablations on the Mini-ARC task, which we added during this rebuttal period (Table B in the general response). Replacing Slerp or the halting rule marginally decreases performance. The highest drop comes from ablating the QK-normalisation.
W2b. Wall-clock and FLOPs. We measured both metrics and added the following table in the revised paper PDF.
Table F. Cost per step on one A100.
The extra time is in the normalisations and the interpolation, which the FLOP counter does not count. Using the nGPT operations instead of the geodesic ones, as the reviewer suggested, lowers the one solver iteration time to 11.4 ms. The training and inference overhead are repaid in faster halting at test time though, as well as in improved performance.
W2c. Halting cost: non-halting samples, batch-level halting, "stays below". Thank you, all these points are fair. For each of the three seeds of each model, we ran the solver on the same test puzzles (batches of 768) for 5,000 iterations without halting, using the default tolerance (τc=0.01 for SFPRM, 0.1 for FPRM). At every iteration we recorded each sequence's residual. From these traces we computed, per sequence, the first iteration at which its residual is below the tolerance ("first below", the rule of Algorithm 1) and the first iteration from which it remains below until the end ("stays below"), and per batch the iteration at which the implemented rule would stop, which requires all sequences of the batch to be below the tolerance at once.
Table G. Sudoku 5k x 200: halting cost at the default tolerances (mean of 3 seeds).