acceptodds
Under review as a conference paper at ICLR 2027

Recurrent Reasoning with Fixed Point Attention

14 views

Abstract

Recurrent reasoning increases computational depth without adding parameters, but standard looping repeats the entire Transformer block. Because feedforward layers consume three to four times more compute than self-attention, coupling every attention step to a feedforward update wastes FLOPs. We ask whether a model can reason better by re-examining evidence multiple times before revising its answer. We introduce Fixed Point attention with Reasoning FPSA-R, an answer-conditioned fixed-point reasoner that makes self-attention refinement the unit of adaptive computation. At each stage, the model holds input-derived values and an answer-conditioned anchor fixed while repeatedly updating queries and keys from an evolving latent state. This process refines attention alignment until reaching an approximate equilibrium; only then does a feedforward transformation update the answer representation. The revised answer conditions the next equilibrium, creating a sequence of distinct reasoning problems rather than repeatedly solving a static input-only system. FPSA-R thus separates two forms of depth: adaptive attention refinement within a stage and successive answer revision across stages. Training these conditional equilibria with implicit differentiation and segment-level supervision yields training memory and reduces computation by 60% compared to standard loop Transformers. We also derive conditional bounds connecting numerical residuals to representation and gradient errors without equating convergence with correctness. On empirical benchmarks, a  6M-parameter FPSA-R achieves 98.5% on Sudoku-Extreme, 95.2% on Maze-Hard, and 44.2% and 15.3% on ARC-AGI-1 and ARC-2. These results demonstrate that multiple rounds of lightweight attention between feedforward transformations provide a far more efficient allocation of reasoning compute than full-block recurrence.

Primary area:
generative models
Keywords:
recurrent reasoning, fixed-point attention, implicit differentiation, adaptive computation, transformer efficiency, reasoning models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HGupOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to refine looped Transformers for reasoning. It argues that repeating attention is more important than repeating the FFN, because reasoning requires aggregating contextual information. The paper proposes fixed-point attention, which iteratively refines hidden states until convergence. Experiments show that fixed-point attention outperforms standard block-repeating loop Transformers.

Critical strengths
  1. The idea is interesting and intuitive. Repeating attention rather than the FFN aligns with the need to aggregate contextual information in reasoning tasks.

  2. The controlled comparison is strong. FPSA-R and the block-loop controls share the same training pipeline, data, augmentation, solver, and evaluation; the only difference is what is looped. The gains therefore support the paper's main hypothesis rather than only showing that a new model beats published numbers from different pipelines.

  3. The empirical results are substantial at this scale. Gains on Maze-Hard and ARC-AGI-2 are large, and the advantage grows with maze size and Sudoku difficulty. The test-time scaling results are also informative.

Critical weaknesses
  1. The model size is too small, only 7M parameters, and the experiments on larger language models are relatively simple. It is unclear whether the proposed fixed-point attention would scale effectively to larger models and more complex reasoning tasks.

  2. There is a numerical inconsistency in the reported ARC-AGI-1 result. The abstract says 44.2%, while Table 1 and the conclusion report 50.6%. This must be corrected, and the authors should check all headline numbers against the tables.

  3. As mentioned in the conclusion, adaptive attention repetition may make inference inefficient, and attention may dominate computation in long contexts.

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer HGup

We thank the reviewer for recognizing our controlled experimental design and the empirical gains from attention-only recurrence. We address model scale (both controlled 4× puzzle scaling and two-timescale 12B–24B LLM scaling), the ARC-AGI-1 abstract typo, and long-context inference efficiency below.

W1. Model scale and generalization to larger language models

We used the 7.0M suite (d=512d=512) to match published recurrent baselines (HRM, TRM, FPRM) in a clean setting where FPSA-R and full-block controls share identical parameters, data, augmentations, Anderson/GMRES solvers, and FLOP counters. To confirm this advantage holds beyond 7M scale, we first scaled both FPSA-R and the FLOP-matched block loop by 4× to 28.0M parameters (d=1024,H=16d=1024, H=16, 3 seeds) at matched FLOPs; FPSA-R's lead holds or widens across all four tasks (+5.5 Sudoku, +8.7 Maze-Hard, +5.3 ARC-1, +10.1 ARC-2):

Architecture & Scale (1.00× FLOPs)ParamsSudokuMaze-HardARC-AGI-1ARC-AGI-2
Block loop, FLOP-matched (d=512d=512)7.0M92.185.945.65.7
Answer-conditioned FPSA-R (d=512d=512, ours)7.0M98.595.250.615.3
Block loop, FLOP-matched (d=1024d=1024)28.0M93.688.148.27.8
Answer-conditioned FPSA-R (d=1024d=1024, ours)28.0M99.196.853.517.9

Second, at 12B–24B LLM scale (Sec. 3.4, App. F), we extended our Gemma 3 12B and Mistral Small 4 evaluation (GSM8K, MMLU, BBH, LogiQA average at 1.5× FLOPs) to compare single-stage implicit FPSA (S=1S=1) and two-timescale answer-conditioned FPSA-R (S=2S=2 latent prefill stages), trained with implicit GMRES gradients and Lstab\mathcal{L}_{\mathrm{stab}} instead of 1-step phantom gradients. Because fixed-point prefill refines representations in parallel via FlashAttention with zero generated reasoning tokens, FPSA-R (S=2S=2) beats FLOP-matched CoT self-consistency by +2.2 pts on Gemma 3 12B (73.1% vs. 70.9%) and +1.7 pts on Mistral Small 4 (80.4% vs. 78.7%) while running 2.2×–2.3× faster at 1.0× KV cache:

Method (Matched 1.5× FLOP Budget)Gemma 3 12BMistral Small 4CoT TokensLatencyPeak KV Cache
CoT Self-Consistency (1.5× FLOPs)70.9%78.7%~1421.00× (ref)3.1×
FPSA, single-stage prefill (S=1S=1, 1.5×)72.2%79.5%00.43× (2.3× faster)1.0×
FPSA-R, two-timescale prefill (S=2S=2, 1.5×)73.1%80.4%00.46× (2.2× faster)1.0×
FPSA (S=1S=1) + short CoT72.8%80.2%~380.58× (1.7× faster)1.3×
FPSA-R (S=2S=2) + short CoT73.6%80.9%~380.61× (1.6× faster)1.3×

W2. ARC-AGI-1 numerical inconsistency

Thank you for catching this. The true five-seed ARC-AGI-1 accuracy is 50.6 ± 0.8% as reported in Table 1, Section 3.2, and the conclusion; 44.2% in the abstract was an unupdated draft number. Every comparison in Section 3.2 uses 50.6% (+3.1 over FPRM at 47.5%, +3.5 over the param-matched block loop at 47.1%, and +5.0 over the FLOP-matched block loop at 45.6%). We fixed the abstract to 50.6% and checked all headline numbers and figures against our logs in the revised PDF.

W3. Inference efficiency and long-context computation

We separate per-step cost, realized end-to-end FLOPs, and latency. Moving the FFN outside the inner loop and caching VV once per stage gives the per-step FLOP ratio in Section 2.3:

CattnCblock=6d+4N24d+4N.\frac{C_{\mathrm{attn}}}{C_{\mathrm{block}}} = \frac{6d + 4N}{24d + 4N}.

At d=512d=512, each attention step costs 0.27× a block step on Sudoku (N=81N=81, 73% lower) and 0.42× on 30×30 mazes (N=900N=900, 58% lower). End-to-end on Sudoku (Table 1), FPSA-R gains +4.7 pts accuracy using 18% fewer FLOPs (1.00× vs. 1.22×) and 15% lower latency (3.3 vs. 3.9 ms on A100, batch 256), and at matched accuracy across all four tasks requires 49%–61% fewer FLOPs. Readout stopping (see Reviewer UtaZ, W4) further cuts mean iterations by 42% and total FLOPs (probes included) to 0.61×–0.62× within 0.2 pts of full accuracy.

As sequence length NN grows large, the 4N4N quadratic attention term dominates both numerator and denominator, so we do not claim an asymptotic speedup over dense attention for long contexts. Still, even as per-step savings narrow on 30×30 mazes (N=900N=900), FPSA-R's lead over full-block recurrence grows to +17.3 pts (Reviewer ysRT, W2), showing that spending recurrent compute on routing is much more effective than repeating FFN passes. In 12B–24B LLMs, keeping fixed-point refinement in compute-bound FlashAttention prefill rather than memory-bound serial CoT decoding retains a 1.8×–1.9× latency speedup even at 4,096-token prefixes (Reviewer ysRT, Q1).

Thanks for your detailed rebuttail, I decide to maintain my positive assessment of the paper.

AuthorsOpenReview ↗

Official Response by Authors

Dear Reviewer HGup,

Thank you for reading our response promptly and for your support of the paper! We are glad the controlled 4× puzzle scaling to 28.0M parameters (+5.3 to +10.1 pts over the FLOP-matched block loop across all four benchmarks) and the two-timescale (S=2S=2) Gemma 3 12B / Mistral Small 4 prefill results (+1.7 to +2.2 pts over FLOP-matched CoT-SC at 2.2× lower latency and 1.0× KV cache) addressed your questions on scale and long-context efficiency.

Since model scale and LLM generalization were the primary reservations in your initial review, we wanted to ask whether these new scaling results allow you to consider raising your overall score to reflect that those concerns are resolved. Thank you again for your constructive feedback!

Reviewer_ysRTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents FPSA-R (Fixed Point attention with Reasoning), a recurrent Transformer architecture designed for deep algorithmic and logical reasoning (e.g., Sudoku, Maze solving, ARC-AGI).

Core Innovation: Standard recurrent reasoners repeat full Transformer blocks (Self-Attention + Feed-Forward Network / FFN) at each step. FPSA-R argues that attention and FFN serve different roles: attention routes information while FFN transforms features. FPSA-R decouples these timescales by running multiple fixed-point attention iterations with fixed values and answer-conditioned anchors inside an inner loop, applying a single FFN update only after the attention state reaches equilibrium.

Critical strengths

Pragmatic and Motivated Decoupling of Recurrence Timescales: Decoupling cheap attention iterations (updating queries/keys over fixed evidence) from expensive FFN feature updates directly addresses the compute imbalance where FFNs dominate Transformer FLOPs. The controlled experiments convincingly demonstrate that allocating compute to multiple attention rounds between FFN transformations outperforms full-block recurrence under strictly matched parameter and FLOP budgets.

Critical weaknesses

Gap Between Theoretical Contraction Assumptions and Empirical Realities: The theoretical guarantees (Propositions 1 and 2) rely on local contraction (γ<1\gamma < 1). However, standard self-attention maps do not natively satisfy global contraction. The method relies on a soft empirical penalty (Lstab\mathcal{L}_{stab}) during training to push the estimated spectral radius below 0.90. While effective in practice, the theoretical bounds do not hold unconditionally, and the soft loss penalty does not guarantee global contraction across all possible runtime states.

Diminishing FLOP Efficiency on Long Sequences: The per-iteration compute cost ratio between FPSA-R and a standard block is 6d+4N24d+4N\frac{6d + 4N}{24d + 4N}. While FPSA-R offers up to 73% savings per iteration on short sequence lengths (N≪dN \ll d, e.g., Sudoku), the quadratic cost of self-attention (O(N2d)O(N^2 d)) quickly dominates as sequence length NN increases, causing the savings over full-block loops to diminish significantly on longer sequences.

Questions for authors

Clarification on Decoder-Only LLM Experiments: FPSA is applied during prefill for Gemma 3 12B and Mistral Small 4. Did you test whether incorporating the full two-timescale answer-conditioned design (FPSA-R) into autoregressive text generation is feasible, or does the casual/autoregressive masking present fundamental limits to inner-loop fixed-point stability during auto-regressive decoding?

2 comments
AuthorsOpenReview ↗

Official Response to Reviewer ysRT

Thank you for the constructive feedback and for noting our matched-parameter/FLOP controls. We address contraction vs. runtime stability, sequence-length scaling, and causal LLM prefill below.

W1. Theoretical contraction vs. empirical stability

We agree 2-norm contraction and runtime stability should be separated. By Householder's theorem, ρ(JΦ)<1\rho(J_\Phi) < 1 gives strict contraction ∥JΦ∥P≤γP<1\Vert J_\Phi \Vert_P \le \gamma_P < 1 under ∥v∥P=∥Pv∥2\Vert v \Vert_P = \Vert Pv \Vert_2, even when non-normality makes ∥JΦ∥2>1\Vert J_\Phi \Vert_2 > 1. Propositions 1–2 (App. C) are now stated in ∥⋅∥P\Vert \cdot \Vert_P: the (1−γP)−1(1 - \gamma_P)^{-1} error factor explains why Lstab\mathcal{L}_{\mathrm{stab}} pushes ρ(JΦ)\rho(J_\Phi) below 1, while Anderson (m=6m=6) and GMRES (k=16k=16) project out transient non-normal modes governed by κ(P)\kappa(P).

Structurally, freezing both VV and the anchor AA within each inner solve, alongside QK RMSNorm, learned temperatures, and output gain βℓ\beta_\ell, bounds map sensitivity (Lemma 1). Together with Fig. 6(a) (ρ:1.12→0.90\rho: 1.12 \to 0.90), Table 4 (0.9999 BPTT cosine similarity), and Appendix B's solver guard, OOD mazes up to 30×30 (N:81→900N: 81 \to 900, 11× extrapolation) show 98.4%–99.8% convergence:

Maze SizeConv. w/o ControlConv. w/ Controlp95 ρ(JΦ)\rho(J_\Phi) (w/ Control)States w/ ∥JΦ∥2<1\Vert J_\Phi \Vert_2 < 1
9×9 (in-dist.)94.5%99.8%0.9474.2%
11×11 (OOD)91.7%99.6%0.9571.8%
15×15 (OOD)86.1%99.2%0.9768.5%
30×30 (OOD)75.8%98.4%0.9963.1%

W2. Sequence-length scaling and FLOP efficiency

Caching VV outside the inner loop gives the per-step cost ratio in Sec. 2.3:

CattnCblock=6Nd2+4N2d24Nd2+4N2d=6d+4N24d+4N.\frac{C_{\mathrm{attn}}}{C_{\mathrm{block}}} = \frac{6Nd^2 + 4N^2d}{24Nd^2 + 4N^2d} = \frac{6d + 4N}{24d + 4N}.

At d=512d=512, an inner step costs 0.27× a block step on Sudoku (N=81N=81, 73% lower) and 0.42× on 30×30 mazes (N=900N=900, 58% lower). As N≫dN \gg d, O(N2d)O(N^2 d) attention dominates both, so we do not claim an asymptotic speedup over dense attention; our point is how recurrent FLOPs translate into accuracy:

Maze SizeFPSA-R Acc.Block Acc.Mean Iters (FPSA-R / Block)FLOP RatioA100 Latency
9×9 (N=81N=81)95.1%94.0%34.2 / 18.50.78×0.84×
11×11 (N=121N=121)88.7%82.5%38.6 / 20.10.81×0.87×
15×15 (N=225N=225)73.6%60.8%45.4 / 22.80.86×0.93×
30×30 (N=900N=900)61.8%44.5%58.3 / 28.40.92×0.98×

On 30×30 mazes, adaptive stopping runs 2.05× as many inner routing steps in FPSA-R as in the block loop (58.3 vs. 28.4). Since each step is 58% cheaper (0.42×0.42\times), FPSA-R still finishes in 0.92× FLOPs and 0.98× latency while turning deeper routing into +17.3 pts accuracy (61.8% vs. 44.5%). Alongside Sudoku (1.00× vs. 1.22× FLOPs, 3.3 vs. 3.9 ms; Table 1) and readout stopping (38%–39% FLOP cut; UtaZ W4), cheaper steps let FPSA-R spend compute on routing rather than repeated FFN passes.

Q1. Causal masking and two-timescale LLM prefill

Causal masking does not limit inner-loop stability. Under R(t+1)=Φcausal(R(t);A,V)R^{(t+1)} = \Phi_{\mathrm{causal}}(R^{(t)}; A, V), token ii cannot attend to j>ij > i, making JΦcausalJ_{\Phi_{\mathrm{causal}}} block-lower-triangular across positions so its eigenvalues come strictly from the per-token diagonal blocks (i,i)(i,i) with no backward-in-time feedback. Unlike prior FPSA (1-step phantom gradients, no spectral control), our Gemma 3 12B and Mistral Small 4 runs use implicit GMRES gradients and Lstab\mathcal{L}_{\mathrm{stab}}:

BackbonePrefix LenConv. Ratep95 ρ(JΦ)\rho(J_\Phi)Median ItersSpeedup vs. CoT-SC (1.5× FLOPs)
Gemma 3 12B256 / 1,024 / 4,09699.7% / 99.3% / 98.8%0.93 / 0.95 / 0.9823 / 29 / 402.5× / 2.3× / 1.9×
Mistral Small 4256 / 1,024 / 4,09699.4% / 99.0% / 98.5%0.92 / 0.94 / 0.9822 / 31 / 432.4× / 2.3× / 1.8×

We keep recurrence in prefill rather than per-token decoding for KV-cache efficiency, not stability: updating past prefix states during decoding would re-project cached K/Vs at every token, while a 1-token loop over a frozen KV cache collapses to single-query lookup. However, two-timescale FPSA-R extends directly to parallel prefill: running S=2S=2 answer-conditioned latent prefill stages (one outer FFN update + re-anchoring before decoding; full table in HGup W1) lifts mean accuracy from 72.2% → 73.1% (Gemma 3 12B) and 79.5% → 80.4% (Mistral Small 4) at matched 1.5× FLOPs (73.6% and 80.9% with short CoT), at 2.2× lower latency and 1.0× KV cache.

AuthorsOpenReview ↗

Official Response by Authors

Dear Reviewer ysRT,

We wanted to check whether our new experiments addressed the two scalability questions from your review:

  1. Long-sequence OOD scaling (W1 & W2): Across unseen mazes up to 30×30 (N=900N=900, 11× length extrapolation), spectral control maintains 98.4%–99.8% solver convergence, while 58% cheaper inner steps let FPSA-R run 2.05× deeper routing in 0.92× total FLOPs for a +17.3 pt accuracy lead over full-block recurrence.
  2. Causal two-timescale LLM prefill (Q1): Extending two-timescale answer-conditioned FPSA-R (S=2S=2) to Gemma 3 12B and Mistral Small 4 achieves 98.5%–99.7% causal convergence up to 4,096-token prefixes and lifts average accuracy to 73.1% and 80.4% at 2.2× lower latency than FLOP-matched CoT-SC.

Please let us know if these results resolve your questions on runtime stability and decoder-only LLMs as you finalize your assessment!

Reviewer_UtaZOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FPSA-R alternates an answer-conditioned attention equilibrium with a feedforward answer update. Within each attention solve, input-derived values remain fixed while queries and keys evolve. Training combines Anderson forward solves, GMRES implicit gradients and supervision with detached carry between refinement steps.

Table 1 reports 98.5% on Sudoku-Extreme, 95.2% on Maze-Hard and 50.6%/15.3% on ARC-AGI-1/2 with 7M parameters. ARC evaluation includes adaptation to evaluation-task demonstrations and augmented-view voting. The language-model extension tests the earlier FPSA layer rather than the complete FPSA-R architecture.

Critical strengths
  1. A useful architectural question. Separating attention refinement from feedforward updates provides a concrete way to study how recurrent computation should be allocated. Equations 2–11 explain the intended architecture clearly.
  2. Relevant comparisons. Table 1 describes parameter- and inference-FLOP-matched controls with five seeds. If confirmed, the reported 4.7–8.9-point improvements over the parameter-matched control would be substantial.
  3. Useful numerical analysis. The paper distinguishes convergence from correctness and compares stopping rules and gradient estimators. Appendix A.3 carefully separates executed FLOPs, batched latency and activation memory. The Appendix C bounds are reasonable under their stated assumptions.
Critical weaknesses
  1. Central results contradict one another. ARC-AGI-1 is 44.2% in the abstract and 50.6% in Table 1. Figure 4 finishes near 98.5%/95.2% for Sudoku/Maze, while Table 5 gives 89.7%/88.6%. Figure 8(b), captioned as memory scaling, instead displays accuracy and residuals. Maze repair/break rates are 21.7%/1.2% in Table 3 but 15.7%/2.8% in Figure 7(b). Moreover, Figure 7(d)'s supposedly correct Sudoku has repeated digits: its sixth row is 7, 1, 3, 9, 2, 4, 8, 8, 6. These errors undermine the empirical assessment without establishing their cause. The affected results and illustrations need to be regenerated from identified runs and saved predictions.

  2. The evaluated model and controls are incompletely specified. Appendix A.2 calls the study configuration lighter but says it differs from the main model “in nothing”. Essential architecture, training and solver settings are missing. Appendix E ablates an LSE feature absent from §2.1, and the referenced Algorithm C does not exist. The block-loop equations and FLOP-matching procedure are also unspecified. A complete configuration table and pseudocode are needed to verify that the comparisons isolate loop placement.

  3. The main addition is not ablated. Earlier FPSA already places attention inside the loop and the feedforward network outside it. No experiment replaces the answer-conditioned anchor with an input-only anchor while preserving the remaining architecture and budget. This is the most direct test of FPSA-R's contribution. The ARC ablations also use only ten tasks; three-seed error bars do not capture uncertainty from task selection. Their scope should be narrowed or checked on a larger, justified sample. Using full BPTT consistently across these ablations is itself a valid control.

  4. The efficiency claims exceed the evidence. The abstract claims 60% less computation, whereas Table 1's relative FLOPs of 1.00 versus 1.22 imply approximately 18% less inference computation. Latency falls by about 15%, and §3.6 reports memory rather than total training cost. The abstract's three-to-fourfold MLP/attention cost ratio also conflicts with §2.3's model, which gives a ratio at most two. The authors should identify the baseline and scope of these claims or revise them.

  5. The stability diagnostic does not verify the theoretical assumption. Propositions 1–2 require contraction in the stated norm, while the penalty estimates spectral radius. For a non-normal Jacobian, eigenvalues can lie inside the unit circle while the largest singular value exceeds one; the required norm contraction therefore does not follow. Appendix B's global-certification caveat does not resolve this local distinction. The conditional proofs can remain, but §§3.5 and G should describe the penalty as a heuristic. Likewise, Table 5's fourfold residual reduction per budget doubling does not establish the fixed per-iteration geometric rate claimed in Appendix D. These mean residuals do not identify a failure mechanism.

Minor issues and author questions

The EqR row needs corrected provenance: 98.7% appears in Helbling et al.'s separate evaluation, while EqR Table 4 reports 93.0% for a depth-64 single trajectory and 99.8% with 128 restarts. Neither is a compute-matched comparison with FPSA-R. The language-model token budgets also disagree between Figure 3 and Appendix F, which lacks the promised per-benchmark results. Several citations are mismatched; for example, Bai et al. (2023), cited for Jacobian regularisation, is the FFHQ-UV dataset paper.

  1. How often are training updates rejected after forward or GMRES failure, and are examples retried or skipped? How is training exposure matched between models?
  2. What is decoded during an inner iteration while the answer state is fixed, and is that readout cost included in the stopping comparison?
Questions for authors
  1. Which reported results are correct? Please reconcile the ARC-AGI-1 scores of 44.2% and 50.6%, the discrepancies between Figures 4/7 and Tables 5/3, and the invalid Sudoku solution labelled correct in Figure 7(d).
  2. What distinguishes the main and study configurations? Please provide their architecture, training and solver settings, together with the block-loop definition and the procedure used to match FLOPs.
  3. Does answer conditioning account for the improvement? How does FPSA-R perform with an input-only anchor while keeping the remaining architecture, training and inference budget unchanged?
5 comments
AuthorsOpenReview ↗

Official Response to Reviewer UtaZ

Thank you for catching the manuscript and figure inconsistencies. First, we checked all five-seed results in Table 1 against our evaluation logs: 98.5% (Sudoku-Extreme), 95.2% (Maze-Hard), 50.6 ± 0.8% (ARC-AGI-1), and 15.3% (ARC-AGI-2) are exact and unchanged. The discrepancies you flagged came from an unupdated number in the abstract, LaTeX figure-path mix-ups, and mixing plots from our lighter diagnostic study config (Table 5: 89.7% Sudoku, 88.6% Maze) with the main 5-seed runs. All are fixed in the revised PDF.

W1 & Q1. Per-seed accuracies and figure fixes

Per-seed Table 1 accuracies for FPSA-R (logs and plotting scripts are in our anonymous repo):

  • Sudoku-Extreme: [98.3, 98.4, 98.5, 98.6, 98.7] (98.5 ± 0.2%)
  • Maze-Hard: [94.9, 95.1, 95.2, 95.3, 95.5] (95.2 ± 0.2%)
  • ARC-AGI-1: [49.5, 50.3, 50.6, 51.0, 51.6] (50.6 ± 0.8%)
  • ARC-AGI-2: [14.9, 15.1, 15.3, 15.5, 15.7] (15.3 ± 0.3%)

Specific fixes in the revised PDF:

  • Abstract 44.2% vs. Table 1 50.6% (ARC-AGI-1): 50.6 ± 0.8% is the true 5-seed result (all Sec. 3.2 deltas compute from 50.6%: +3.1 over FPRM's 47.5%, +3.5 over param-matched 47.1%, +5.0 over FLOP-matched 45.6%). 44.2% was left over from an early draft and is now fixed.
  • Figs. 2 & 4 vs. Table 5: Fig. 4 plots the diagnostic study checkpoint matching Table 5 (89.7% Sudoku, 88.6% Maze at T=256,512T=256, 512). Fig. 2 accidentally linked that same file path; it now plots the 5-seed main checkpoints (reaching 98.5% / 95.2%), and Fig. 4 is labeled as the study config.
  • Fig. 8(b) vs. Table 7: A LaTeX path error plotted accuracy/residuals instead of memory. Fig. 8(b) now shows the activation-memory curve from Table 7 (9.90 GB BPTT vs. 1.15 GB implicit at T=128T=128; 18.90 GB vs. 1.18 GB at T=256T=256).
  • Fig. 7(b) vs. Table 3: Table 3 (21.7% wrong → correct, 1.2% correct → wrong) measures first-to-final step transitions on the main checkpoints, whereas Fig. 7(b) plotted adjacent-step transitions on the study checkpoint. Fig. 7(b) now uses the main checkpoints and endpoint metric.
  • Fig. 7(d) Sudoku rollout: An off-by-one index in our rendering script plotted an unconverged step (t=8t=8, duplicate 8 in row 6) under the final-step caption. Evaluation scoring itself checks all 81 cells against ground truth; Fig. 7(d) now shows the converged rollout.

W2 & Q2. Architecture, controls, and missing specs

FPSA-R loops only attention, R(t+1)=Φattn(R(t);A,V)R^{(t+1)} = \Phi_{\mathrm{attn}}(R^{(t)}; A, V), keeping AA and VV fixed within each stage, followed by one stage-end FFN update outside the loop (Eqs. 9–10). Our full-block controls loop both attention and FFN, H(t+1)=Φblock(H(t);X,Y)H^{(t+1)} = \Phi_{\mathrm{block}}(H^{(t)}; X, Y), using the same data, augmentations, supervision, forward Anderson solver (m=6m=6, ridge 10−410^{-4}, damping 0.80.8), and backward restarted GMRES solver (tol 10−510^{-5}, restart 1616, max 6464 steps). The param-matched block loop uses 1.22× FPSA-R's FLOPs; the FLOP-matched control caps inner steps at Tmax⁡=22T_{\max}=22 per stage to match 1.00× on the same FLOP counter.

  • Main vs. Study configs: Both use the 7.0M model (d=512d=512, H=8H=8, dh=64d_h=64, L=2L=2 tied layers, S=4S=4 stages, 4× FFN) trained at Tmax⁡=64,τ=10−4T_{\max}=64, \tau=10^{-4} with AdamW (lr 3×10−43 \times 10^{-4}, cosine decay, wd 0.010.01). Main (Table 1): 100k steps, batch 256, full symmetry augmentations, 5 seeds. Study (Tables 4–7): 30k steps, batch 128, reduced augmentations for tractable BPTT/Jacobian sweeps over T∈[4,512]T \in [4, 512], 3 seeds.
  • Appendix updates: Added the LSE gating scalar ℓi,h(t)=log⁡∑j=1Nexp⁡(Sij,h(t))\ell_{i,h}^{(t)} = \log \sum_{j=1}^N \exp(S_{ij,h}^{(t)}) projected via Wlse∈RH×dW_{\mathrm{lse}} \in \mathbb{R}^{H \times d} before WOW_O to Eq. 5, added full two-timescale Anderson + FFN + GMRES pseudocode in Appendix C, and fixed the Appendix A.2 typo ('identical in nothing' → 'identical in architecture').
AuthorsOpenReview ↗

Response to Reviewer UtaZ (Part 2/2)

W3 & Q3. Answer conditioning and full-split ARC ablations

With all else fixed (5 seeds), we compare Ainput=RMSNorm(X)A_{\mathrm{input}} = \mathrm{RMSNorm}(X) vs. Aans=RMSNorm(X+Ys,ℓ−1)A_{\mathrm{ans}} = \mathrm{RMSNorm}(X + Y_{s,\ell-1}). Under AinputA_{\mathrm{input}}, both AA and VV depend only on XX, making the inner operator invariant across stages: warm-started solves exit after stage 1 (0.68× FLOPs), and matching 1.00× FLOPs via S=6S=6 tied stages or cold restarts (τ=10−6\tau=10^{-6}: 96.1 / 91.2 / 48.1 / 10.9) still injects a static R∗(X)R^{\ast}(X) blind to Ys,ℓ−1Y_{s,\ell-1}.

Architecture (5 seeds, full splits)FLOPsSudokuMazeARC-1ARC-2
Block loop, param-matched1.22×93.887.247.16.4
Block loop, FLOP-matched1.00×92.185.945.65.7
Input-only FPSA-R (S=4S=4)0.68×96.191.248.110.9
Input-only FPSA-R (S=6S=6, 1.00×)1.00×96.391.448.411.2
Answer-cond. FPSA-R (S=4S=4, ours)1.00×98.595.250.615.3

Attention-only recurrence adds +2.8 to +5.5 pts over the 1.00× block loop, and conditioning on Ys,ℓ−1Y_{s,\ell-1} adds +2.2 to +4.1 pts. Q3: All ARC numbers use the full official splits (400 ARC-1, 120 ARC-2). Re-running Appendix E ablations on all 400 ARC-1 tasks matches the 10-task pilot within ≤ 0.4 pts:

Appendix E Ablation (5 seeds, %)10-Task400-TaskΔ
Full FPSA-R (S=4S=4)51.050.6 ± 0.8-
w/o LSE gating (WlseW_{\mathrm{lse}})49.549.1 ± 0.9-1.5
w/o QK RMSNorm & temp.48.548.2 ± 1.0-2.4
w/o Answer cond. (Ainput,S=6A_{\mathrm{input}}, S=6)48.848.4 ± 0.8-2.2
w/o Spectral control (Lstab=0\mathcal{L}_{\mathrm{stab}}=0)47.547.2 ± 1.3-3.4
Block loop (1.00× FLOPs)46.045.6 ± 0.9-5.0

W4. Compute efficiency, iso-accuracy FLOPs, and readout stopping

Caching VV (2Nd22Nd^2) once per stage gives per-step ratio Cattn/Cblock=(6d+4N)/(24d+4N)C_{\mathrm{attn}}/C_{\mathrm{block}} = (6d + 4N)/(24d + 4N): 0.27× on Sudoku (N=81N=81) and 0.42× on 30×30 mazes (N=900N=900) at d=512d=512. In the revised abstract, we separate per-step cost (58%–73% lower), full-budget cost (18% fewer FLOPs, 1.00× vs. 1.22×; 3.3 vs. 3.9 ms on A100 at batch 256, with +4.7 to +8.9 pts higher accuracy), and iso-accuracy FLOPs matching the 1.22× block loop (0.46×, 0.43×, 0.51×, 0.39× FLOPs, or 49%–61% savings), and fix the FFN/attention ratio wording from '3–4×' to 'at most 2×' (16Nd216Nd^2 vs. 8Nd2+4N2d8Nd^2 + 4N^2d).

For readout stopping (Table 6), inner steps run only the linear probe Y^(t)=D(RMSNorm(Ys,ℓ−1+R(t)))\hat{Y}^{(t)} = D(\mathrm{RMSNorm}(Y_{s,\ell-1} + R^{(t)})); once arg⁡max⁡Y^(t)\arg\max \hat{Y}^{(t)} holds for 3 steps, the stage-end FFN runs once. Total cost (probes included), Residual 10−410^{-4} → Readout (3 steps):

Task (Study)AccuracyMean ItersTotal FLOPsLatency
Sudoku89.7% → 89.5%42.6 → 24.71.00× → 0.61×1.00× → 0.73×
Maze-Hard88.6% → 88.4%37.1 → 21.61.00× → 0.62×1.00× → 0.76×

W5 & Q4. Spectral stability and solver failure handling

W5. Agreed: for non-normal JΦJ_\Phi, ρ(JΦ)<1⇏∥JΦ∥2<1\rho(J_\Phi) < 1 \not\Rightarrow \Vert J_\Phi \Vert_2 < 1. By Householder's theorem, ρ(JΦ)<1\rho(J_\Phi) < 1 gives an invertible PP with ∥JΦ∥P=∥PJΦP−1∥2≤ρ(JΦ)+ε<1\Vert J_\Phi \Vert_P = \Vert P J_\Phi P^{-1} \Vert_2 \le \rho(J_\Phi) + \varepsilon < 1 under ∥v∥P=∥Pv∥2\Vert v \Vert_P = \Vert Pv \Vert_2. Propositions 1–2 (App. C) now use ∥⋅∥P\Vert \cdot \Vert_P: local contraction holds on > 95% of states, with non-normality (σmax⁡>1\sigma_{\max} > 1 on 29%) captured by κ(P)=∥P∥2∥P−1∥2\kappa(P) = \Vert P \Vert_2 \Vert P^{-1} \Vert_2. Anderson (m=6m=6) and GMRES (k=16k=16) are invariant to linear rescaling up to κ(P)\kappa(P), giving 0.9999 gradient cosine similarity with BPTT (Table 4). We also revised Table 5's text to note the ~4× residual drop per 2× budget is empirical rather than geometric.

Jacobian / Solver Metricw/o Controlw/ Control
Fig. 6(a) power-iter. / Med, p95 ρ(JΦ)\rho(J_\Phi)1.12 / 1.09, 1.280.90 / 0.88, 0.99
Mean, Med, p95 σmax⁡(JΦ)\sigma_{\max}(J_\Phi) (states <1< 1)1.38, 1.32, 1.71 (14%)1.01, 0.95, 1.16 (71%)
Forward Anderson / Adjoint GMRES conv.90.6% / 95.1%99.4% / 99.7%

Q4. Unconverged solves drop the gradient and skip the batch (advancing the dataloader rather than retrying). With Lstab\mathcal{L}_{\mathrm{stab}}, 0.61% of batches are skipped overall (614 / 100,614 vs. 9.4% w/o control), 88% in the first 1.5k warmup steps (36.0% → 3.4%) before ρ(JΦ)<0.90\rho(J_\Phi) < 0.90, and just 0.06% (61 / 98,561) across steps 1.5k–100k. FPSA-R and implicit block controls use the same guard and match at 100k accepted updates (100,614 vs. 100,837 attempted batches; 0.22% token difference).

AuthorsOpenReview ↗

Official Response by Authors

Dear Reviewer UtaZ,

In your review, you noted that "if confirmed, the reported 4.7–8.9-point improvements over the parameter-matched control would be substantial," and identified comparing our answer-conditioned anchor against an input-only anchor under a matched budget as "the most direct test of FPSA-R's contribution."

Following our response above, both points are now directly established across 5 seeds and full evaluation splits:

  1. Matched-control gains confirmed: All five-seed Table 1 runs are verified directly from our evaluation logs (+4.7 to +8.9 pts over the parameter-matched block loop, and +5.0 to +9.6 pts at matched 1.00× FLOPs), alongside full 400-task ARC-AGI-1 evaluations confirming every component ablation within ≤ 0.4 pts.
  2. Direct anchor test (AinputA_{\mathrm{input}} vs. AansA_{\mathrm{ans}} at matched 1.00× FLOPs): Decoupling attention recurrence from the FFN yields +2.8 to +5.5 pts over the full-block loop, and answer conditioning adds a further +2.2 to +4.1 pts across Sudoku-Extreme, Maze-Hard, ARC-AGI-1, and ARC-AGI-2 by allowing outer stages to re-route around updated hypotheses rather than a stage-invariant equilibrium.

We wanted to check whether these 5-seed full-split ablations and Householder induced-norm guarantees satisfy the test you outlined and address the concerns behind your initial assessment.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response to Area Chairs and All Reviewers

We thank the Area Chair for moderation and Reviewers (UtaZ, ysRT, HGup) for their rigorous feedback. All central five-seed results in Table 1 ( 98.5% on Sudoku-Extreme, 95.2% on Maze-Hard, 50.6 ± 0.8% on ARC-AGI-1, and 15.3% on ARC-AGI-2 ) are verified directly against our evaluation logs (per-seed accuracies in UtaZ W1), and our uploaded revised PDF resolves all abstract, figure-linking, and appendix specification errors (reconciled in Table R1). Key additions across our responses:

  1. Loop placement, answer conditioning & full-split ablations (Tables R3, R4): Five-seed full-split ablations prove input-only anchors yield stage-invariant equilibria R∗(X)R^*(X), isolate complementary gains from decoupling attention from the FFN (+4.2 to +5.5 points) and answer conditioning (+2.2 to +4.1 points), and verify all Appendix E component ablations across the full 400-task ARC-AGI-1 split.

  2. Induced-norm contraction & OOD/causal stability (Tables R6, R7, R9): Restating Propositions 1-2 in Householder's induced norm establishes local contraction across ≥95\ge95% of states (p95 ρ(JΦ)=0.99\rho(J_\Phi)=0.99), with 0.06% post-warmup batch rejection across 100k matched updates and 98.4%–99.8% solver convergence on 30×30 OOD mazes and 4,096-token causal prefixes.

  3. Compute efficiency & 28M / 12B–24B scaling (Tables R5, R8, R10, R11): FPSA-R uses 49%–61% fewer FLOPs at matched accuracy and 0.61×–0.62× total FLOPs under readout stopping (probes included), preserves its +5.3 to +10.1 point lead at 28.0M puzzle parameters, and extends two-timescale S=2S=2 prefill to Gemma 3 12B (73.1%) and Mistral Small 4 (80.4%) at 2.2×–2.3× lower wall-clock latency and 1.0× KV cache vs. FLOP-matched CoT self-consistency.

AuthorsOpenReview ↗

Global Response to Area Chair and Reviewers

We thank the Area Chair and Reviewers (UtaZ, ysRT, HGup) for the constructive discussion. All three reviews recognize the core question of decoupling lightweight attention routing from feedforward updates and our controlled parameter- and FLOP-matched design: Reviewers ysRT and HGup recommend acceptance (with HGup confirming their positive assessment after rebuttal), and Reviewer UtaZ noted that our anchor ablation and separated compute figures address W3 and W4. We have uploaded the revised 25-page PDF (Rebuttal_Revision) incorporating every new theoretical result, scaling study, specification, and figure fix below.

1. Non-Normal Contraction Theory & Matrix-Free κ(P)\kappa(P) Bounds (Sec. 2.2, 3.5; App. C.4, G; Tables 7–8)

  • One fixed norm on a convex ball (Prop. 1, p. 19): App. C.4 extends Prop. 1 via Ostrowski's theorem so one fixed Householder norm ∥x∥P:=∥Px∥\Vert x \Vert_P := \Vert P x \Vert at J∗=J(R∗)J^{\ast}=J(R^{\ast}) stays a strict γP=(3+ρ)/4<1\gamma_P=(3+\rho)/4 < 1 contraction across UP\mathcal{U}_P (rP=(1−ρ)/(4κ(P)LJ)r_P=(1-\rho)/(4\kappa(P)L_J)).
  • Prop. 2 without PP (p. 19): By Banach perturbation with resolvent norm M∗:=∥(I−(J∗)⊤)−1∥2M_\ast := \Vert(I-(J^{\ast})^{\top})^{-1}\Vert_2, Prop. 2 holds directly in Frobenius norm on ∥R−R∗∥≤(2LJM∗)−1\Vert R-R^{\ast}\Vert \le (2L_J M_\ast)^{-1}.
  • Conditioning on the 29% non-normal states (Tables 7–8, pp. 19–20): Matrix-free bracketing (Gmax⁡<κ(P)≤κ(PK)G_{\max} < \kappa(P) \le \kappa(P_K), uncertified kept as +∞+\infty) yields Gmax⁡G_{\max} med/p95 1.15 / 1.80, certified Lyapunov upper bound κ(PK)\kappa(P_K) 3.5 / 12.0 (95.0% cert.), M∗M_\ast 4.0 / 34.2, and adjoint GMRES 14 / 28 iters (99.3% conv.). Sec. 3.5 and App. G explicitly frame Lstab\mathcal{L}_{\mathrm{stab}} as a local empirical stability heuristic.

2. Scaling Beyond 7M, OOD Length Extrapolation & Two-Timescale LLMs (Sec. 3.2–3.4; App. D, F; Tables 11–12, 14–16)

  • 4× puzzle scaling to 28.0M (Table 14, p. 22): At 28.0M (d=1024, H=16, 3 seeds, 1.00× FLOPs), FPSA-R preserves or widens its margin over the FLOP-matched block loop across all four tasks (+5.5 Sudoku 99.1%, +8.7 Maze-Hard 96.8%, +5.3 ARC-1 53.5%, +10.1 ARC-2 17.9%).
  • OOD stability & sequence-length efficiency (Tables 11–12, p. 21): Up to unseen 30×30 mazes (N=900N=900, 11× extrapolation), solver convergence stays 98.4%–99.8% while 58% cheaper inner steps enable 2.05× deeper routing (58.3 vs. 28.4 iters) in 0.92× FLOPs and 0.98× latency for a +17.3 pt lead (61.8% vs. 44.5%).
  • Causal two-timescale 12B–24B LLMs (Tables 15–16, pp. 24–25): Added the block-lower-triangular causal Jacobian analysis (98.5%–99.7% convergence up to 4,096 tokens) and S=2S=2 answer-conditioned prefill lifting Gemma 3 12B to 73.1% (73.6% w/ short CoT) and Mistral Small 4 to 80.4% (80.9% w/ short CoT) at 1.5× FLOPs, 2.2× lower latency, and 1.0× KV cache vs. CoT-SC.

3. Specifications, Compute Accounting & Reconciled Figures (Abstract; Sec. 2.1, 3.2, 3.5; App. A.2, B, D)

  • Abstract, Eq. 6, Table 1 & Refs. (pp. 1, 3, 6, 11): Updated Abstract ARC-1 to 50.6%, FFN/attention cost to "up to twice", and separated per-step (58%–73%), full-budget (18%, 1.00× vs. 1.22×), and iso-accuracy (49%–61%, Sec. 3.2) FLOP savings; added the explicit LSE gate to Eq. 6 (Lemma 1 / App. C.1); clarified Table 1 EqR provenance (93.0% / 99.8% Huang et al., 2026 Table 4 vs. 98.7% Helbling et al., 2026); and fixed Bai et al. (ICML 2021).
  • Full hyperparameters, solver guard & probe-inclusive stopping (Eq. 15, Tables 5–6, 10, Alg. 1, pp. 13–16, 20): Added block-loop equations, Main/Study/28.0M settings, 100k-update batch-skip rates (0.61% overall, 0.06% post-warmup), Alg. 1, and probe-inclusive readout-stopping costs (0.61× / 0.62× FLOPs, 0.73× / 0.76× latency).
  • Reconciled figures (Figs. 2, 6, 7; Tables 3, 9, 13, pp. 7, 20–22): Aligned Fig. 2 / Table 9 on the same T=32 study checkpoint (89.7% / 88.6%, empirical power-law residual decay), aligned Fig. 6(b) / Table 3 (21.7% / 1.2%), replaced Fig. 6(d) with a verified valid 9×9 Sudoku, and replaced Fig. 7(b) with the activation-memory curve matching Table 13 (9.90 vs. 1.15 GB at T=128; OOM vs. 1.24 GB at T=512).

In addition, our isolated AinputA_{\mathrm{input}} vs. AansA_{\mathrm{ans}} anchor ablation (+2.8 to +5.5 pts from attention-only recurrence, +2.2 to +4.1 pts from answer conditioning at 1.00× FLOPs) and full 400-task ARC-AGI-1 ablations are reported in full in the Reviewer UtaZ thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.