Recurrent Reasoning with Fixed Point Attention

Recurrent reasoning increases computational depth without adding parameters, but standard looping repeats the entire Transformer block. Because feedforward layers consume three to four times more compute than self-attention, coupling every attention step to a feedforward update wastes FLOPs. We ask whether a model can reason better by re-examining evidence multiple times before revising its answer. We introduce Fixed Point attention with Reasoning FPSA-R, an answer-conditioned fixed-point reasoner that makes self-attention refinement the unit of adaptive computation. At each stage, the model holds input-derived values and an answer-conditioned anchor fixed while repeatedly updating queries and keys from an evolving latent state. This process refines attention alignment until reaching an approximate equilibrium; only then does a feedforward transformation update the answer representation. The revised answer conditions the next equilibrium, creating a sequence of distinct reasoning problems rather than repeatedly solving a static input-only system. FPSA-R thus separates two forms of depth: adaptive attention refinement within a stage and successive answer revision across stages. Training these conditional equilibria with implicit differentiation and segment-level supervision yields $O(1)$ training memory and reduces computation by 60% compared to standard loop Transformers. We also derive conditional bounds connecting numerical residuals to representation and gradient errors without equating convergence with correctness. On empirical benchmarks, a  6M-parameter FPSA-R achieves 98.5% on Sudoku-Extreme, 95.2% on Maze-Hard, and 44.2% and 15.3% on ARC-AGI-1 and ARC-2. These results demonstrate that multiple rounds of lightweight attention between feedforward transformations provide a far more efficient allocation of reasoning compute than full-block recurrence.