- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to refine looped Transformers for reasoning. It argues that repeating attention is more important than repeating the FFN, because reasoning requires aggregating contextual information. The paper proposes fixed-point attention, which iteratively refines hidden states until convergence. Experiments show that fixed-point attention outperforms standard block-repeating loop Transformers.
Critical strengths
-
The idea is interesting and intuitive. Repeating attention rather than the FFN aligns with the need to aggregate contextual information in reasoning tasks.
-
The controlled comparison is strong. FPSA-R and the block-loop controls share the same training pipeline, data, augmentation, solver, and evaluation; the only difference is what is looped. The gains therefore support the paper's main hypothesis rather than only showing that a new model beats published numbers from different pipelines.
-
The empirical results are substantial at this scale. Gains on Maze-Hard and ARC-AGI-2 are large, and the advantage grows with maze size and Sudoku difficulty. The test-time scaling results are also informative.
Critical weaknesses
-
The model size is too small, only 7M parameters, and the experiments on larger language models are relatively simple. It is unclear whether the proposed fixed-point attention would scale effectively to larger models and more complex reasoning tasks.
-
There is a numerical inconsistency in the reported ARC-AGI-1 result. The abstract says 44.2%, while Table 1 and the conclusion report 50.6%. This must be corrected, and the authors should check all headline numbers against the tables.
-
As mentioned in the conclusion, adaptive attention repetition may make inference inefficient, and attention may dominate computation in long contexts.
Questions for authors
see weakness
3 comments
Thanks for your detailed rebuttail, I decide to maintain my positive assessment of the paper.
Official Response by Authors
Dear Reviewer HGup,
Thank you for reading our response promptly and for your support of the paper! We are glad the controlled 4× puzzle scaling to 28.0M parameters (+5.3 to +10.1 pts over the FLOP-matched block loop across all four benchmarks) and the two-timescale () Gemma 3 12B / Mistral Small 4 prefill results (+1.7 to +2.2 pts over FLOP-matched CoT-SC at 2.2× lower latency and 1.0× KV cache) addressed your questions on scale and long-context efficiency.
Since model scale and LLM generalization were the primary reservations in your initial review, we wanted to ask whether these new scaling results allow you to consider raising your overall score to reflect that those concerns are resolved. Thank you again for your constructive feedback!
Response to Reviewer HGup
We thank the reviewer for recognizing our controlled experimental design and the empirical gains from attention-only recurrence. We address model scale (both controlled 4× puzzle scaling and two-timescale 12B–24B LLM scaling), the ARC-AGI-1 abstract typo, and long-context inference efficiency below.
W1. Model scale and generalization to larger language models
We used the 7.0M suite (d=512) to match published recurrent baselines (HRM, TRM, FPRM) in a clean setting where FPSA-R and full-block controls share identical parameters, data, augmentations, Anderson/GMRES solvers, and FLOP counters. To confirm this advantage holds beyond 7M scale, we first scaled both FPSA-R and the FLOP-matched block loop by 4× to 28.0M parameters (d=1024,H=16, 3 seeds) at matched FLOPs; FPSA-R's lead holds or widens across all four tasks (+5.5 Sudoku, +8.7 Maze-Hard, +5.3 ARC-1, +10.1 ARC-2):
Second, at 12B–24B LLM scale (Sec. 3.4, App. F), we extended our Gemma 3 12B and Mistral Small 4 evaluation (GSM8K, MMLU, BBH, LogiQA average at 1.5× FLOPs) to compare single-stage implicit FPSA (S=1) and two-timescale answer-conditioned FPSA-R (S=2 latent prefill stages), trained with implicit GMRES gradients and Lstab instead of 1-step phantom gradients. Because fixed-point prefill refines representations in parallel via FlashAttention with zero generated reasoning tokens, FPSA-R (S=2) beats FLOP-matched CoT self-consistency by +2.2 pts on Gemma 3 12B (73.1% vs. 70.9%) and +1.7 pts on Mistral Small 4 (80.4% vs. 78.7%) while running 2.2×–2.3× faster at 1.0× KV cache:
W2. ARC-AGI-1 numerical inconsistency
Thank you for catching this. The true five-seed ARC-AGI-1 accuracy is 50.6 ± 0.8% as reported in Table 1, Section 3.2, and the conclusion; 44.2% in the abstract was an unupdated draft number. Every comparison in Section 3.2 uses 50.6% (+3.1 over FPRM at 47.5%, +3.5 over the param-matched block loop at 47.1%, and +5.0 over the FLOP-matched block loop at 45.6%). We fixed the abstract to 50.6% and checked all headline numbers and figures against our logs in the revised PDF.
W3. Inference efficiency and long-context computation
We separate per-step cost, realized end-to-end FLOPs, and latency. Moving the FFN outside the inner loop and caching V once per stage gives the per-step FLOP ratio in Section 2.3:
CblockCattn=24d+4N6d+4N.
At d=512, each attention step costs 0.27× a block step on Sudoku (N=81, 73% lower) and 0.42× on 30×30 mazes (N=900, 58% lower). End-to-end on Sudoku (Table 1), FPSA-R gains +4.7 pts accuracy using 18% fewer FLOPs (1.00× vs. 1.22×) and 15% lower latency (3.3 vs. 3.9 ms on A100, batch 256), and at matched accuracy across all four tasks requires 49%–61% fewer FLOPs. Readout stopping (see Reviewer UtaZ, W4) further cuts mean iterations by 42% and total FLOPs (probes included) to 0.61×–0.62× within 0.2 pts of full accuracy.
As sequence length N grows large, the 4N quadratic attention term dominates both numerator and denominator, so we do not claim an asymptotic speedup over dense attention for long contexts. Still, even as per-step savings narrow on 30×30 mazes (N=900), FPSA-R's lead over full-block recurrence grows to +17.3 pts (Reviewer ysRT, W2), showing that spending recurrent compute on routing is much more effective than repeating FFN passes. In 12B–24B LLMs, keeping fixed-point refinement in compute-bound FlashAttention prefill rather than memory-bound serial CoT decoding retains a 1.8×–1.9× latency speedup even at 4,096-token prefixes (Reviewer ysRT, Q1).