- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the performance drop that can occur in looped Transformers as test-time iterations increase, when new states overwrite previously correct intermediate results. It proposes InfiLoop, which aggregates historical hidden states using content-based scoring and temporal decay. Experiments show competitive results on Sudoku, Maze, and ARC; on Sudoku-Extreme, its accuracy continues to improve as the number of executed layers increases.
Critical strengths
- The method is simple yet effective. Content-based scoring and temporal decay determine which historical states to retain, and the method can be interpreted as an adaptive residual update.
- InfiLoop is memory-efficient. The derived streaming recurrence uses constant aggregation memory with respect to the number of iterations, without storing the full history of states.
- The paper includes depth-scaling, ablation, and state-perturbation analyses.
Critical weaknesses
- The relationship to related work could be discussed more fully. The paper mentions methods such as Attention Residuals, DenseFormer, and FPRM, but does not sufficiently clarify what InfiLoop adds beyond existing learned residual aggregation and gated recurrent updates.
- The experimental scope remains limited to Sudoku, Maze, and ARC, while the main depth-scaling and stability analyses focus on Sudoku-Extreme. It is unclear whether InfiLoop offers similar benefits on more general tasks or standard, non-nested looped Transformers.
Questions for authors
See weaknesses.
2 comments
Dear Reviewer 8ojc,
Thank you again for your questions about InfiLoop’s connection to prior work and its use beyond Sudoku. InfiLoop weights each new state against its history using constant memory across iterations.
On the single-state URM backbone, InfiLoop reached 84.3% versus 75.9% for our reproduction. On MATH-500 with Qwen3-1.7B-Base, increasing inference depth from three to five loops raised accuracy from 44.2% to 45.8% for InfiLoop, while the standard loop fell from 43.2% to 40.4%.
We’d welcome your thoughts on whether these results address your questions.
Best regards,
The Authors
Response to Reviewer 8ojc
W1. "The relationship to related work could be discussed more fully. The paper mentions methods such as Attention Residuals, DenseFormer, and FPRM, but does not sufficiently clarify what InfiLoop adds beyond existing learned residual aggregation and gated recurrent updates."
A1. Due to the page limit, we placed this comparison in Appendix B. We will move it into the main text and expand it as follows.
Full Attention Residuals gives each layer its own query and stores all previous outputs, so it applies to a fixed depth and its memory grows with depth. InfiLoop shares one scorer across all iterations and adds an exponential decay. This gives an exact two-variable recurrence whose memory does not depend on the number of iterations, so one rule applies at any depth. The decay is also what makes this aggregation trainable in a loop: with uniform weights (β=1, Attention Residuals with a shared query) the model reaches 38.4% at H3 L6, against 82.1% for InfiLoop (Table 2).
DenseFormer learns one scalar weight per pair of layers. The weights do not depend on the input and the depth is fixed. InfiLoop computes a content-dependent weight per token and is defined for any number of iterations. Removing the content score decreases accuracy by 6.8% (Appendix D).
FPRM stabilizes the iteration with pre-norm, residual scaling and damped fixed-point updates, and halts on convergence. While FPRM introduces scaling on residuals, it does not introduce a residual connection designed for looping, and Pre-LN is known to be insufficient for deep models (Curse of Depth; DepthBench). InfiLoop learns from the task loss which candidate states to keep. InfiLoop reaches 90.9% at 1,872 layers, above FPRM's 90.1% at 3,744 layers, and 92.74% against 91.63% at 24,960 layers (Section 4.4).
A gated update sets the step size from the current input and state. The step size of InfiLoop, γt=et/(βDt−1+et) (Eq. 5), is set relative to the accumulated history weight, which makes the update an exact normalized average over the whole history. Following Reviewer FzAQ, we compared InfiLoop with two learned sigmoid gates on the same backbone and training:
W2. "The experimental scope remains limited to Sudoku, Maze, and ARC, while the main depth-scaling and stability analyses focus on Sudoku-Extreme. It is unclear whether InfiLoop offers similar benefits on more general tasks or standard, non-nested looped Transformers."
A2. We applied InfiLoop to a standard looped Transformer with a single recurrent state: URM, with its official code and recipe and only the connection between loop iterations changed.
InfiLoop improves the single-state looped Transformer from 77.6% to 84.3%. InfiLoop is defined for a single recurrent state (Eq. 4, Section 3.3), and TRM applies it to each of its two states; Table 2 shows that each state alone also improves with depth.
We run the depth and perturbation analyses on Sudoku-Extreme because every prediction can be checked exactly, which lets us follow individual puzzles and cells across tens of thousands of layers. The depth benefit also holds on ARC-AGI: with 4× the training depth, InfiLoop improves from 49.1% to 52.4% on ARC-AGI-1 and from 13.6% to 14.2% on ARC-AGI-2.
Furthermore, to better address your concern regarding general language tasks and standard non-nested architectures, we conducted continual pre-training (CPT) on Qwen3-1.7B-Base to integrate InfiLoop. Specifically, starting from the 28-layer Qwen3-1.7B-Base without adding any extra layers, all recurrent iterations share the weights of the original layers. We trained on 2B tokens sampled from a Nemotron data mixture, a global batch size of 64 × 1024 tokens, and a cosine learning rate schedule with 2% warmup decaying to 10% of the peak rate (backbone lr = 1e-5, query lr = 1e-3). During training, the number of recurrent iterations K is uniformly sampled from {1,2,3,4,5} at each optimizer step. We evaluated the resulting models across different inference loop counts (K∈{1,2,3,5}) on MATH-500:
With more loops at inference, the standard loop degrades from K=3 to K=5 (43.2% → 40.4%), while InfiLoop keeps improving (44.2% → 45.8%).