- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Gated Looped Attention (G-Loop), which places recurrence inside the attention sublayer and applies the FFN only once after the loop in each layer. At each recurrent step, an input-dependent gate combines the fixed layer input with the evolving state to construct the values, followed by a damped state update. Their 5.7M-parameter model achieves 72.5% accuracy on Sudoku-Extreme and 85.9% on Maze-Hard, with lower inference FLOPs than HRM and TRM-Att.
Critical strengths
1. The ablation studies are thorough, well-grounded in their design, and logically sound.
Table 3 goes beyond a simple comparison between "gated" and "non-gated" variants. It evaluates several alternatives, including using only anchors, fixed scalar mixing, learnable scalar mixing, static channel-wise mixing, and linear concatenation. Taken together, these results suggest that the performance gains are unlikely to be merely a byproduct of an increased number of parameters. For researchers working on small-scale recurrent reasoning models, the finding that "retaining the input path during recurrence while adaptively integrating intermediate results" is both concrete and practical.
2. The model offers a meaningful accuracy–compute trade-off on the evaluated grid tasks.
While G-Loop does not achieve the highest absolute accuracy, it performs competitively: on Sudoku, it trails TRM-Att by only 2.2 percentage points, it also performs comparably on Maze-Hard. Meanwhile, the inference cost is drastically reduced—from approximately 457.2 to 43.0 GFLOPs for Sudoku, and from 5,350.3 to 286.1 GFLOPs for Maze-Hard.
Critical weaknesses
1. The arguments regarding looping and FFN lack strong supporting evidence.
The authors cite Saunshi et al. [1] to argue that looping helps reasoning but hinders language modeling, and cite Lan et al. [2] to suggest that sharing the FFN is a main source of this loss. They then use these findings to motivate keeping the FFN outside the loop. However, neither study clearly distinguishes whether the observed performance loss stems from a reduction in the total number of parameters or from the repeated use of the same set of weights.
The study [1] does not control computational cost or parameter simultaneously. The loop model exhibits higher perplexity than the baseline matched for computational cost. Conversely, when compared to a baseline matched for parameter count, the loop model actually improves perplexity. Consequently, these results do not definitively demonstrate that looping itself is detrimental to language modeling. More recent work [3], reports lower language-modeling loss and better downstream performance with looped MoE models while approximately matching FLOPs and parameters.
[2] presents a similar issue: sharing only the FFN reduces the parameter count from 89M to 38M, while sharing only the attention mechanism leaves 64M parameters.
2. The strongest evidence is concentrated on two tasks governed by fixed rules; it remains unclear whether this design is broadly applicable.
I do not consider the use of Sudoku and mazes to be a flaw in itself. These tasks are good for testing iterative constraint propagation, and focused evaluation can certainly support valuable research. My concern is that the strongest evidence comes from fixed-size Sudoku-Extreme and Maze-Hard tasks. It is therefore difficult to determine whether the proposed design are generally effective, or if they are simply well-suited for grid-based problem-solving scenarios.
[1] Reasoning with latent thoughts: On the power of looped transformers.
[2] ALBERT: A lite BERT for self-supervised learning of language representations.
[3] SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers
Questions for authors
- Have you experimented with running the FFN every two, four, or eight attention steps, rather than just the two extremes shown in Table 5? Keeping the number of attention steps constant would help reveal the benefits and costs of more frequent FFN execution. Furthermore, adjusting the loop depth while maintaining a similar FLOP budget could provide insight into the optimal allocation of computational resources. These results will directly influence designs that rely solely on attention.
3 comments
Thank you for your response which resolves my questions. I keep my original score.
Resolved questions. Requesting remaining concerns. W2 results beyond grid tasks
Thank you for your quick reply, and we are glad the new FFN-schedule experiments resolved your questions.
Since you kept your original score, could you tell us which concerns remain under W1 or W2? Our W2 results (applicability beyond fixed-rule grid tasks) are in the general responses, so we summarize them here in case they were easy to miss:
- 1-D content-dependent reasoning (-hop induction, Saunshi et al. [1]). G-Loop reaches %. That is 6.1 points above the matched non-adaptive control (%; % CI [5.1, 7.0]) and 9.2 points above a non-looped model (%). The state-only loop stays near chance (32.1%, chance 25%).
- Language understanding (DeBERTa-v3-base, GLUE-6 and SQuAD 2.0). We added two controls: a widened FFN with the same parameter count, and linear mixing with the same parameters and compute. G-Loop beats both, by +0.93 and +0.63 on GLUE-6 and by +0.70 and +0.50 on SQuAD 2.0. Its lead over linear mixing grows with loop depth, from +0.05 at $N=1 $ to about +0.6 by $N=4 $, where it levels off.
For W1, we replaced the argument from prior work with the controlled FFN-placement experiments in our reply. If any concern still holds the score below acceptance, please let us know and we will address it before the discussion period ends.
Response to Reviewer 9U5T
Thank you for the review. Your Q1 led to our most informative new experiment on FFN placement, and it changed how we state that claim.
The protocol for the new runs is in General response 1/2: 5 seeds, fixed depth, and a learning rate selected per model on held-out training puzzles. Results marked † use the submission's recipe instead (3 seeds unless stated).
W1: the motivation from prior work. We agree and have rewritten the paragraph. Saunshi et al.'s looped models trail compute-matched baselines in perplexity but beat parameter-matched ones. In ALBERT, sharing the FFN removes far more parameters than sharing attention (from 89M to 38M, against 64M). Neither result isolates a cost of weight reuse at fixed capacity, and SMELT reports gains from looping under closely matched budgets. We now cite all three as context and base the FFN-placement claim on the controlled experiments below.
W2: evaluation beyond grid puzzles. p-hop induction extends the evaluation to content-dependent sequence reasoning (General response 1/2), and the matched DeBERTa controls extend the value-pathway evidence to language understanding (General response 2/2).
Q1: how often does the FFN need to run? We ran both settings you asked for. In short: at fixed depth, the better-integrated every-step FFNs show no statistically resolved accuracy difference from G-Loop, at 2.15x the FLOPs on Sudoku. At approximately matched FLOPs, a sparse schedule also shows no resolved difference from G-Loop, and FFN-heavy schedules are worse under the plain-residual integration we tested.
(i) Fixed depth, N=32, submission's recipe (†).
From left to right the cost grows from 43 to 92 GFLOP on Sudoku and from 280 to 463 GFLOP on Maze. The Maze differences are not statistically resolved (largest +0.8, 95% CI [-1.6, 3.3]).
The Sudoku decline is sensitive to optimization and to how the FFN enters the loop. Write z~ for the damped attention update and f for the FFN with its LayerNorm. With per-model learning rates and 5 seeds, the every-step FFN gives:
The scaled-residual and in-candidate variants differ from G-Loop by -0.5 (95% CI [-1.5, 0.4]) and -0.3 ([-1.3, 0.6]), respectively. This qualifies our reading of Table 5, which we have revised.
(ii) Approximately matched FLOPs. Sudoku-Extreme, each model trained at its own depth with its own learning rate. The in-loop schedules use the plain residual and run the FFN every m-th step and at the last step.
Accuracy rises as compute moves from FFN calls to attention steps, and every-8 and once-per-layer show no resolved difference (+0.1, 95% CI [-0.9, 1.1]). One caveat: the scaled-residual and in-candidate variants were not run across this sweep, so it does not establish an optimal allocation. On Maze (†, 5 seeds), cutting depth from 32 to 19 to fit an FFN at every step costs 3.5 points (95% CI [-4.8, -2.3]) at nearly the same FLOPs.
Two further checks. Extra attention beyond N=32 adds little (73.3 ± 0.4 at N=48), and removing the FFN entirely costs 10.6 points, with 57% fewer parameters.
Our revised claim is that one FFN per layer suffices for competitive accuracy on these benchmarks, at less than half the compute of the every-step variants on Sudoku.