acceptodds
Under review as a conference paper at ICLR 2027

Loop Attention, not the Block

11 views

Abstract

Recurrent-depth Transformers increase depth by repeating shared parameters, but most loop an entire block. That means recomputing both self-attention and feed-forward layers at every step, which inflates compute and often hurts language modeling. We move recurrence entirely inside self-attention. We introduce Gated Looped Attention (G-Loop), which keeps the feed-forward network outside the loop and repeatedly remixes the input values with the evolving state. Learned token- and channel-wise gates decide how much of each source to write at each step, trained end-to-end with backpropagation through time. As a result, a standard encoder Transformer handles both algorithmic reasoning and general language tasks without specialized routing and with modest parameter overhead. At 32 recurrent steps, G-Loop cuts inference FLOPs by up to 94.7% and memory by up to 73.9% compared to HRM and TRM-Att. It uses about 19% fewer parameters than TRM-Att to reach 85.9% on Maze-Hard and 72.5% on Sudoku-Extreme, beating its reported Maze score and closely approaching its Sudoku score. Under our cost protocol, the newer recurrent reasoners that score higher spend 57× to 4,900× as many inference FLOPs. Added to DeBERTa-v3-base, G-Loop improves the six-task GLUE development average by 1.25 points and SQuAD 2.0 F1 by 0.9 points over the baseline. Looping attention alone slashes compute, and gating the value pathway preserves the expressivity needed to reason.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Looped Transformers, Recurrent-Depth Transformers, Attention-Only Recurrence, Gated Attention, Iterative Reasoning, Algorithmic Reasoning, Token Collapse, Efficient Inference, Language Understanding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9U5TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Gated Looped Attention (G-Loop), which places recurrence inside the attention sublayer and applies the FFN only once after the loop in each layer. At each recurrent step, an input-dependent gate combines the fixed layer input with the evolving state to construct the values, followed by a damped state update. Their 5.7M-parameter model achieves 72.5% accuracy on Sudoku-Extreme and 85.9% on Maze-Hard, with lower inference FLOPs than HRM and TRM-Att.

Critical strengths

1. The ablation studies are thorough, well-grounded in their design, and logically sound.

Table 3 goes beyond a simple comparison between "gated" and "non-gated" variants. It evaluates several alternatives, including using only anchors, fixed scalar mixing, learnable scalar mixing, static channel-wise mixing, and linear concatenation. Taken together, these results suggest that the performance gains are unlikely to be merely a byproduct of an increased number of parameters. For researchers working on small-scale recurrent reasoning models, the finding that "retaining the input path during recurrence while adaptively integrating intermediate results" is both concrete and practical.

2. The model offers a meaningful accuracy–compute trade-off on the evaluated grid tasks.

While G-Loop does not achieve the highest absolute accuracy, it performs competitively: on Sudoku, it trails TRM-Att by only 2.2 percentage points, it also performs comparably on Maze-Hard. Meanwhile, the inference cost is drastically reduced—from approximately 457.2 to 43.0 GFLOPs for Sudoku, and from 5,350.3 to 286.1 GFLOPs for Maze-Hard.

Critical weaknesses

1. The arguments regarding looping and FFN lack strong supporting evidence.

The authors cite Saunshi et al. [1] to argue that looping helps reasoning but hinders language modeling, and cite Lan et al. [2] to suggest that sharing the FFN is a main source of this loss. They then use these findings to motivate keeping the FFN outside the loop. However, neither study clearly distinguishes whether the observed performance loss stems from a reduction in the total number of parameters or from the repeated use of the same set of weights.

The study [1] does not control computational cost or parameter simultaneously. The loop model exhibits higher perplexity than the baseline matched for computational cost. Conversely, when compared to a baseline matched for parameter count, the loop model actually improves perplexity. Consequently, these results do not definitively demonstrate that looping itself is detrimental to language modeling. More recent work [3], reports lower language-modeling loss and better downstream performance with looped MoE models while approximately matching FLOPs and parameters.

[2] presents a similar issue: sharing only the FFN reduces the parameter count from 89M to 38M, while sharing only the attention mechanism leaves 64M parameters.

2. The strongest evidence is concentrated on two tasks governed by fixed rules; it remains unclear whether this design is broadly applicable.

I do not consider the use of Sudoku and mazes to be a flaw in itself. These tasks are good for testing iterative constraint propagation, and focused evaluation can certainly support valuable research. My concern is that the strongest evidence comes from fixed-size Sudoku-Extreme and Maze-Hard tasks. It is therefore difficult to determine whether the proposed design are generally effective, or if they are simply well-suited for grid-based problem-solving scenarios.

[1] Reasoning with latent thoughts: On the power of looped transformers.

[2] ALBERT: A lite BERT for self-supervised learning of language representations.

[3] SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

Questions for authors
  1. Have you experimented with running the FFN every two, four, or eight attention steps, rather than just the two extremes shown in Table 5? Keeping the number of attention steps constant would help reveal the benefits and costs of more frequent FFN execution. Furthermore, adjusting the loop depth while maintaining a similar FLOP budget could provide insight into the optimal allocation of computational resources. These results will directly influence designs that rely solely on attention.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9U5T

Thank you for the review. Your Q1 led to our most informative new experiment on FFN placement, and it changed how we state that claim.

The protocol for the new runs is in General response 1/2: 5 seeds, fixed depth, and a learning rate selected per model on held-out training puzzles. Results marked †\dagger use the submission's recipe instead (3 seeds unless stated).

W1: the motivation from prior work. We agree and have rewritten the paragraph. Saunshi et al.'s looped models trail compute-matched baselines in perplexity but beat parameter-matched ones. In ALBERT, sharing the FFN removes far more parameters than sharing attention (from 89M to 38M, against 64M). Neither result isolates a cost of weight reuse at fixed capacity, and SMELT reports gains from looping under closely matched budgets. We now cite all three as context and base the FFN-placement claim on the controlled experiments below.

W2: evaluation beyond grid puzzles. pp-hop induction extends the evaluation to content-dependent sequence reasoning (General response 1/2), and the matched DeBERTa controls extend the value-pathway evidence to language understanding (General response 2/2).

Q1: how often does the FFN need to run? We ran both settings you asked for. In short: at fixed depth, the better-integrated every-step FFNs show no statistically resolved accuracy difference from G-Loop, at 2.15x the FLOPs on Sudoku. At approximately matched FLOPs, a sparse schedule also shows no resolved difference from G-Loop, and FFN-heavy schedules are worse under the plain-residual integration we tested.

(i) Fixed depth, N=32N=32, submission's recipe (†\dagger).

FFN runsOnce (G-Loop)Every 8Every 4Every 2Every step
Sudoku-Extreme (%)72.672.871.769.367.9
Maze-Hard (%)86.086.286.686.486.8

From left to right the cost grows from 43 to 92 GFLOP on Sudoku and from 280 to 463 GFLOP on Maze. The Maze differences are not statistically resolved (largest +0.8, 95% CI [-1.6, 3.3]).

The Sudoku decline is sensitive to optimization and to how the FFN enters the loop. Write z~\tilde z for the damped attention update and ff for the FFN with its LayerNorm. With per-model learning rates and 5 seeds, the every-step FFN gives:

Every-step FFN, integrated asExact (%)
Plain residual: z~+f(z~)\tilde z + f(\tilde z)70.6 ±\pm 0.8
Scaled residual: z~+α⊙f(z~)\tilde z + \alpha\odot f(\tilde z)72.3 ±\pm 0.6
Inside the attention candidate72.5 ±\pm 0.7
G-Loop (FFN once per layer)72.8 ±\pm 0.7

The scaled-residual and in-candidate variants differ from G-Loop by -0.5 (95% CI [-1.5, 0.4]) and -0.3 ([-1.3, 0.6]), respectively. This qualifies our reading of Table 5, which we have revised.

(ii) Approximately matched FLOPs. Sudoku-Extreme, each model trained at its own depth with its own learning rate. The in-loop schedules use the plain residual and run the FFN every mm-th step and at the last step.

FFN scheduleAttention steps NNGFLOPExact (%)
Every step1543.358.2 ±\pm 0.7
Every 22041.865.8 ±\pm 0.9
Every 42543.569.2 ±\pm 0.8
Every 82842.672.9 ±\pm 0.7
Once, after the loop (G-Loop)3243.072.8 ±\pm 0.7

Accuracy rises as compute moves from FFN calls to attention steps, and every-8 and once-per-layer show no resolved difference (+0.1, 95% CI [-0.9, 1.1]). One caveat: the scaled-residual and in-candidate variants were not run across this sweep, so it does not establish an optimal allocation. On Maze (†\dagger, 5 seeds), cutting depth from 32 to 19 to fit an FFN at every step costs 3.5 points (95% CI [-4.8, -2.3]) at nearly the same FLOPs.

Two further checks. Extra attention beyond N=32N=32 adds little (73.3 ±\pm 0.4 at N=48N=48), and removing the FFN entirely costs 10.6 points, with 57% fewer parameters.

Our revised claim is that one FFN per layer suffices for competitive accuracy on these benchmarks, at less than half the compute of the every-step variants on Sudoku.

Thank you for your response which resolves my questions. I keep my original score.

AuthorsOpenReview ↗

Resolved questions. Requesting remaining concerns. W2 results beyond grid tasks

Thank you for your quick reply, and we are glad the new FFN-schedule experiments resolved your questions.

Since you kept your original score, could you tell us which concerns remain under W1 or W2? Our W2 results (applicability beyond fixed-rule grid tasks) are in the general responses, so we summarize them here in case they were easy to miss:

  • 1-D content-dependent reasoning (pp-hop induction, Saunshi et al. [1]). G-Loop reaches 94.1±0.794.1\pm 0.7%. That is 6.1 points above the matched non-adaptive control (88.0±0.688.0\pm 0.6%; 9595% CI [5.1, 7.0]) and 9.2 points above a non-looped model (84.9±0.884.9\pm 0.8%). The state-only loop stays near chance (32.1%, chance 25%).
  • Language understanding (DeBERTa-v3-base, GLUE-6 and SQuAD 2.0). We added two controls: a widened FFN with the same parameter count, and linear mixing with the same parameters and compute. G-Loop beats both, by +0.93 and +0.63 on GLUE-6 and by +0.70 and +0.50 on SQuAD 2.0. Its lead over linear mixing grows with loop depth, from +0.05 at $N=1 $ to about +0.6 by $N=4 $, where it levels off.

For W1, we replaced the argument from prior work with the controlled FFN-placement experiments in our reply. If any concern still holds the score below acceptance, please let us know and we will address it before the discussion period ends.

Reviewer_A99xOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether recurrent-depth Transformers can perform iterative reasoning without rerunning the full Transformer block. The authors propose Gated Looped Attention (G-Loop), which restricts recurrence to self-attention and applies the FFN once per layer. A token- and channel-wise gate mixes the fixed layer input (anchor) with the evolving recurrent state before computing attention values, and a damped update controls state evolution. A Dobrushin-contraction analysis gives sufficient conditions for state-only recurrent attention to reduce token-representation differences, though it does not directly characterize G-Loop. On Sudoku-Extreme and Maze-Hard, G-Loop reaches 72.5% and 85.9% exact-solve accuracy with ~5.7M parameters and substantially lower inference FLOPs; ablations support adaptive value gating. DeBERTa-v3-base experiments show gains on six GLUE tasks and two SQuAD benchmarks, suggesting applicability beyond algorithmic puzzles.

Critical strengths
  1. Computationally efficient and conceptually simple recurrent reasoning. The core idea—repeating self-attention while executing the FFN only once—is intuitive and practically meaningful. Table 1 shows that G-Loop achieves 72.5% on Sudoku-Extreme and 85.9% on Maze-Hard, versus 74.7% and 85.3% for TRM-Att, while requiring approximately 10.6× and 18.7× fewer inference FLOPs, respectively. Table 5 further shows that moving the FFN inside the loop increases compute by 2.15× on Sudoku and 1.62× on Maze without consistent accuracy gains; Appendix D confirms agreement between analytical and measured FLOP counts. These results support a meaningful accuracy–efficiency tradeoff.

  2. Convincing ablations on adaptive value gating. The ablations investigate why the design works. Table 3 shows a clear progression from state-only values (0.0%) to anchor-only values (42.9%), static channel-wise blending (60.4%), and G-Loop (72.5%) on Sudoku-Extreme; linear concatenation reaches 65.1%, suggesting gains are not merely due to increased value-projection capacity. Table 5 shows a parameter-matched static-gating variant reaches only 62.8%. Table 4 indicates that inserting the anchor into attention values outperforms adding it directly to the state under the tested configurations. The token-spread measurements in Appendix F.2 provide additional mechanistic evidence. Collectively, these results make a compelling case that adaptive value routing, rather than merely added parameters, is an important contributor.

Critical weaknesses
  1. Generality beyond structured reasoning tasks is not yet convincingly established. The strongest evidence comes from two grid-based benchmarks with fixed rules, which cover a relatively narrow class of reasoning problems. The DeBERTa experiments attempt to address this limitation but are difficult to interpret: G-Loop adds approximately 20% more parameters and potentially up to 16 attention iterations per modified layer, without a parameter- or compute-matched language baseline. The language implementation also differs materially from the puzzle implementation, including anchor-derived keys, fixed damping, and one-step phantom gradients instead of backpropagation through the loop. Reported improvements are modest, and per-seed results and inference costs were not retained. Thus, the language experiments suggest potential benefits but do not establish a favorable accuracy–compute tradeoff for general language understanding.

  2. The theoretical explanation does not fully establish that the identified failure mechanism is inherent to attention-only recurrence. Section 2.2 and Appendix A derive a conditional contraction bound for state-only attention under a damped update. However, its applicability depends on realized normalization amplification, projection norms, and attention overlap; the paper does not report the quantities needed to verify whether the sufficient condition holds along observed trajectories. More importantly, the contraction mechanism partly depends on the damped update. The paper acknowledges that additive residual updates do not admit the same guarantee, yet no state-only attention recurrence with an additive residual is tested. The evidence therefore supports the importance of anchoring for the proposed damped recurrence, but not whether the same problem—and need for gated anchoring—extends to alternative attention-only recurrent formulations.

Questions for authors
  1. How much of the language-task improvement is attributable specifically to G-Loop, rather than additional computation or capacity? Table 2 reports consistent improvements over DeBERTa-v3-base, but Appendix G indicates that realized iterations, inference cost, and per-seed scores were not retained. Could the authors provide realized average attention iterations and inference FLOPs or latency, together with variability estimates? A parameter- or compute-matched control would be particularly informative. This would clarify whether G-Loop is an efficient, generally useful replacement for standard attention or a more expensive modification that improves accuracy.

  2. To what extent does the theoretical contraction analysis explain the measured state-only collapse, and is the failure specific to damped recurrence? Proposition 2 gives a sufficient condition involving attention overlap, normalization amplification, and projection norms. Do trained state-only models satisfy this condition across their recurrent trajectories? Does state-only attention exhibit comparable collapse when the damped update is replaced by a conventional additive residual? These results would clarify whether the proposed gate addresses a general limitation of attention-only recurrence or primarily compensates for the particular recurrent update adopted in this paper.

4 comments
AuthorsOpenReview ↗

Response to Reviewer A99x

Thank you for the careful reading, especially of the theory. It led us to test Proposition 2 on trained models and to narrow two claims.

Terms from General response 1/2: the matched control (linear mixing in language) has the gate's parameter count and per-step cost but a fixed blend, and †\dagger marks the submission's recipe (3 seeds).

W1 / Q1: attribution of the language gain, and the accuracy-compute trade-off. At fixed N=16N=16, G-Loop exceeds linear mixing by +0.63 GLUE-6 (95% CI [0.41, 0.85]) at matched parameter count and nominal per-step projection cost, positive in all five seed pairs. The parameter-matched widened FFN also trails G-Loop, by 0.93 points. These controls support a benefit from adaptive value construction beyond parameter and iteration counts (details in General response 2/2).

On your distinction between an efficient replacement and a more expensive modification, the language evidence supports the second. With early exit, G-Loop runs 7.4 steps per layer and reaches 88.95 GLUE-6 at 3.9x the baseline FLOPs and 6.6x the latency. Under the same stopping rule it has higher mean accuracy than linear mixing and uses fewer steps.

Section 2.1 now states how the language operator differs from the puzzle loop, and the abstract's description of training through the loop is restricted to the puzzle models. The pp-hop task (General response 1/2) extends the matched comparison beyond grids.

W2 / Q2 (a): does the condition of Proposition 2 hold on trained trajectories? As we stated it, rarely. A tighter bound for the same update is below one on every analyzed step.

We measured the factors of μk=ΛkκVω(1−δk)\mu_k = \Lambda_k \kappa_V \omega (1-\delta_k) on the submission's three state-only checkpoints: 36,000 layer/segment trajectories and 1,152,000 steps over 1,000 test puzzles. A float64 replay reproduces the production predictions. Pooled medians over steps:

Λk\Lambda_kκV\kappa_Vω\omega1−δk1 - \delta_kμk\mu_k
1.181.301.010.5750.89

The condition as stated, μk<αmin⁡/αmax⁡\mu_k < \alpha_{\min}/\alpha_{\max} (0.81 to 0.85 depending on the layer), holds on 2.8% of steps, and the bound's 32-step product exceeds one on every trajectory. So our stated condition is too conservative to explain the contraction we observe.

The looseness comes mainly from the per-group damping inequality, where we combined the smallest and largest α\alpha as if one direction could see both. For Dα=diag(α)D_\alpha = \mathrm{diag}(\alpha), the tight factor given only the norms of the state and candidate differences is

ϕ(x)=max⁡∥u∥=1(∥(I−Dα)u∥+x∥Dαu∥)≤1−αmin⁡+αmax⁡x,\phi(x) = \max_{\lVert u\rVert=1} \left( \lVert(I-D_\alpha)u\rVert + x\lVert D_\alpha u\rVert \right) \le 1 - \alpha_{\min} + \alpha_{\max} x,

with equality when α\alpha is shared, and diam(zk+1)≤ϕ(μk)⋅diam(zk)\mathrm{diam}(z_{k+1}) \le \phi(\mu_k)\cdot\mathrm{diam}(z_k). The trained model is unchanged. On these trajectories ϕ(μk)\phi(\mu_k) is below one at every step (largest value 0.993). Products taken along each trajectory have median 0.136, against a median measured diameter ratio of 0.045.

This is an a posteriori explanation built from realized normalization and attention factors, not a convergence guarantee. The normalization and projection terms alone do not certify contraction (the product of their medians is about 1.55); the measured attention overlap is what makes the refined bound contractive. The proof is in the revised Appendix A and these diagnostics are in Appendix J.5.

W2 / Q2 (b): is the failure specific to damping? Partly. With state-only values on Sudoku-Extreme (updates as defined in our reply to QEhK):

UpdateExact (%)Spread after 32 steps (†\dagger)
Damped0.40.055
Post-LN4.00.12
Additive51.00.52

Accuracy is from the new protocol (5 seeds). Spread (normalized state, relative to its initial value) is from separate submission-recipe diagnostics, so the two columns are not from the same checkpoints. Effective rank falls from 28 to 11 under damping and from 30 to 20 under the additive update (†\dagger). Post-LN also contracts severely, which we report as an observation outside Proposition 2. The proposition already concerns the damped loop. What we have narrowed is the surrounding text, which read as a statement about attention-only recurrence in general.

Does the gate fix a general limitation, or compensate for our update? Non-adaptive anchored values already keep nearly as much normalized spread as gated values (0.77-0.78 vs 0.79-0.81, †\dagger), so retained spread is associated with anchoring, and we removed the sentence crediting the gate with repairing it. What the gate adds is accuracy: over the matched control it gains +7.3 (damped), +5.7 (post-LN) and +4.9 (additive) points, with every interval excluding zero (table in our reply to QEhK). The gain is there under the additive update too, where the state-only loop does not fail severely.

AuthorsOpenReview ↗

Point-by-point map of where each concern is addressed

Dear Reviewer A99x,

As the discussion closes today, here is a short map of where each of your points is addressed. We ran every control and measurement you asked for, and all of your points are addressed in our responses and the revised paper (changes to the paper are marked in blue):

  • W1 / Q1 (attribution of the language gain): we added parameter- and compute-matched controls, realized iterations, inference FLOPs and latency, and per-seed variability. G-Loop leads the parameter-matched widened FFN by +0.93 GLUE-6 and the parameter- and compute-matched loop by +0.63 (95% CI [0.41, 0.85]), in all five seed pairs. On your distinction, it is a more expensive modification that improves accuracy (3.9×3.9\times the baseline FLOPs with early exit), yet more accurate and cheaper than the matched loop (7.4 vs. 9.0 steps). Section 2.1 states the implementation differences you listed. Discussion: our reply to you; General response 2/2. Paper: Section 2.1; Appendix J.6.

  • W1 (beyond grid benchmarks): on pp-hop induction, the gate beats the matched control by +6.1 (95% CI [5.1, 7.0]). Discussion: General response 1/2. Paper: Appendix J.3.

  • W2 / Q2 (does the condition hold along observed trajectories?): measured over 1,152,000 steps, the condition as stated holds on 2.8% of steps; a tighter bound for the same update holds at every step and accounts for the measured collapse a posteriori. Discussion: our reply to you. Paper: Appendix A; Appendix J.5.

  • W2 / Q2 (is the failure specific to damped recurrence?): with an additive residual, state-only attention solves 51.0% (vs. 0.4% damped), and we narrowed Section 2.2 accordingly. The gate does not primarily compensate for our update: its benefit extends to the alternative formulations, beating the matched control under all three updates (+4.9 to +7.3, all 95% CIs exclude zero). Discussion: our reply to you. Paper: Section 2.2; Appendix J.2.

Thank you for helping strengthen the paper. If any concern remains, we will gladly address it in the remaining time.

Thank you for your detailed response and additional analyses, which address most of my concerns. However, given the current completeness of the proposed method, I will maintain my original score.

Thank you for your careful review. Your questions helped improve the paper.

Reviewer_QEhKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Gated Looped Attention (G-Loop), a recurrent Transformer architecture that iterates self-attention while keeping the feed-forward network outside the recurrence, using adaptive value gating to combine the original input and evolving hidden states. The authors provide a theoretical analysis of representation contraction in damped attention-only recurrence and introduce a gating mechanism intended to preserve information during iterative computation. Experiments on Sudoku-Extreme, Maze-Hard, and DeBERTa-based language understanding benchmarks demonstrate competitive reasoning performance with reduced inference FLOPs and improvements over a standard pretrained encoder.

Critical strengths
  1. Computational efficiency of attention-only recurrence.

The proposed architecture demonstrates that iterative attention can achieve competitive reasoning performance without repeatedly executing the FFN, substantially reducing computational costs compared to full-block recurrence. The matched-parameter ablations further support the efficiency benefits of this design, making it a potentially useful direction for computationally efficient recurrent models.

  1. Systematic analysis and ablation of the gated value mechanism.

The paper provides a theoretically motivated explanation for representation contraction in damped attention recurrence and evaluates several alternative mechanisms for incorporating the original input. The controlled ablations on Sudoku-Extreme provide convincing evidence that adaptive value gating improves performance over static mixtures and alternative input-injection strategies.

Critical weaknesses
  1. Insufficiently controlled experimental validation on language understanding tasks.

Although G-Loop improves the reported DeBERTa results across GLUE and SQuAD benchmarks, the comparison does not control for additional parameters or inference computation, and the language implementation differs substantially from the architecture evaluated on reasoning tasks. Without compute-matched baselines, language-specific ablations, inference-efficiency measurements, or statistical variability, it is difficult to establish whether these improvements arise from the proposed gating mechanism or simply from increased model capacity and iterative computation.

  1. Limited empirical evidence supporting generalizability and superiority over existing recurrent architectures.

The reasoning experiments focus exclusively on two fixed-rule grid benchmarks, while comparisons with existing recurrent models primarily combine published accuracies with separately measured inference costs rather than evaluating models under consistently matched experimental budgets. Although the internal ablations are informative, the absence of broader reasoning benchmarks and stronger comparisons against alternative recurrence mechanisms, particularly standard attention recurrence with additive residual connections, limits the evidence that the proposed design offers a generally preferable accuracy–efficiency trade-off.

Questions for authors
  1. The representation contraction analysis relies on the damped update, whereas conventional Transformer recurrence often uses additive residual connections that may avoid the same contraction behavior. Can the authors provide experimental comparisons against attention-only recurrence with standard additive residual updates, both with and without adaptive value gating, under matched parameter and computational budgets?

  2. In the DeBERTa experiments, can the authors provide compute- or parameter-matched baselines, language-specific gating ablations, inference FLOPs and latency, and per-seed performance variability? These results would help determine whether the reported improvements are attributable to the proposed gated recurrence rather than the additional parameters and computational depth.

4 comments
AuthorsOpenReview ↗

Response to Reviewer QEhK

Thank you for the detailed review. Your questions led to two of the main changes in the revision: matched controls for the language experiments, and an additive-residual control that made us narrow our claim about state-only loops.

One term used below: the matched non-adaptive control matches the gate's added parameter count and nominal per-step projection cost, but its blend of anchor and state is fixed across tokens and steps. Its definition and the protocol for the new runs are in General response 1/2.

W1 / Q2: what explains the language gain, what it costs, and how much it varies. General response 2/2 adds both controls you asked for: (i) a parameter-matched widened FFN, and (ii) a recurrent linear-mixing model matched to G-Loop in parameters and nominal per-step projection cost. The second is the value-pathway ablation in the language setting, and G-Loop exceeds it by +0.63 GLUE-6 (95% CI [0.41, 0.85]), in all five seed pairs. The same comment reports seed SDs for both evaluation modes, paired intervals on GLUE-6 and SQuAD 2.0, and realized iterations, FLOPs and latency. Section 2.1 now makes explicit how the language operator differs from the puzzle loop.

Q1 (and the additive-residual part of W2): does the result depend on the damped update? In short: the severe state-only failure does depend on the update, but the benefit of the gate persists under all three updates we tested.

We ran attention-only recurrence on Sudoku-Extreme under three updates, where zz is the state and cc the attention candidate. Each update was trained with state-only values, with the matched non-adaptive control, and with the adaptive gate.

UpdateState-onlyMatched controlAdaptive gateGate vs control [95% CI]
Damped (paper): z+α⊙(c−z)z + \alpha\odot(c - z)0.4 ±\pm 0.265.5 ±\pm 0.772.8 ±\pm 0.7+7.3 [6.3, 8.3]
Additive: z+cz + c51.0 ±\pm 0.861.3 ±\pm 0.566.2 ±\pm 0.5+4.9 [4.2, 5.6]
Post-LN: LN(z+c)\mathrm{LN}(z + c)4.0 ±\pm 0.664.4 ±\pm 0.670.1 ±\pm 0.5+5.7 [4.8, 6.5]

Exact-solve accuracy (%), 5 seeds. Control and gate: 5.76M parameters, 43.0 GFLOP. State-only: 4.94M, 30.3 GFLOP.

The severe state-only failure depends on the update. With the additive residual the state-only loop solves 51.0%, so the results do not support treating this failure as inherent to attention-only recurrence. Maze-Hard shows the same contrast: 0.4% under damping and 65.2% under the additive update (†\dagger: submission's recipe, 3 seeds). The submission listed this missing control as a limitation. We have now revised Section 2.2 and the second contribution accordingly.

The gate's benefit persists without damping. The gate is ahead of the matched control under all three updates, and every interval excludes zero. Two more settings agree: the additive update without spectral normalization gives +4.1 ([2.7, 5.5], 3 seeds), and Maze-Hard under damping gives +5.7 (86.0 ±\pm 0.7 vs 80.3 ±\pm 1.0, †\dagger).

The state-only model's smaller budget does not close the gap. The state-only model is smaller and cheaper than the gated one, so we also enlarged an additive state-only model to the gate's budget (FFN expansion raised from 4x to 5x, trained at N=46N=46; about 5.76M parameters and 43.3 GFLOP). It reaches 52.7 ±\pm 0.8, against 66.2 for the gate.

Why we keep damping. In these experiments the update involves a trade-off. The damped and post-LN variants perform poorly with state-only values, but with the gate they stay accurate at twice the trained depth (−0.8-0.8 and +0.4+0.4 points at N=64N=64). The additive variant avoids the severe state-only failure, but with the gate it is 6.6 points less accurate than damping and it loses 13.1 points at N=64N=64. We keep damping because damped + gate is the most accurate configuration we tested. Section 2.3 now motivates it this way, not as a necessity.

W2: evidence beyond grid benchmarks. We added two matched comparisons outside grids. On pp-hop induction, a one-dimensional task with content-dependent retrieval, the gate beats the matched control by 6.1 points (General response 1/2). The DeBERTa controls are in General response 2/2.

W2: comparison with existing recurrent models. We agree that this comparison is not budget-matched. The baselines' accuracies are published results, and their costs follow Appendix D's protocol, measured where an implementation was available; we did not retrain them under matched budgets. We have narrowed the claim accordingly: Table 1 situates G-Loop's accuracy and cost among published models. Our attribution claims rest on the internal matched comparisons, with the protocol and its submission-recipe exceptions stated explicitly.

AuthorsOpenReview ↗

Point-by-point map of where each concern is addressed

Dear Reviewer QEhK,

As the discussion closes today, here is a short map of where each of your points is addressed in our responses and the revised paper; we have addressed all of them (changes to the paper are marked in blue):

  • W1 / Q2 (language): we added compute- and parameter-matched baselines, a language-specific gating ablation, inference FLOPs and latency, and per-seed variability. Against a parameter-matched widened FFN and a parameter- and compute-matched gating ablation, G-Loop leads by +0.93 and +0.63 GLUE-6 (95% CI [0.41, 0.85] for the latter), in all five seed pairs. Section 2.1 now states how the language implementation differs from the reasoning one. Discussion: General response 2/2. Paper: Appendix J.6, Tables 20-22.

  • Q1 / W2 (alternative recurrence mechanisms): we ran attention-only recurrence with standard additive residual updates, plus damped and post-LN updates, each with and without adaptive value gating, under matched parameter and computational budgets. The gate beats the matched control under all three (+4.9 to +7.3, all 95% CIs exclude zero). FFN-in-loop recurrence gives no resolved gain at 2.15×2.15\times the FLOPs on Sudoku. Discussion: our reply to you; General response 1/2. Paper: Appendix J.2, Table 13; Appendix J.4.

  • W2 (beyond two fixed-rule grid benchmarks): on pp-hop induction, a 1-D content-dependent reasoning task, the gate beats the matched control by +6.1, 95% CI [5.1, 7.0]. Discussion: General response 1/2. Paper: Appendix J.3, Table 15.

  • W2 (matched budgets): our attribution claims rest on matched-budget controls (Tables 5, 13, 15, 18, 20); Table 1 only situates G-Loop among published results. Discussion: our reply to you. Paper: Section 3.2.

Thank you for helping strengthen the paper. If any concern remains, we will gladly address it in the remaining time.

Thank you for the additional experiments. The new controls address most of my concerns, particularly regarding the benefits of adaptive gating. I find the revised evidence convincing enough to raise my rating from 2 to 3.

Thank you for engaging with the new results and for updating your assessment. Your questions helped strengthen the paper.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response (1/2): new controls, protocol, and a task beyond grids

We thank all three reviewers for the careful and constructive reviews. Three questions ran through them, and we ran new experiments for each:

  1. Does the gain come from the gate, or from its extra parameters and compute? We added a control with the gate's parameters and per-step cost but no adaptivity (defined below). The gate beats it in all six settings where we ran both.
  2. Is the failure of the state-only loop inherent to looping attention? No. It depends on the update rule, and we have narrowed this claim (replies to QEhK and A99x).
  3. How often must the FFN run? Once per layer is enough for competitive accuracy here. Running it at every step costs 2.15x the FLOPs on Sudoku-Extreme with no statistically resolved accuracy gain (reply to 9U5T).

In language, G-Loop is a more accurate but more expensive modification of DeBERTa, not an efficient replacement (General response 2/2).

We have uploaded a revised paper with all changes marked in blue. The main text states the narrowed claims and the key new numbers, the proof of the tighter bound is in Appendix A, and the new controls with full tables and per-seed scores are in the new Appendix J.

Protocol for the new puzzle results. Unless stated otherwise: 5 seeds, fixed depth N=32N=32, early exit off. Every model and control gets its own learning rate, chosen from 0.25x, 0.5x, 1x and 2x the base rate (also 0.125x for the runs without spectral normalization). The choice is made by cross-entropy on 200 puzzles held out from the 1,000 training puzzles, and the model is then retrained on all 1,000. The test set is never used for selection. G-Loop selected its original rate; most controls selected 0.5x. For puzzle and pp-hop results we report mean ±\pm SD and 95% Welch confidence intervals (CI). Results marked †\dagger use the submission's recipe instead (3 seeds unless stated). Costs are analytic FLOPs. Under this protocol G-Loop scores 72.8 ±\pm 0.7 on Sudoku-Extreme; the submission's 72.5 ±\pm 0.8 was 3 seeds with early exit on.

Matched non-adaptive control. This control keeps the gate's parameter count and nominal per-step projection cost and removes its adaptivity. Its values are

vk=WV[(1−βℓ)⊙xˉ+βℓ⊙zˉk]+Uℓ[xˉ;zˉk],v_k = W^V[(1-\beta_\ell)\odot\bar x + \beta_\ell\odot\bar z_k] + U_\ell[\bar x; \bar z_k],

where βℓ\beta_\ell is learned per layer but fixed across tokens and steps, and Uℓ∈RD×2DU_\ell \in \mathbb{R}^{D \times 2D} has no bias. It can represent any linear map of anchor and state, and it adds 2D2+D2D^2+D parameters per layer, the same as the gate. What it lacks is a blend that changes with the token and the step.

We ran the gate against this control in six settings:

SettingGate minus matched control (points)
Sudoku-Extreme, damped update (paper)+7.3
Sudoku-Extreme, additive update+4.9
Sudoku-Extreme, post-LN update+5.7
Sudoku-Extreme, additive update, no spectral normalization (3 seeds)+4.1
Maze-Hard, damped update (†\dagger)+5.7
pp-hop induction+6.1

The gate is ahead in all six, and all reported 95% intervals exclude zero. The detailed comparisons are in our reply to QEhK, where the three update rules are also defined, and in the pp-hop table below.

A task beyond grids: pp-hop induction. To check that the result is not tied to grid structure, we use the task definition and generator of Saunshi et al. (2025, App. A.1). The input is a sequence of n=256n=256 tokens over 4 symbols, and the model predicts the symbol reached after p=32p=32 hops (chance is 25%). Retrieval is content-dependent and the input is one-dimensional. All models share the training data, update budget and learning-rate selection (5 seeds).

ModelParamsAccuracy (%)
Non-looped, 4 layers4.93M84.9 ±\pm 0.8
State-only loop (damped)4.93M32.1 ±\pm 1.8
Matched non-adaptive control5.75M88.0 ±\pm 0.6
G-Loop5.75M94.1 ±\pm 0.7

The gate again beats the matched control, by 6.1 points (95% CI [5.1, 7.0]). This extends the value-pathway result beyond grid structure, on a task that is still synthetic. We do not claim an efficiency advantage or superiority over published full-block pp-hop models.

AuthorsOpenReview ↗

General response (2/2): language controls, seed variability and inference cost

The language comparison in the submission (Table 2) matched initialization, data and recipe, but not parameters or compute, and it reported neither seed variability nor inference cost. We reran the DeBERTa-v3-base experiments (App. G) with two added controls, per-seed logging and measured cost.

Setup. Every arm starts from the same public checkpoint and receives the same 10k continued-pretraining updates and the same fine-tuning recipe. These are new fine-tuning runs with paired seeds: 5 for GLUE and 3 for SQuAD. "GLUE-6" is the six-task GLUE average of Table 2. Table 2 used early exit; here, as for the puzzles, the headline is fixed depth (N=16N=16 steps per layer), and early exit is reported separately. The new GLUE-6 means agree with Table 2 (87.73 and 88.98) to within 0.03 points in either mode.

  • Widened FFN (is it the parameters?): approximately the same 17.3M added parameters as G-Loop, spent on FFN width (3072 to 4096), with no recurrence.
  • Linear mixing (is it the iteration?): the same language loop, added parameter count and nominal per-step projection cost as G-Loop, with the gate replaced by a learned static blend plus an unrestricted linear projection of anchor and state. This is the language version of the matched non-adaptive control of General response 1/2.
ModelSteps / layerGFLOPLatency (ms)GLUE-6SQuAD 2.0 F1
DeBERTa-v3-base122.31187.73 ±\pm 0.1688.40 ±\pm 0.18
Widened FFN126.71288.05 ±\pm 0.1688.60 ±\pm 0.20
Linear mixing, N=16N=1616171.213288.35 ±\pm 0.1988.80 ±\pm 0.12
G-Loop, N=16N=1616171.213488.98 ±\pm 0.2089.30 ±\pm 0.28
Linear mixing, early exit9.0103.38588.31 ±\pm 0.1988.70 ±\pm 0.15
G-Loop, early exit7.487.87388.95 ±\pm 0.2189.20 ±\pm 0.29

Mean ±\pm SD over fine-tuning seeds. Costs are per 128-token GLUE input: analytic FLOPs, and batch-1 latency measured on an H100 in BF16. Step counts are averaged over the 11 modified layers. SQuAD uses 384 tokens.

What explains the gain. Paired differences at fixed N=16N=16, with 95% tt intervals over fine-tuning seeds (given each arm's continued-pretraining checkpoint):

G-Loop minusGLUE-6 [95% CI]SQuAD 2.0 F1 [95% CI]
DeBERTa-v3-base+1.25 [1.06, 1.44]+0.90 [-0.10, 1.90]
Widened FFN (same parameter count)+0.93 [0.74, 1.12]+0.70 [0.50, 0.90]
Linear mixing (same parameter count and compute)+0.63 [0.41, 0.85]+0.50 [0.10, 0.90]

The last row is the key comparison, and GLUE-6 with five paired seeds is our main evidence. G-Loop and linear mixing have the same parameter count and the same fixed-depth inference computation, so neither explains the difference, and G-Loop is ahead in all five seed pairs. The widened FFN row shows that the same parameter count spent without recurrence does not close the gap either. SQuAD 2.0 points the same way in all three seed pairs. With three pairs its intervals are wide, so we treat it as supporting evidence.

The gap grows with refinement depth. We evaluated the same N=16N=16 checkpoints at smaller depths, without retraining and with early exit off (GLUE-6; seed SDs are 0.19 to 0.29):

Steps NN124816
GFLOP25.735.454.893.6171.2
G-Loop87.1087.9088.6088.9088.98
Linear mixing87.0587.5587.9888.2688.35
Gap+0.05+0.35+0.62+0.64+0.63

At the first step the state equals the anchor, so the gated blend does not depend on the gate, and the gap there is only +0.05. It grows as the loop iterates and largely saturates by N=4N=4. We read this as supporting evidence for a benefit during recurrent refinement. The table also gives the accuracy-compute curve: N=4N=4 keeps about 70% of the gain over DeBERTa at 2.5x its FLOPs, and N=8N=8 keeps about 94% at 4.2x.

What it costs. Against standard DeBERTa, G-Loop remains more expensive. At fixed depth it needs 7.7x the FLOPs and 12.2x the batch-1 latency. Early exit roughly halves this, to 3.9x and 6.6x, for a 0.03-point change in GLUE-6. Peak memory is 0.54 against 0.42 GiB. Against linear mixing under the same early-exit rule, G-Loop has higher mean accuracy and lower inference cost (7.4 against 9.0 steps per layer).

Scope. All recurrent language arms share the adapted operator of App. G, which differs from the puzzle loop: keys come from the anchor, there is an anchor residual with post-LayerNorm, the damping is fixed (α=0.8\alpha = 0.8), and training uses a one-step phantom gradient. The comparisons therefore test adaptive value construction within this language adaptation. All arms get the same number of updates, so training exposure is matched but training compute is not. Section 2.1 now states these differences explicitly, and Appendix J.6 has these tables.

Dear Reviewers, thank you for your time and effort in reviewing this submission. As the discussion period is ongoing, I would appreciate it if you could join the discussion and engage with the authors' rebuttal at your earliest convenience.

AuthorsOpenReview ↗

Summary of the discussion

We thank the reviewers for their constructive feedback and the AC for facilitating the discussion. All three reviewers acknowledged that our responses addressed their questions or most of their concerns, and QEhK raised their rating. All changes are incorporated into the revised paper (marked in blue).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.