acceptodds
Under review as a conference paper at ICLR 2027

Agreement rather than normalisation keeps depth reusable in looped language models

Abstract

Looped language models apply one block of layers repeatedly, and the design pays off only if depth is reusable: loop steps beyond the trained budget must remain usable at inference. We ask whether that rests on normalising the state between loop steps, as the released Ouro models do, or on agreement between the loop a model is trained with and the loop it is run with. On released weights this loop-boundary normalisation can be removed without adding parameters and with the readout path bit-identical, and the operation is an identity at one loop step, which serves as an implementation check. Removing it takes Ouro-1.4B and Ouro-2.6B to zero accuracy by the trained budget on a balanced binary task (chance 0.500). State-norm growth, the failure that prior accounts predict, is not the cause: the norm stays within a factor of 2.021 while the head commits to a single wrong token. Yet in 19M-parameter models trained from scratch on two synthetic tasks, 16 seeds per configuration, training without the normalisation is never worse than training with it and extrapolates better: the two are closest at the trained depth and separate on both sides of it (Cliff's delta up to 0.867 beyond the budget). Both results follow if agreement, and not the normalisation itself, is what keeps depth reusable. The damage sits in the readout together with the last layer of the block: in the models trained from scratch, refitting the head over the frozen modified loop recovers a third of the lost accuracy and plateaus, while refitting the last block layer with it recovers all of it. Finally, comparing looped with stacked depth at an equal inference loop count is confounded by unequal overshoot of the trained budget; at matched overshoot the advantage of looping vanishes on one task and shrinks on the other.

Primary area:
foundation or frontier models, including LLMs
Keywords:
looped language models, recurrent depth, loop-boundary normalisation, test-time compute, depth extrapolation, readout collapse

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_EdCqOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how boundary normalization affects looped language models when more loops are used at test time. The authors modify pretrained models and compare small models trained from scratch with and without boundary normalization. Removing it from pretrained models hurts performance, while training without it leads to less degradation with extra loops. The authors argue that keeping the loop computation consistent between training and inference is important for maintaining performance.

Critical strengths

1.The paper studies why models lose accuracy when more loops are used at test time and how boundary normalization affects this behavior.

2.The authors combine changes to pretrained models with controlled experiments on models trained from scratch. Multiple random seeds and implementation checks make the results more reliable.

Critical weaknesses

1.The paper mainly shows that training without boundary normalization makes accuracy drop less when more loops are used. However, it does not show how extra loops can improve performance, so the contribution remains limited.

2.In the main experiments, more loops require more computation but do not improve accuracy. The paper does not clearly explain why this extra cost is needed.

3.The authors provide several analysis experiments, but they do not directly test their explanation of why boundary normalization affects performance with more loops.

4.A framework showing the loop structure, normalization placement, and experimental steps is missing, which makes the paper harder to follow.

Questions for authors

Can extra loops improve accuracy on harder tasks? In what practical situations would more loops be useful?

Reviewer_og97OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies boundary normalization in looped transformers, finding that removing it from pretrained models damages performance, while small models trained without it better tolerate additional inference iterations.

Critical strengths

The controlled normalization uses 16 training seeds per configuration. The paper reports negative results and important limitations.

Critical weaknesses
  1. The presentation is poor. The writing is extremely hard to follow and seems AI generated.
  2. The central argument is difficult to assess because key concepts are insufficiently specified. "Agreement" is explained informally, but the paper moves between matching training/inference topology and maintaining compatibility between hidden states and the readout. These are different conditions. Explicit recurrence equations, training objectives, and operational definitons would substantially improve clarity.
  3. The main causal conclusions are not established. Damaging a pretrained model by changing its recurrent computation does not establish that architectural agreement is necessary for reusable depth.
  4. The evidence is limited. The controlled experiments use small models on two synthetic tasks and do not show that extra loops help solve harder problems.
Questions for authors
  • What is "agreement"?
Reviewer_geTAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Looped language models reuse one block of layers several times, and this paper asks why some can usefully run more loop steps than they were trained with. The authors test whether that depends on renormalising the state between loop steps, as the Ouro models do. Removing this normalisation from released Ouro models makes them collapse within a few loop steps, yet the state's size barely changes; instead, the output head stops producing valid answers. To separate cause from effect, they train small models from scratch with and without the normalisation and find that the models trained without it do as well or better, especially when run with more or fewer loop steps than in training. They conclude that what matters is the model being run with the same loop setup it was trained with, and that the normalisation itself actually limits extrapolation. They also show that comparing looped and stacked models at the same loop count is misleading: once each is compared at the same multiple of its trained depth, the apparent advantage of looping disappears on one task and shrinks on the other.

Critical strengths

None

Critical weaknesses
  1. This paper looks heavily generated by AI. The writing was difficult to follow and has no clear structure. It also features many "X, not Y" sentence structures, and frequently use terms such as "gate", "axis", "arm" (words that Claude use heavily). The paper is full of unclear sentences.
  2. The contributions of the paper is very weak. The results section seems like an ablation study. The observed results are very much expected. Removing a component a model was trained with and seeing it break is not very surprising. The paper doesn't seem to contribute any new knowledge or insight.
Questions for authors

How does the findings of this paper lead to better training strategies for looped models?

Reviewer_Z3WkOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies the role of normalization in looped transformers, where the same block of layers is applied repeatedly. It compares different placements of normalization, mainly applying it within each loop iteration versus applying it once after the loop finishes. It also looks at mismatches between training and inference: models trained with normalization but run without it, and models trained without normalization but given it at inference time. As part of this, the authors pretrain looped transformers from scratch with no normalization inside the looped layers, and suggest that normalization may not be necessary there.

I find the direction interesting and appreciates the effort of training normalization-free looped models from scratch, but is unconvinced by the contributions. The main results largely confirm what one would expect: changing the normalization scheme between training and inference disrupts information flow and degrades performance. I therefore see little novel insight and limited practical value. They also consider it premature to conclude that normalization is unnecessary, since the experiments use small models, rely only on synthetic data, and cover a narrow set of tasks that are not clearly described, leaving it unclear whether the findings would generalize to larger models or realistic downstream tasks.

Presentation is another major concern. Although the underlying problem is not complex, the paper is hard to follow, and it lacks a clear mathematical formulation of the looped transformer architecture, which matters because looped and recurrent transformers are distinct concepts that the current description could blur. Overall, the reviewer judges that the paper is not ready for publication at ICLR in its current form.

Critical strengths
  • The research direction is interesting.
  • The authors made the effort to pretrain looped transformers from scratch without normalization inside the looped layers.
Critical weaknesses
  • Unsurprising, intuitive findings. It is expected that changing normalization between training and inference would disrupt information flow and hurt performance. The paper offers little insight beyond this.
  • The conclusions have little practical use.
  • No clear mathematical formulation. The looped transformer architecture is not formally defined. This leaves room to confuse it with recurrent transformers, which are a different concept.
  • All experiments use synthetic data, and the tasks are not explained well enough to judge how significant or general the results are.
  • The models are small and the task set is narrow, so it is unclear whether the conclusions hold for larger models or realistic downstream tasks.
  • Premature conclusion as it is too early to claim that normalization is unnecessary based on these experiments.
  • Poor clarity. Although the problem itself is not complex, the paper is hard to follow, which obscures the methodology and contributions.
Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.