acceptodds
Under review as a conference paper at ICLR 2027

How Training Loop Counts Shape Computation in Looped Transformers

Abstract

A looped transformer applies the same block repeatedly before producing an answer. We ask how the number of loops used during training changes the computation learned by that block. We study sequence tasks with a known correct output at each position. Reading these outputs after each loop reveals a computation frontier that advances from the start of the sequence. Its average advance per loop is frontier speed. For cumulative products of permutations of four objects, models trained with fewer loops show faster measurable frontiers. This relation holds when models share initial weights and training examples and receive the same number of optimizer updates. It also appears when accuracy determines advancement through a length curriculum. We then continue training two copies of each source model. Reducing the loop count increases frontier speed relative to keeping it fixed. Replacing intermediate states disrupts later outputs. Additional loops can extend correct computation to the last position and then reduce the fraction of fully correct answers. These findings show how training shapes progress inside a shared block and how that progress determines when its output is useful.

Primary area:
interpretability and explainable AI
Keywords:
looped transformers, weight sharing, mechanistic interpretability, algorithmic reasoning, training compute, recurrent computation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ceiMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers the speed at which a looped transformer correctly predicts a token. For example, it looks at At(x)A^t(x) where tt is the transformer operator, and sees how many tokens it correctly predicts. For their experiments, they used a "running parity" example alongside some (mathematical) groups to have analytical answers for the sequences they model. What they found is that training speed does have a dependence on the loop count, while there seems to be a sweet point (Figure 2) where loop count/frontier speed is maximized alongside "more loops" giving the correct answer.

Critical strengths

The question of how much each loop contributes to the final answer is interesting, and relevant: while looped transformer saves on weights, early stopping of loops will trivially save on inference costs. If a small amount of loops are enough to get 90% of the solution, then that's a huge win. Furthermore, the experimental design of using the running parity seems like a great idea: allows one to have infinite datasets generation, and is simple enough to expect 100% accuracy.

Critical weaknesses
  1. There's too much text devoted to running additional experiments, and definitely not enough on the analysis of why. As discussed in the paper, the problems analyzed are generally smaller and simpler than LLM, thus analytical insights should be the goal IMO so that we can transfer it to bigger problems. The additional experiments, like section 4 and 5, seems quite obvious and doesn't provide new insights: it's known that changing intermediate tokens in causal attention will change downstream results.

  2. Writing: the subsections are not labeled (e.g. "Changing the training loop count." is not bold). Figure 6 caption can use better formatting with the \ge, \le and spacing etc. "primary cohort" "confirmation cohort" vs "training set" "validation/testing set". The data for the groups problem isn't written down in the paper i think. Overall, needs a large degree of polishing.

Questions for authors
  1. I'm confused about the the value of mm: it's supposed to be "fixed loop multiplier", which indicates to me it should be an integer, but it's clear from later reading that mm is a fraction for some experiments. Can you explain what this mm is concretely? TT I understand: using Figure 1 notation, we just loop F1,F2,...FTF^1, F^2, ... F^T, but why is mm (and correspondingly dd) the correct indicators to look at in the figures.

  2. Figure 2 looks like the training is not complete? For d=1, the frontier never reaches 32 suggesting that learning is incomplete. I do wonder if it's because you're using such a large transformer for a such a small problem.

  3. Figure 3: It took me awhile to figure out that the second row correspond to different seeds and the results: it really does feel like more seeds need to be run or results cleaned up. At the same time, why is "full by 4n" and "invalid by 4n" no the exact inverse of each other? Similar to question 1, why 4n? Concrete number and examples here would be appreciated.

Reviewer_aZ2NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  1. This paper investigates how the number of loops used during training changes the computation that a looped transformer learns.
  2. It asks whether training with fewer loops teaches each loop to do more work.
  3. To test this, the authors use sequence tasks where the correct output at every position is known. This lets them check, after each loop, how far along the sequence the model’s answers are already correct
  4. Models trained with fewer loops make faster progress per loop. This holds even when all models start from the same weights, see the same training data, and get the same number of training updates.
  5. This speed can also change after training starts. If you take a trained model and continue training it with fewer loops, it becomes faster.
  6. Running extra loops at test time can first complete the answer and then make it worse. So choosing when to read the output matters.
Critical strengths
  1. The paper’s main strength is its clear and well-controlled finding that the number of training loops shapes how a looped transformer computes. They find interesting behaviour when changing this number of training loops (as I mentioned in the paper summary).
Critical weaknesses
  1. The paper is difficult to read and would benefit from substantial improvements in writing and presentation. In its current form, it does not meet the standards of clarity and presentation expected of papers published at ICLR. The writing makes it challenging to understand the paper’s main contributions.
Questions for authors

None

Reviewer_GwEfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how training loop counts affect the progress of looped transformers on sequence tasks. Progress is measured by a frontier of consecutive positions exceeding an accuracy threshold. On cumulative S4 products, fewer training loops produce faster measurable frontiers. Paired continuation experiments test whether this behavior changes when trained models receive a different loop budget. The paper also examines intermediate-state interventions and accuracy under additional inference loops.

Critical strengths

The main experiments have good controls. Matching initialization, training examples, and optimizer updates makes the effect of loop count easier to interpret. The same direction under an adaptive curriculum adds support.

The stay/switch experiment is the strongest part. Both models continue training from identical weights, so the comparison separates changing the loop count from training longer. Tightening the budget increases speed in all four pairs, with median speed changing from 1.000 to 1.895.

The paper distinguishes frontier coverage from exact-match accuracy and first completion from subsequent correctness. The loss of complete answers despite high per-token accuracy, and the alternating even/odd readout behavior, are useful observations. Failed runs and undefined measurements are generally reported clearly.

Critical weaknesses

The contribution needs sharper distinction from prior work. Yang et al. (2024) already observed that smaller training loop budgets can accelerate convergence, with consequences for stability. The contribution here is the frontier measurement and controlled sequence-task evidence, particularly the paired continuations. The general observation that training loop counts affect computational pace is less novel.

The speed fits use a selected subset of models. Only 9/16 qualify in each fixed-update cohort. Eight full-window evaluations fail numerically, seven at demand 0.5 and one at demand 1. The confirmation cohort has no qualified frontier at its most generous loop budget. Reporting these failures is good, but the fitted relationship remains conditional on qualification. Threshold sensitivity and uncertainty estimates preserving pairing by training seed would strengthen the result.

The continuation experiment is small. Tightening has four defined pairs; relaxation has only two. The reported two-sided sign-flip probabilities are .125 and .5. This is promising evidence that speed can change after training, but limited evidence for reliable adjustment in both directions. Effective pair counts should appear alongside the main claim.

The mechanistic analysis is less conclusive. Patching one checkpoint per task shows that intermediate states affect later outputs but does not identify what those states compute. Donor states may disrupt the recipient through general incompatibility. The longer-input correlations are exploratory, especially with only seven complete A5 cases and score directions assessed on the same models.

Questions for authors
  1. Does the speed relationship survive changes to the accuracy and coverage thresholds, with uncertainty estimated over paired seeds?

  2. Can a small higher-precision evaluation distinguish numerical failure during additional loops from failure to learn under longer-loop conditions?

  3. Can the continuation results show speed, completion time, and exact-match accuracy together, including the undefined reverse-direction speeds?

  4. Do donors with the same relevant prefix product cause less damage than donors with a different product? This would help establish what the intermediate states carry.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.