- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work is focused on weight tying in looped transformers, where a block of layers is reused multiple times during a forward pass to iteratively solve a given task. The authors investigate its effect on directed permutation traversal task, where the model's goal is to predict a node reached after H hops. Prior work (Sesodia et al., 2026) showed that on this task, looped models expose intermediate hops along a "staircase", where one hop corresponds roughly to one iteration. The paper's main contribution is the claim that gradient pooling (the summation of gradients per each iteration into the shared block's parameters) is the cause of this behavior. The authors use training interventions to validate this argument (e.g. interpolation between pooled and per-iteration updates or replacement of the summed gradient with alternatives). Linear probes are used to observe where intermediate states appear, and causal interventions are used to test whether the model uses them.
Critical strengths
- The task is well chosen: its intermediate states are exactly defined but never supervised, which lets the authors track the model's progress layer by layer with probes and causal interventions, both with proper controls.
- Gradient routing (§4.2, Fig. 5) keeps the tied forward computation and changes only how the gradient is aggregated, which is a useful attempt to separate "the block is applied repeatedly" from "its gradient is summed", two explanations that are normally inseparable. The partial-pooling results are also consistent: the more pooled model solves the task first in 24 of 30 pairs (§B.2).
- The paper first establishes that neither the task (untied models solve it) nor the architecture (the no-curriculum looped run) forces the staircase (§4). The cyclic vs. contiguous result (§4.4, Fig. 7) also gives a plausible explanation for an observation from prior work (Takase & Kiyono, 2023; Mixture-of-Recursions).
- Limitations are reported honestly, and the work is reproducible. Negative or unclear results (the register machine and N=32 in §C, the optimizer confounds in §A.7) are reported, and the setup, seeds and code are fully specified (§A, §A.8).
Critical weaknesses
- The experiments do not isolate the effect of the sum (§4.1, §4.2). When a block is reused, the sum of its gradients is simply its correct gradient, so the two always go together. Any α<1 lets the copies drift apart, so the block is no longer exactly reused either. At α=0 the model is weaker than the paper's own untied model (Table B.5 vs. A.4). Gradient routing replaces the correct gradient with a wrong one (last iteration only) or a noisy one (random iteration), so their failure does not single out the sum; the random variant fails even H=1 in 2 of the 3 seeds. Also, only α=1 forms the full staircase in Fig. 1(a), although α=0.75 also solves the task (Table B.5), which suggests that exact tying matters, not only the amount of pooling.
- A fully pooled model trained without the curriculum solves the task without the staircase (one run, Fig. B.4), and the curriculum's role is not otherwise tested. So pooling on its own may not be enough.
- The evidence for a reused one-hop step (§4.3) does not separate pooling from reuse either. The within-block result fits any tied model doing one hop per iteration, and the burst is compared with an untied model that starts as eight copies of the looped block and is much weaker than the paper's own untied model (Table B.6 vs. A.4). In one looped model every hop is readable, yet swapping hidden states between prompts never makes it follow the swapped-in state (matched-rate seed 44, Tables B.1 and B.4).
- One small synthetic task, small models and mostly three seeds, yet the title and conclusion present a general mechanism and extend it to large looped models (§6).
Questions for authors
- Does the staircase need exactly identical weights, or is the summed update enough? For example, give each iteration its own copy of the block, start the copies from different initialisations, and update all of them with the summed gradient (W1). The learned part is then still shared, but the weights used at each iteration differ.
- Does the random-iteration result hold with less noise? For example, by summing a random subset of iterations, or averaging several random draws per step (W1). If summing still clearly beats these, it would substantially strengthen the main claim.
- The one fully pooled run without the curriculum shows no staircase (Fig. B.4). Does this hold over more seeds, and does more pooling still help without the curriculum (W2)? If the staircase needs the curriculum, please state this in the main claims.
4 comments
AC Request
Dear Reviewer,
Thank you very much for your time and feedback. I would greatly appreciate if you could confirm if the rebuttal addresses your concerns and let them know about any remaining concerns.
Best regards, AC
Thank you for the response and for running the experiments from Q1 and Q2. The rebuttal addresses most of my concerns about the claims.
The new experiments went against the original claim, and the revision narrowed it. The paper now says what the results show: the staircase needs tied or nearly tied weights, and the summed update is not enough on its own (W1, Q1). The claims are now also limited to training with the curriculum (W2, Q3), the probes paragraph says that readable states are not always used (W3), and the extension to large models is removed from the conclusion (W4).
Two smaller points remain. First, the abstract still says that "the underlying mechanism is gradient pooling", and Fig. 1 is still titled "Gradient pooling elicits iterative solutions". Both read as if pooling were enough, with no mention that it needs tied or nearly tied weights, or that the claims hold only for training with the curriculum.
Second, on Q2, an update built from one or two iterations is noisier than one built from four, while the expected update is the same in all cases. So the result may only show that a very noisy update fails, which is true of training in general and says little about pooling.
What keeps me from raising my score is the contribution (W4). Sesodia et al. (2026) already showed the staircase on this task, and this paper builds on it. What the paper mainly adds is when the staircase forms: with tied or nearly tied weights, and with updates built from several iterations. That is useful to know, but it is shown on one small task with small models, and it is not yet known whether the finding carries over to other tasks. The paper is open about this (§5), and it is clear and carefully done, but to me the contribution is modest and specific to this one setup.
I keep my rating at 2.
Author Response to Reviewer vnhf
Thank you for your reply and continued engagement. We respond to your remaining points below.
First, the abstract still says that "the underlying mechanism is gradient pooling", and Fig. 1 is still titled "Gradient pooling elicits iterative solutions". Both read as if pooling were enough, with no mention that it needs tied or nearly tied weights, or that the claims hold only for training with the curriculum.
Our response: The title and the abstract's second sentence frame our pooling claims within weight-tied looped transformers. Since our partial-pooling and distinct-initialization experiments also applied pooling to untied weights to probe the mechanism, we have, for the avoidance of doubt, revised the abstract to state that tied or nearly tied weights are needed, and the title of Fig. 1 to state weight tying.
The revised abstract also now states the curriculum condition, which is echoed in Contribution 1, the training details of §3, and the Scope paragraph of §5.
Second, on Q2, an update built from one or two iterations is noisier than one built from four, while the expected update is the same in all cases. So the result may only show that a very noisy update fails, which is true of training in general and says little about pooling.
Our response: We agree that learning may fail if the updates are too noisy. For a tied block, though, pooling is the mechanism that reduces this noise, which is consistent with our hypothesis in §4.1. The result therefore shows that weight-tied models need each update to combine several iterations.
What keeps me from raising my score is the contribution (W4). [...] it is shown on one small task with small models, and it is not yet known whether the finding carries over to other tasks. [...] to me the contribution is modest and specific to this one setup.
Our response: Looped transformers are widely motivated by an assumed bias towards iterative computation, and we believe that providing a mechanism for how training produces it is a valuable contribution. While our evidence comes from one task with small models, we chose the task deliberately as the minimal sequential task with uniquely defined intermediate states, which is what allowed us to follow the mechanism inside the model. The paper also offers an explanation for the previously unexplained advantage of cyclic over sequential reuse (Takase & Kiyono, 2023; Bae et al., 2025; §4.4).
We hope these revisions and clarifications address your remaining concerns, and would be grateful if you would consider them in your final rating.
Author Response to Reviewer vnhf
Thank you for your feedback! We appreciate that you recognized the merits of the task design. We ran both experiments you suggested in Q1 and Q2, and narrowed our central claim accordingly.
Our response: We agree that the sum cannot be separated from reuse, and in the revised pdf we have removed language that suggested otherwise. We also agree that α<1 lets the copies drift apart, which §4.1 now states as a caveat, and that tying matters, which the experiment you proposed in Q1 confirms, although nearly tied weights suffice. The revised claim is therefore that summing gradients from several iterations builds the operator only on tied or nearly tied weights, and we address the noise in gradient routing in Q2.
Our response: We agree, and the revised pdf states that our claims concern training with the curriculum. Two further looped seeds trained without the curriculum do not solve the task, and neither do untied models reliably (Appendix B.1). The solving run serves only to show that the looped architecture does not force the staircase.
Our response: We agree that the within-block analysis and the burst do not separate the sum from reuse. Their role is to characterize the operator that pooling builds. For the looped model you mention, our discussion of probes now states that readable states are not always used.
Our response: We agree, and in the revised pdf we removed the extrapolation to large models from the conclusion. The Scope paragraph (§5) states the single task, small models and seed counts, and now also that our claims concern training with the curriculum.
Our response: Thank you for suggesting this experiment, which we ran (§4.1). With the summed update, copies that start identical or close together form the looped staircase, while independent copies do not solve the eight-hop task. The staircase therefore needs nearly identical weights, though not exactly identical ones, and the summed update alone is not enough.
Our response: We ran random subsets of one, two and four of the eight iterations, rescaled so that each has the full sum as its expected update (§4.2, updated Fig. 5). Summing four solves the task like the full sum, while one or two do not. A correct expected update is therefore not enough, and so we revised the title of §4.2 to "Pooling several iterations per update builds the operator".
Our response: Without the curriculum, neither looped nor untied models solve the task reliably across seeds (Appendix B.1), so we could not test pooling in that setting. Contribution 1 and the Scope paragraph (§5) now state that our claims concern curriculum training.