acceptodds
Under review as a conference paper at ICLR 2027

When Do Two Hops Suffice? Learning Reusable Composition in Looped Transformers

Abstract

Learning individual operations does not guarantee reliable composition. We study a synthetic task in which each operation maps one entity to another. Training uses all single-step examples and selected two-step answers, together with state targets derived from the same shallow labels, random input prefixes, and a self-supervised loss encouraging waiting-state preservation on constructed states. With one loop per operation, the resulting full-causal looped Transformer achieves 100% accuracy at tested depths of 256, 512, and 1024 and 99% at depth 2048. State interventions show that intermediate states can decode correctly yet fail as inputs to the next operation. These results show how learning to preserve usable states supports deep composition from shallow answer supervision.

Primary area:
interpretability and explainable AI
Keywords:
loop transformer, compositional generalization, interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_v9FxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates whether looped Transformers can generalize to long compositions using only shallow supervision (extrapolation).

The task setting includes 500 entities and 20 random permutation relations. Training includes all one-hop facts and selected two-hop compositions, while testing extends to 2048 hops.

The model is trained using shallow answer supervision, intermediate-state anchors, random prefixes, and a constructed-state consistency loss that encourages preservation of waiting states.

The proposed method achieves 100% accuracy through depth 1024 and 99% at depth 2048, while standard cross-entropy training fails beyond shallow compositions.

Critical strengths

I think this paper focus on a timely problem: extrapolation of looped transformer, and is a first step towards the understanding of looped tf.

Critical weaknesses

The paper is interesting. That said, there exists some weaknesses to me.

  1. The error-accumulation explanation is not particularly novel. I think this intuition comes already from some early works in skill composition like [1] (though this paper proposes some mechanism other than the normal error accumulation). And in this paper [2], you can even understand where this imperfection comes. (though [2] isn't a looped tf paper, rigorously speaking.)

[1] h1: Bootstrapping Models to Reason over Longer Horizons via Reinforcement Learning

[2] Transformers Provably Learn Algorithmic Solutions for Graph Connectivity, But Only with the Right Data

  1. Intermediate-state restoration is an expected solution to accumulated errors. By projecting intermediate states back to canonical entity embeddings and restoring waiting relations, the intervention kind of resets accumulated representation errors after each step (therefore to me this is something trivial?).

I kind of expecting the author to justify what are the new messages the authors are trying to convey.

Best

Questions for authors

See the weaknesses above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer v9Fx (1/2)

We thank the reviewer for the comments, and for recognising that the paper is interesting, addresses a timely problem (the extrapolation of looped Transformers) and is a first step towards understanding them. We answer each weakness and the question in turn. "New" marks results obtained for this response; all of them use answer labels of depth ≤ 2 only.

W1. The error-accumulation explanation is not particularly novel ([1], [2])

We agree that the general intuition, that errors build up over a long horizon, is not new, and we do not claim it. We will cite and discuss [1] and [2]. What the paper adds is specific to looped Transformers: it locates, by intervening on the internal states, which state carries the error, and it removes that error with a training objective that needs no answer labels.

  • [1] studies chain-of-thought in pretrained LLMs. It argues that independent step errors (p^h) do not account for long-horizon failure and hypothesises horizon-dependent skills such as state tracking, which it trains by reinforcement learning on answers to progressively longer problems. It does not locate the skill in the model's states.
  • [2] studies fixed-depth Transformers without weight sharing on graph connectivity: an L-layer model covers 3^L hops, and training graphs beyond that capacity lead it to a degree heuristic; the remedy is to restrict the training data. Our failing models do not lack capacity (shared loop, one loop per hop), and their single step is already correct, as shown next.

Every single step is right, yet depth fails (new). All numbers below are for seed 7, the seed and setting of Table 3. Trained with anchors only or with wait-CS only, a state written from clean inputs is used correctly by the next operation in 4,800 of 4,800 and 4,797 of 4,800 events (the test of Table 3, where CE only gives 56.56%). If steps failed independently at these rates, accuracy at depth 128 would be at least 92% and 81% (95% bounds). Measured: 0.20% and 0.21%.

Which state carries the error (new). Correct chains of 300 at depth 128, with analysis-only edits after every loop. Reset: relations whose turn has not come are set back to their input embeddings. Projection: the state just computed is replaced by its nearest entity embedding (the model's own choice, no gold values).

TrainingNext-step reuse (test of Table 3)NaturalReset onlyProjection onlyBoth
CE only (Table 3)56.56%212298
Wait-CS only (no anchors)99.94%100235
Anchors only (no CS)100.00%33000300
Ours, at 120k updates99.81%254257298298
Ours, at 200k updates (Table 3)100.00%300300300300

With anchors only, the error sits entirely in relations that have not been used yet: restoring them recovers every chain, while cleaning every computed value recovers none. At 120k updates of our recipe, the remaining error is instead in freshly computed values. These are two separable sources. The first, the drift of operands still waiting for their turn, is what wait-CS removes without labels.

So the state-management skill that [1] conjectures beyond step accuracy is here identified, measured and trained from two-hop labels. Our baselines trained on deeper answers instead stop within twice their label depth (Table 1: answers to depth 32 give 0.25% at depth 64; rewards to depth 8 give at most 0.15% at depth 32).

AuthorsOpenReview ↗

Response to Reviewer v9Fx (2/2)

W2. Intermediate-state restoration is an expected solution to accumulated errors

As a test-time intervention, we agree, and we do not propose it as a solution. In the paper it is a diagnostic (Fig. 4, App. F.6.1); our model is evaluated with frozen weights and continuous states, without any projection or reset (App. B.2). We will say this more plainly. What we consider non-trivial is the following.

  1. Restoration discriminates between failures, and its two parts do different things (table in W1; the paper mostly reports the two combined, App. F.8.2). With CE only, neither part alone helps and both together recover 298 of 300. With anchors only, the reset alone suffices; projecting computed states onto entity embeddings, the part closest to "resetting the accumulated error" of the computation, recovers nothing. With wait-CS only, even both together recover only 235. For our recipe at 120k, the projection alone suffices.
  2. Training towards the restored states is not enough. Anchors are the training-time counterpart of restoration: on the labelled shallow rows they pull each computed state to its entity embedding and each waiting relation to its input embedding. Alone they give 0.20% at depth 128 (W1; 0.09% in Table 1). Consistency on completed values (root + hold) also fails (0.24%), whereas consistency on waiting positions alone gives 100% (Table 2). Which state has to be preserved, and that this can be trained on constructed states without labels, is the result.
  3. After training, nothing has to be restored. Restoration must be told which position has just executed, and it discretises that state. The trained model needs neither: natural execution gives 300 of 300, and 0 errors in 120,000 test chains at depths 2–128. Over training, restoration already gives 283 of 300 depth-128 chains at 5k updates, while natural execution reaches 299 only at 130k (Fig. 4).

W3. What are the new messages?

Two labelled hops suffice for deep composition in a standard looped Transformer (full causal attention, continuous states, nothing added at inference), if training also asks, without labels, that inputs not yet used survive a loop. Alternatives stop near their label depth: answers to depth 32 fail at 64 and rewards to depth 8 fail at 32 (Table 1); chain-of-thought with every intermediate entity supervised on the same data gives 0% from depth 3 (new; response to Reviewer tGPj, W1).

What decides depth is a write–removal balance. Compositional generalisation is related to two measured quantities: the output error of each newly written state, and the loop's ability to remove noise from the states it carries. The state at depth d does hold what earlier loops left in it, but that residue is the outcome of error written against error removed. It is not independent per-step compounding (accuracy p^h): with every single step right, depth 128 is at 0.2% (W1). New: our recipe, seed 7, rerun with a checkpoint every 10k updates.

Updates20k60k100k110k120k130k140k150k200k
Output error of a new state4.853.332.362.051.601.361.130.780.24
Noise left after 16 loops / noise injected18142.538.50.562.833.100.480.290.0017
Depth 128, correct of 300023212221253283299300300

Output error: distance of a freshly written state from its entity embedding, in units of half the distance to the nearest other entity (largest of five positions). Noise: relative size 0.1 on the waiting relations of constructed states, squared error (probe of App. F.8.1). Up to 130k the loop amplifies injected noise at every checkpoint but one; from 140k it removes it and depth 128 is reached, while the output error keeps falling. The larger world of App. F.8.1 shows the balance lost: its one-step error improves, noise is amplified over 16 loops, and depth-128 accuracy falls from 299 to 2 of 300.

Q1. "See the weaknesses above."

Answered in W1–W3. In the revision we will (i) cite [1] and [2] and state which part of the explanation is prior intuition and which is our evidence; (ii) present restoration explicitly as a diagnostic, split into its two parts; (iii) add the tables and results above; (iv) state the two messages of W3 at the end of the Introduction.

We thank the reviewer again for these comments, which helped us state more precisely what is new. We hope the clarifications and the new results address the concerns, and we would be grateful if the reviewer would reconsider the rating in their light.

Reviewer_Rqb1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies compositions of permutation relations over 500 entities. Training provides all atomic facts and selected two-hop answers, together with shallow state anchors, random entity prefixes and constructed-state consistency (CS). CS trains selected positions in synthetic states to preserve their inputs, leaving the executing frontier unconstrained. The strongest variant, wait-CS, applies this loss only to waiting relations.

The resulting full-causal looped Transformer uses continuous states and one externally specified loop per operation. It achieves near-perfect depth-128 accuracy across four optimisation seeds, 100% at depths 256–1024 across three seeds, and a reported 99% at depth 2048. State interventions show that correct intermediate decoding need not imply successful reuse by the next operation. The result is therefore about learning deep execution from shallow answers with an explicitly trained state interface; answer cross-entropy alone does not produce the same behaviour.

Critical strengths
  1. The main ablation shows a substantial benefit from CS. With anchors and prefixes but no CS, depth-128 accuracy is 0.09%; adding wait-CS raises it to approximately 100% (Table 1). The backbone, shallow supervision and 200k-update endpoint are retained. Although the seed cohorts differ, this is strong evidence that CS contributes beyond the shallow state targets.

  2. In Table 3, all 4,800 CE-only written states decode correctly, but only 56.56% support the next operation. Replacing the written state with the correct entity embedding resolves the failures. This is more informative than observing intermediate decodability alone, and the operand interventions provide complementary evidence about the local computation (Appendices F.2–F.4).

  3. The analysis includes useful qualifications and failure cases. The authors distinguish canonical-state interventions from natural rollouts, identify unmatched diagnostic seeds, and examine displacement after removing a direction invisible to LayerNorm. The larger-world failures and deterioration of multistep recovery despite improved one-step consistency are particularly informative (Appendices E and F.8).

Critical weaknesses
  1. The mechanistic contribution needs clearer positioning against DiscoLoop. This uncited work already demonstrates decodable but poorly reusable bridge representations and repairs them through embedding realignment. The submission's writer–reader controls and long-chain preservation analysis add evidence, but the general observation should not receive the same novelty credit as the CS objective. Please discuss this overlap explicitly. A DiscoLoop-style baseline would also be relevant, although its published results on different tasks are not a matched comparison.

  2. The published-method comparisons are difficult to interpret. These rows reach only about 77–90% on the mixed depth-two metric before deep extrapolation is tested (Table 1). Since this metric includes held-out adjacencies, it does not establish that the baselines underfit their training data. Appendix D.3 also uses different optimisation settings and forces one loop per operation, which need not suit every method's learned computation. Please report atomic, training-pair and held-out-pair accuracies separately, and check reasonable recurrence budgets for the closest baselines. Equal update counts do not establish equal computation. These issues weaken the comparative claims, while leaving the internal CS ablation informative.

  3. Robustness of unchanged wait-CS is not yet established beyond the main world. The main seed variation concerns optimisation, with the relational world and supervision graph fixed (Appendix C). Appendix E provides encouraging results on a second world seed, but uses full-CS. In the larger worlds, that recipe falls near chance at depth 128 and needs world-specific denoising or walkback changes. The authors disclose this limitation, but it remains unclear how readily the main recipe can be reused. Running unchanged wait-CS on another world or supervision graph would be a focused way to strengthen the evidence.

  4. Table 3 compares CE-only with the complete recipe, so it cannot determine whether the improved write–read interface comes mainly from anchors, CS or their interaction. Adding the existing anchor-only model to that diagnostic would help. CS-without-anchors and prefix-free ablations would further clarify which ingredients are necessary. Their absence does not erase the incremental benefit shown in Table 1, but it limits the account of how the ingredients work together.

Questions for authors
  1. Please give depth-2048 error counts by seed and query distribution. Appendix C.4 states that seed variance is not reported there. Since the same questions are used across seeds, optimisation-seed variation and query-sampling uncertainty should be distinguished.

  2. Please specify the sampling distribution of the completed-prefix length and relation sequences in the standard CS branch (Appendix B.3), and report measured training and long-chain inference costs. The cubic dense-attention cost is acknowledged in Section 7, but runtime would make the practical trade-off clearer.

  3. Can any diagnostic that does not use deep answer labels distinguish successful from unsuccessful checkpoints? This would be useful given the larger-world deterioration in Appendix F.8, though I would not require a general checkpoint-selection solution for acceptance.

The API comparison is best treated as context rather than evidence of a matched compositional advantage. The knowledge-access and inference protocols differ, and Claude's failures in this cohort are refusals (Appendix C.5). The paper already discloses these facts; they should remain prominent when interpreting Figure 2.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Rqb1 (1/4)

We thank the reviewer for the careful reading and for recognising the strengths of the paper: that the main ablation gives strong evidence that CS contributes beyond the shallow state targets (0.09% → about 100% at depth 128 with the same backbone, shallow labels and endpoint); that the decoding-versus-reuse test of Table 3, with the operand interventions, is more informative than intermediate decodability alone; and that the qualifications and failure cases, in particular the larger-world failures and the deterioration of multistep recovery despite improved one-step consistency, are informative. Unless stated otherwise, every new run keeps the paper's protocol (answer labels of depth ≤ 2, 200k updates, fixed endpoint, R = d).

W1. Positioning against DiscoLoop; a DiscoLoop-style baseline

We agree and thank the reviewer for the pointer. DiscoLoop (arXiv, July 2026) shows for two-hop questions that the bridge entity is decodable after the first loop while its hidden state is misaligned with the entity's embedding, and that realigning it nearly closes the generalization gap. We will cite and discuss it in Sections 1, 6.1 and 7, credit it with this observation, and claim as ours only the CS objective and the long-chain analysis of which states must be preserved.

Over long chains, realigning the written state is not enough. We applied DiscoLoop's realignment to our checkpoints: after every loop the newly written state h becomes (1 − α)h + α·E(b), b its decoded token (α ∈ {0.1, 0.25, 0.5, 0.75, 1}, E raw or norm-matched; best of the 10 shown). Depth 128, correct of 300:

ModelNaturalRealignmentRealignment + waiting relations reset
CE-only (3 runs)0–20–2298–299
Anchor-only (Table 1)00–2300
wait-CS at 60k / 120k36 / 254259 / 298263 / 298

This agrees with Table 3, a one-step test from canonical states (correct entities before the write, input relations after it): the last column keeps the chain close to such states and, like Table 3's gold control, succeeds. Realignment alone does not repair CE-only loops beyond depth 2 or anchor-only loops beyond depth 4: the waiting relations must also be restored, and preserving them is what wait-CS trains. Only on mid-training wait-CS checkpoints does realignment do most of the repair.

We also ported DiscoLoop itself (discrete embedding channel at every position and loop, α = 1, τ = 1, K = 2, final-answer CE, the same depth ≤ 2 rows and the ports' optimisation settings): atomic facts and training pairs 100.00 (0.00), held-out pairs at depth 2 84.16 (1.38), and at most 0.31% at any depth ≥ 3 under the best fixed budget R = 1…256 (chance 0.2%). DiscoLoop is evaluated at two and three hops, not on long-chain extrapolation, and our port did not reproduce its published two-graph effect, so we report it as a port, not as evidence about DiscoLoop.

W2. The published-method comparisons

We agree with each point and will present these rows as ports under their own settings.

Shallow split (%, mean (variance) over; training pairs = 8,000 of the 80,000 training two-hop questions):

PortAtomic factsTraining pairsHeld-out pairs, depth 2
Recall + progressive loss99.84 (0.05)99.92 (<0.01)77.32 (4.39)
NDR100.00 (0.00)100.00 (0.00)84.91 (0.73)
Input injection100.00 (0.00)100.00 (0.00)85.89 (18.35)
Dynamic recurrence100.00 (0.00)100.00 (0.00)81.43 (24.84)
FPR mixing100.00 (0.00)100.00 (0.00)89.92 (0.33)
Ours100.00 (0.00)100.00 (0.00)100.00 (0.00)

The ports fit their training data. Their 77–90% at depth 2 in Table 1 is the held-out-pair column: through depth 128, Table 1 reports trained rows on the held-out-adjacency test set (one new), which we will state in the caption. The ports thus lose accuracy on new adjacencies before depth matters.

Recurrence budgets. We read every fixed budget R = 1…256 (4,500 chains per depth, pooled over seeds and distributions). Only once does a budget other than R = d help by ≥ 10 points: recall + progressive loss at depth 3 (0.20% → 19.62% at R = 208). From depth 8 on, the best budget gives at most 0.38% for recall, NDR, input injection and dynamic recurrence; FPR mixing is best at R = d. We did not test adaptive halting.

Settings and compute. The port runs use 512 queries per update for 50,000 updates (25.6M queries, as many as our 128 × 200,000) at learning rate 2 × 10⁻⁴. A port run trains in 13–24 min, a wait-CS run in 6.9 h, so equal queries are not equal compute.

AuthorsOpenReview ↗

Response to Reviewer Rqb1 (2/4)

W3. Robustness of unchanged wait-CS beyond the main world

We ran the paper's wait-CS unchanged on another supervision graph and on other worlds (part 1), and one shared variant on the problems where it fails (part 2). Every run uses the same three terms (answer CE at depth ≤ 2, state anchors, CS) with the wait-CS role mask (one marked row uses the full-CS mask); a new problem only specifies which positions wait and where results are written. Accuracy (%) at depth 128, mean (variance) over three seeds unless marked; every listed run is ≥ 99% at every depth 2–128 in every test distribution.

(1) Problems that wait-CS solves as in the paper (constructed depth 32, no noise).

ProblemDepth 128
Main world100.00 (<0.01)
Same facts, another graph of trained relation pairs100.00 (0.00)
New random relations (world seed 43; App. E used full-CS)100.00 (0.00)
2 trained successors per relation instead of 8 (¼ of the two-hop labels)100.00 (<0.01)
4 trained successors per relation99.98 (<0.01)

(2) Problems that need denoising wait-CS. Same loss term and role mask; only the constructed CS inputs change. Gaussian noise of relative scale up to 0.1 is added to the waiting and completed states, the waiting positions must still return to their clean states, and the constructed depth is 128 instead of 32. Answer data, anchors, prefixes and the 200k updates are unchanged. Both changes are in the paper (Section 3.3 and App. E: denoising for R40, constructed depth 128 for E1000); using them together on wait-CS, as one setting for the problems below, is new in this response, and every problem below other than R40 and E1000 was designed after the setting was fixed. They target the failure measured in App. F.8: one-loop preservation of clean states does not ensure recovery from deviations that build up over many loops. On these problems the paper's wait-CS passes on no seed wherever we ran it.

ProblemDepth 128
40 relations (R40, App. E)100.00 (0.00)
1,000 entities (E1000, App. E)100.00 (0.00)
Navigation: 24 × 25 grid, 100 walls, 20 moves; a blocked move stays in place, so the maps are not bijections100.00 (0.00)
Group world: the 120 elements of S5, relations = right multiplication (one seed)100.00
1,000 entities × 40 relations (full-CS form; one seed)99.42
In-context tables of 8 / 12 / 16 rows (16 rows: 400k updates)100.00 (0.00) / 99.85 (0.045) / 99.90 (0.03)

In the in-context version each input carries its own random permutation of E entities as E key–value rows, then a start entity and d identical hop tokens; answers are labelled at one and two hops only, and the hop positions play the role of the relations (1,000 fresh tables per depth). All 8-row tables occur in training, so only the depth is new; about 5% of the 12-row and at most 3.1 × 10⁻⁶ of the 16-row test tables can occur in training, so those are new. The same denoising setting also passes on seeds 1, 7 and 42 of every problem in part (1) where we ran it (main world, other graph, world seed 43, 4 successors).

AuthorsOpenReview ↗

Response to Reviewer Rqb1 (3/4)

W4. Which ingredient improves the write–read interface in Table 3

Both anchors and wait-CS improve next-step reuse when added individually to CE. At the Table 3 setting, reuse rises from 2,715/4,800 with CE alone to 4,800/4,800 with anchors alone and 4,797/4,800 with wait-CS alone. The original CE-only versus full-recipe comparison therefore cannot attribute the improvement to either ingredient individually.

All rows below use seed 7 and the Table 3 setting, except the prefix-free row, which uses the Table 1 setting. Next-step reuse follows the 4,800-event diagnostic in Table 3. The remaining columns give the number of correct final answers on 300 depth-128 chains. Reset restores each still-waiting relation to its input embedding. Projection replaces the newly written state with its nearest entity embedding, chosen from the model state without using the gold entity. These edits are applied between loops during evaluation.

TrainingNext-step reuseNo editsReset onlyReset + projection
CE only (Table 3)56.56%21298
Anchors only100.00%3300300
Wait-CS only99.94%10235
Anchors + wait-CS, no prefixesSee below000
Full recipe (Table 3)100.00%300300300

At the Table 1 setting, we also retrained both single-ingredient variants with matched seeds 1, 7, and 42. Each variant had zero reuse failures in 14,400 events pooled across its three seeds. The three original anchor-only models from Table 1 likewise had zero failures in 14,400 events.

Accurate one-step reuse did not ensure reliable long-chain execution. Only the full recipe solved all 300 depth-128 chains without edits. For anchors alone, resetting waiting relations raised the count from 3 to 300. For wait-CS alone, reset plus projection raised it from 1 to 235. For the prefix-free model, even supplying the gold entity embedding gave only 28.8% next-step accuracy, so this diagnostic cannot isolate a defect in the written state. We will add these results to Appendix.

Q1. Long-chain errors for the four seeds

We evaluated training with four seeds. All four models used the same long-chain inference setting and were evaluated on the same chains at each depth. We will document the inference setting in Sections 2.1 and 4.

Each cell reports errors out of 200. At depths 256, 512, and 1,024, each test set contained 100 all-seen and 100 random queries. At depth 2,048, we evaluated 200 queries per distribution and report the errors separately.

Training seed2565121,0242,048 all-seen2,048 random
100000
700000
4300000
202400000
AuthorsOpenReview ↗

Response to Reviewer Rqb1 (4/4)

Q2. Sampling in the standard CS branch; measured costs

From update 20,000, each update adds 32 synthetic chains of depth 32 without prefix. The root is uniform over the 500 entities; the 32 relations are i.i.d. uniform over all 20 (adjacent pairs are not restricted to trained pairs); the number of completed positions k is uniform on {0, …, 31}; positions 1…k hold i.i.d. uniform entity embeddings, independent of the relations and of the true prefix values. One loop is applied; the frontier k + 1 has no target; wait-CS penalises positions k + 2…32, normalised by all non-frontier positions.

Measured costs (one H100 unless stated): training 6.9 h per wait-CS run, 6.9–7.6 h with constructed depth 128, 13–24 min per port run (hardware not logged); inference at depth 2,048 (2,048 loops over 2,049 tokens, batch 5) 22.4 s per chain, 2.5 GPU-hours for 400 chains; the Q3 diagnostic about 130 s per checkpoint on a seventh of an H100.

Q3. A diagnostic without deep answer labels

Yes. D_127 = relative displacement of the last waiting relation (position 128) from its input embedding after 127 loops on 300 depth-128 inputs; no answers are used. Threshold 0.9, fixed on 185 earlier checkpoints, then frozen. Correct calls on 120 later checkpoints that are clearly good (≥ 270 of 300 right at depth 128) or clearly bad (≤ 30):

Label-free measureE1000R401,000 × 40OthersAll
D_12730/3027/2712/1326/2695/96
Recovery probe of App. F.8, clean, 1 step24/3023/278/1325/2680/96
Same, noisy, 16 steps24/3025/2711/1326/2686/96
Depth-2 accuracy19/3021/279/1325/2674/96

All 418 measured checkpoints (pass = ≥ 297 of 300 at depth 128, no middle band):

FailsPasses
D_127 > 0.92684
D_127 ≤ 0.923123

Late regression: D_127 flags 12 of 14 good-to-bad checkpoint pairs in two navigation runs, but none of 5 in R40 seed 1 (its only good checkpoint is at 0.944). It detects failures; it is not a selection method.

Q4. The API comparison as context

We agree: the panel shows a practical reliability gap, not a matched compositional advantage (knowledge access, inference protocol and compute differ). We will say so next to Fig. 2 and give the outcome counts (of 40 per depth):

ModelCorrect at 128 / 256 / 512 / 1,024 / 2,048Failures at 2,048
Claude Opus 5.538 / 28 / 9 / 2 / 238 refusals
GPT-6 Sol32 / 27 / 11 / 2 / 017 output-limit stops, 23 wrong
Gemini 3.8 Flash37 / 31 / 24 / 21 / 1011 output-limit stops, 19 wrong
DeepSeek-V4.1-Flash32 / 18 / 1 / 0 / 040 wrong
Muse Spark 1.333 / 13 / 0 / 0 / 040 wrong

Every non-refused Claude Opus 5.5 answer is correct.

We thank the reviewer again for these constructive comments. We will add these results to the paper, and we would be grateful if the reviewer would reconsider their rating in light of them.

Reviewer_tGPjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how well does looped transformer generalize to bigger depths (from shallow training). They consider the following multi-hop problem. Fix 20 permutations p1... p20 learned at training. Then at inference, we are given x and r1...r_t, and compute p_{r_i} consecutively starting from x (which is in an alphabet of 500).

The main result is that, there is a way to train the transformer, using shallow/short length sequences only, which then generalize to much deeper/longer sequences. (The depth of the transformer is ~ length of the sequence.)

The important aspect is what training loss to use. The authors encode certain guidance into the loss function, including anchoring, CS, random prefixes. In short, they teach the model to keep inactive token (embeddings) precisely, so that there's minimal drift over 1000s of steps.

Critical strengths
  • the extrapolation from very shallow-depth sequences to nearly 2048 is impressive. in particular learning of the actual permutations p_i happens on 2-hop variant only (hence the paper name). Though there are slightly deeper training sequences (up to 128 at most).
  • the authors identify sound loss function to use to train such a transformer.
Critical weaknesses

The main critique I have is that the loss function hard-codes a lot of structure of the algorithm to solve the considered multi-hop problem. In particular, the anchoring/CS training loss directly guide the transformer to keep intact the entries that should stay intact (in the natural algorithm that just replaces r_i with the current iterate after i hops).

It is not at all clear to me how to generalize this to other problems (even for possibly other way to implement a solution for this specific problem!). Overall, is one expected to adapt the loss function to any ICL problem?

Questions for authors
  • how would you generalize this? To specific implementation for specific ICL problem? To possibly an unknown algorithm for a fixed ICL problem (ie, we may not know what is a good algorithm for a specific problem -- see, e.g., the results on k-hop problem where the best algorithm uses depth log(k) using a doubling algorithm)? And most importantly, to an unforeseen ICL problem?
2 comments
AuthorsOpenReview ↗

Response to Reviewer tGPj (1/2)

We thank the reviewer for the careful reading and for recognising the strengths of the paper: the extrapolation from very shallow training to depths 2048, the fact that the permutations themselves are learned from the two-hop variant only, and that the paper identifies a sound loss for training such a looped Transformer.

Training depth. No training row has an answer label deeper than two hops. The longer inputs (up to 131 tokens) carry no labels: random entity prefixes before shallow queries (answer unchanged) and CS states filled with randomly sampled entities. Extending the prefixes from 0–128 to 0–192 tokens raises accuracy at depth 192 from 72% to 100% (seed 1, 200 chains), still with labels of depth ≤ 2 only.

W1. The loss hard-codes the structure of the algorithm

We agree that anchors and CS encode the state layout of the serial algorithm: which positions wait and which hold their value. They do not encode the 20 maps, any intermediate value beyond the two-hop labels, or what the executing position computes (CS gives it no target). Encoding the structure alone does not produce depth generalization. Same depth ≤ 2 data in every row; accuracy (%), mean (variance) over three seeds:

Trainingd2d3d8d32d128
Chain-of-thought: every intermediate entity supervised100.00 (0.00)0.00 (0.00)0.00 (0.00)0.00 (0.00)0.00 (0.00)
Loop + anchors, no CS99.95 (0.01)99.43 (0.96)94.63 (73.08)42.19 (2618.99)15.59 (715.50)
Loop + CS, no anchors99.92 (0.02)97.73 (5.13)20.28 (411.69)0.15 (<0.01)0.15 (0.02)
Anchors + CS without weight sharing (8 steps of 4 layers)100.00 (0.00)0.00 (0.00)0.00 (0.00)0.00 (0.00)0.00 (0.00)
Loop + anchors + CS + prefixes (ours)100.00 (0.00)100.00 (0.00)100.00 (<0.01)100.00 (0.00)100.00 (0.00)

Accuracy pools the three test distributions (15,000 chains per depth and seed; chain-of-thought 3,000). 200k updates; chain-of-thought 50k, at 0 for depth ≥ 3 at every evaluation from 5k on. The model without weight sharing has 8 steps, so we report it only up to depth 8.

Supervising the algorithm's trace, state targets alone, or the same losses without a shared loop: none reaches depth 128. Deep composition needs both a shared loop that reuses one operation and label-free preservation on long constructed states.


AuthorsOpenReview ↗

Response to Reviewer tGPj (2/2)

W2. Must the loss be adapted to each problem?

The loss is not re-engineered per problem: every run below uses the same three terms (answer CE at depth ≤ 2, state anchors, CS) with the wait-CS role mask (one marked row uses the full-CS mask); a new problem only specifies which positions wait and where results are written. Accuracy (%) at depth 128, mean (variance) over the listed seeds; every listed run is ≥ 99% at every depth 2–128 in every test distribution.

(1) Problems that wait-CS solves as in the paper (constructed depth 32, no noise).

ProblemDepth 128
Main world100.00 (<0.01)
Same facts, another graph of trained relation pairs100.00 (0.00)
New random relations (world seed 43)100.00 (0.00)
2 trained successors per relation instead of 8 (¼ of the two-hop labels)100.00 (<0.01)
4 trained successors per relation99.98 (<0.01)

(2) Problems that need denoising wait-CS. Same loss term and role mask; only the constructed CS inputs change. Gaussian noise of relative scale up to 0.1 is added to the waiting and completed states, the waiting positions must still return to their clean states, and the constructed depth is 128 instead of 32. Answer data, anchors, prefixes and the 200k updates are unchanged. Both changes are in the paper (Section 3.3 and App. E: denoising for R40, constructed depth 128 for E1000); using them together on wait-CS, as one setting for the problems below, is new in this response. They target the failure measured in App. F.8: one-loop preservation of clean states does not ensure recovery from deviations that build up over many loops. On these problems the paper's wait-CS passes on no seed wherever we ran it.

ProblemDepth 128
40 relations (R40, App. E)100.00 (0.00)
1,000 entities (E1000, App. E)100.00 (0.00)
Navigation: 24 × 25 grid, 100 walls, 20 moves; a blocked move stays in place, so the maps are not bijections (designed after the setting was fixed)100.00 (0.00)
Group world: the 120 elements of S5, relations = right multiplication100.00(one seed)
1,000 entities × 40 relations (full-CS form)99.42(one seed)
In-context tables of 8 / 12 / 16 rowssee Q1

The same denoising setting also passes on seeds 1, 7 and 42 of every problem in part (1) where we ran it (main world, other graph, world seed 43, 4 successors).

Q1. A specific implementation for a specific ICL problem

Our main task stores the relations in the weights, so we built an in-context version: each input carries its own random permutation of E entities as E key–value rows, then a start entity and d identical hop tokens; answers are labelled at one and two hops only. The loss is unchanged; the hop positions play the role of the relations. Denoising wait-CS, accuracy (%), mean (variance), 1,000 fresh tables per depth:

Rowsd8d32d64d127d128
8100.00 (0.00)100.00 (0.00)100.00 (0.00)100.00 (0.00)100.00 (0.00)
12100.00 (0.00)100.00 (0.00)100.00 (0.00)99.95 (0.005)99.85 (0.045)
16100.00 (0.00)100.00 (0.00)100.00 (0.00)99.95 (0.005)99.85 (0.045)

All 8-row tables occur in training, so only the depth is new; about 5% of the 12-row and at most 3.1 × 10⁻⁶ of the 16-row test tables can occur in training, so those are new.

Q2–Q3. Unknown algorithms and unforeseen problems

  • CS is a general preservation objective that uses no answer labels: beyond which positions are to be preserved, it needs no task-specific labels.
  • As the reviewer points out, we solve the k-hop problem with one hop per loop step, and more efficient algorithms (e.g. log(k) depth by doubling) may exist. We show that solving it this way enables compositional generalization far beyond the labelled depth (tested up to depth 2048). We do not claim that this algorithm is optimal in efficiency, and our method does not discover such an algorithm by itself.
  • Our method is hard to apply directly to unforeseen problems, since it needs the state layout. Still, step-by-step execution, as in chain-of-thought, is a highly general strategy used across many problems, and CS can serve as a training objective for such a CoT-like process over loop steps.

We will state these points and add the results above to the paper. We thank the reviewer again for these constructive comments, which helped us clarify the scope of the method, and we would be grateful if the reviewer would reconsider their rating in light of these results.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.