- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper investigates whether looped Transformers can generalize to long compositions using only shallow supervision (extrapolation).
The task setting includes 500 entities and 20 random permutation relations. Training includes all one-hop facts and selected two-hop compositions, while testing extends to 2048 hops.
The model is trained using shallow answer supervision, intermediate-state anchors, random prefixes, and a constructed-state consistency loss that encourages preservation of waiting states.
The proposed method achieves 100% accuracy through depth 1024 and 99% at depth 2048, while standard cross-entropy training fails beyond shallow compositions.
Critical strengths
I think this paper focus on a timely problem: extrapolation of looped transformer, and is a first step towards the understanding of looped tf.
Critical weaknesses
The paper is interesting. That said, there exists some weaknesses to me.
- The error-accumulation explanation is not particularly novel. I think this intuition comes already from some early works in skill composition like [1] (though this paper proposes some mechanism other than the normal error accumulation). And in this paper [2], you can even understand where this imperfection comes. (though [2] isn't a looped tf paper, rigorously speaking.)
[1] h1: Bootstrapping Models to Reason over Longer Horizons via Reinforcement Learning
[2] Transformers Provably Learn Algorithmic Solutions for Graph Connectivity, But Only with the Right Data
- Intermediate-state restoration is an expected solution to accumulated errors. By projecting intermediate states back to canonical entity embeddings and restoring waiting relations, the intervention kind of resets accumulated representation errors after each step (therefore to me this is something trivial?).
I kind of expecting the author to justify what are the new messages the authors are trying to convey.
Best
Questions for authors
See the weaknesses above.
2 comments
Response to Reviewer v9Fx (2/2)
W2. Intermediate-state restoration is an expected solution to accumulated errors
As a test-time intervention, we agree, and we do not propose it as a solution. In the paper it is a diagnostic (Fig. 4, App. F.6.1); our model is evaluated with frozen weights and continuous states, without any projection or reset (App. B.2). We will say this more plainly. What we consider non-trivial is the following.
- Restoration discriminates between failures, and its two parts do different things (table in W1; the paper mostly reports the two combined, App. F.8.2). With CE only, neither part alone helps and both together recover 298 of 300. With anchors only, the reset alone suffices; projecting computed states onto entity embeddings, the part closest to "resetting the accumulated error" of the computation, recovers nothing. With wait-CS only, even both together recover only 235. For our recipe at 120k, the projection alone suffices.
- Training towards the restored states is not enough. Anchors are the training-time counterpart of restoration: on the labelled shallow rows they pull each computed state to its entity embedding and each waiting relation to its input embedding. Alone they give 0.20% at depth 128 (W1; 0.09% in Table 1). Consistency on completed values (root + hold) also fails (0.24%), whereas consistency on waiting positions alone gives 100% (Table 2). Which state has to be preserved, and that this can be trained on constructed states without labels, is the result.
- After training, nothing has to be restored. Restoration must be told which position has just executed, and it discretises that state. The trained model needs neither: natural execution gives 300 of 300, and 0 errors in 120,000 test chains at depths 2–128. Over training, restoration already gives 283 of 300 depth-128 chains at 5k updates, while natural execution reaches 299 only at 130k (Fig. 4).
W3. What are the new messages?
Two labelled hops suffice for deep composition in a standard looped Transformer (full causal attention, continuous states, nothing added at inference), if training also asks, without labels, that inputs not yet used survive a loop. Alternatives stop near their label depth: answers to depth 32 fail at 64 and rewards to depth 8 fail at 32 (Table 1); chain-of-thought with every intermediate entity supervised on the same data gives 0% from depth 3 (new; response to Reviewer tGPj, W1).
What decides depth is a write–removal balance. Compositional generalisation is related to two measured quantities: the output error of each newly written state, and the loop's ability to remove noise from the states it carries. The state at depth d does hold what earlier loops left in it, but that residue is the outcome of error written against error removed. It is not independent per-step compounding (accuracy p^h): with every single step right, depth 128 is at 0.2% (W1). New: our recipe, seed 7, rerun with a checkpoint every 10k updates.
| Updates | 20k | 60k | 100k | 110k | 120k | 130k | 140k | 150k | 200k |
|---|---|---|---|---|---|---|---|---|---|
| Output error of a new state | 4.85 | 3.33 | 2.36 | 2.05 | 1.60 | 1.36 | 1.13 | 0.78 | 0.24 |
| Noise left after 16 loops / noise injected | 181 | 42.5 | 38.5 | 0.56 | 2.83 | 3.10 | 0.48 | 0.29 | 0.0017 |
| Depth 128, correct of 300 | 0 | 23 | 212 | 221 | 253 | 283 | 299 | 300 | 300 |
Output error: distance of a freshly written state from its entity embedding, in units of half the distance to the nearest other entity (largest of five positions). Noise: relative size 0.1 on the waiting relations of constructed states, squared error (probe of App. F.8.1). Up to 130k the loop amplifies injected noise at every checkpoint but one; from 140k it removes it and depth 128 is reached, while the output error keeps falling. The larger world of App. F.8.1 shows the balance lost: its one-step error improves, noise is amplified over 16 loops, and depth-128 accuracy falls from 299 to 2 of 300.
Q1. "See the weaknesses above."
Answered in W1–W3. In the revision we will (i) cite [1] and [2] and state which part of the explanation is prior intuition and which is our evidence; (ii) present restoration explicitly as a diagnostic, split into its two parts; (iii) add the tables and results above; (iv) state the two messages of W3 at the end of the Introduction.
We thank the reviewer again for these comments, which helped us state more precisely what is new. We hope the clarifications and the new results address the concerns, and we would be grateful if the reviewer would reconsider the rating in their light.
Response to Reviewer v9Fx (1/2)
We thank the reviewer for the comments, and for recognising that the paper is interesting, addresses a timely problem (the extrapolation of looped Transformers) and is a first step towards understanding them. We answer each weakness and the question in turn. "New" marks results obtained for this response; all of them use answer labels of depth ≤ 2 only.
W1. The error-accumulation explanation is not particularly novel ([1], [2])
We agree that the general intuition, that errors build up over a long horizon, is not new, and we do not claim it. We will cite and discuss [1] and [2]. What the paper adds is specific to looped Transformers: it locates, by intervening on the internal states, which state carries the error, and it removes that error with a training objective that needs no answer labels.
Every single step is right, yet depth fails (new). All numbers below are for seed 7, the seed and setting of Table 3. Trained with anchors only or with wait-CS only, a state written from clean inputs is used correctly by the next operation in 4,800 of 4,800 and 4,797 of 4,800 events (the test of Table 3, where CE only gives 56.56%). If steps failed independently at these rates, accuracy at depth 128 would be at least 92% and 81% (95% bounds). Measured: 0.20% and 0.21%.
Which state carries the error (new). Correct chains of 300 at depth 128, with analysis-only edits after every loop. Reset: relations whose turn has not come are set back to their input embeddings. Projection: the state just computed is replaced by its nearest entity embedding (the model's own choice, no gold values).
With anchors only, the error sits entirely in relations that have not been used yet: restoring them recovers every chain, while cleaning every computed value recovers none. At 120k updates of our recipe, the remaining error is instead in freshly computed values. These are two separable sources. The first, the drift of operands still waiting for their turn, is what wait-CS removes without labels.
So the state-management skill that [1] conjectures beyond step accuracy is here identified, measured and trained from two-hop labels. Our baselines trained on deeper answers instead stop within twice their label depth (Table 1: answers to depth 32 give 0.25% at depth 64; rewards to depth 8 give at most 0.15% at depth 32).