- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates the causes of failure to achieve compositional generalization in Looped Transformers. Models are trained of small (short) instances of a task and asked to perform longer instances. For the chosen tasks the reference computational algorithm is known. On longer instances the model must infer the syntax of the instance and compute the answer. The authors investigate possible causes for failures and find that syntax inference is implicated.
Critical strengths
The setup is very good, with a ladder tasks (some related) that make operand location progressively more difficult. The novel experimental setup helps separate syntax inference from syntax execution. This is conceptually valuable. The finding that training models with oracle-provided operand locations helps them generalize (and the converse) is good. Showing that introducing oracle-provided attention masks at inference time can increase model accuracy is strong experimental evidence. Ditto brief adaption training, and top k restriction. The use of left caterpillar, right caterpillar and complete tree tasks adds detail to the picture. Compare to many in distribution / OOD papers this is a detailed and nuanced exploration. Well done.
Critical weaknesses
The paper assumes that the reference DAG is the algorithm that the model learns. Without showing the algorithm the model learnt, this remains an assumption. Other algorithm-variants are mathematically feasible.
The model has at least these capabilities: A) Identify the operands by determining the correct dependency positions B) Retrieve their values (sometimes via intermediate representations), and C) execute the calculation. The paper groups A and B together. I'd like to see them named/investigated separately, via a few more experiments.
The oracle mask intervention is good evidence, but may hide other mechanisms. Maybe the model's attention distribution is too diffuse to retrieve the intermediate representations reliably.
The abstract's "We establish syntax inference as the cause of this generalization failure" seems overstated.
With additional experiments that directly measure binding representations and intermediate execution, this could be a very useful framework for mechanistic investigation of how models learn and fail to generalize compositional algorithms.
Questions for authors
Are more experiments planned? I would like to see:
- Directly test whether the models infer the correct operand identities by training an explicit probe on the models' intermediate representations.
- Measure execution errors separately from retrieval errors.
3 comments
Thank you for your detailed engagement with the reviewers. As a result of your efforts I'm sure your paper is stronger now. Best wishes
Thank you again for your kind words and constructive feedback. Your suggestions directly motivated the new analyses: the separate measurement of identification, retrieval and execution errors (§4.3, L.427–476), the operand-identity probe (L.470–472, Fig. E.4), and the abstract restated by task and depth (L.24–28). We believe they have strengthened the paper.
If you feel that our revisions have further strengthened the case for acceptance of our paper, we would be very grateful if you would consider updating your score to reflect this.
Thank you for the encouraging review. You suggested that, with probes and separate execution measurements, this could become a useful framework for mechanistic investigation. We ran exactly these, in a new §4.3 and Appendix E.
W1. The reference DAG is assumed. New perturbation tests (L.463–466, App. B.2, Fig. E.3). If the model computes as in Eq. (1), each operator's output depends only on its operands' outputs. We test this three ways: resampling the leaves under an operand, swapping the operand's state, and resampling leaves outside the subtree. In the training range all three hold on ragged trees: at least 92% of tested operators respond as Eq. (1) predicts. So its computation is consistent with the reference DAG there, which makes the decomposition informative. At depth 8, the three properties still hold on ragged trees (86–97%) but fail on complete trees (45–59%). That is where operators stop reading their operands, the identification failure (A, see W2).
W2 and Q2. Separating A, B and C. New §4.3 (method in App. B.2). We first set aside errors propagated from a wrong operand. Each other wrong operator gets one label: • (A) identification, tested by its attention on its operands. • (B) retrieval, tested by a linear probe on its state. • (C) execution, otherwise. For the base model on complete trees at depth 8, 84% of operators are wrong. Almost all are classified as (A): 78% of all operators, against under 1% for (B) and (C) (Fig. 6, E.1). This also measures execution separately from retrieval (Q2).
W3. Diffuse attention. Sharpening attention does not solve the problem. On complete trees at depth 8, the left-operand head of wrong operators places 0.94 of its attention on tokens other than the operator and its operands (Fig. E.2). Sharpening it to the model's own top positions (top-k, Fig. 4) does not help. On scans you are partly right. A training-length window repairs them as well as the true mask, because a scan's operand is always recent. The same window does not help trees (L.367–369, Fig. E.8).
W4. Overstated abstract. We agree, and rewrote it to state the claim by task and depth (L.24–28).
Q1. Probe for operand identities. New probe (L.470–472, Fig. E.4). A linear probe reads where the left operand lies from the operator's state. It succeeds for 87% of correct operators but only 54% of wrong ones (ragged trees, depth 8).
Thank you again. Your suggestions made the analysis much more precise.