- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper provides a study on how supervision density (the number of tokens supervised in the loss function) shapes the type of solution learned by a transformer and how well that solution generalizes. They train 12-layer transformer to solve a multi-hop variable binding task, and supervise a range of tokens (p=0 refers to the standard last-token supervision, while p=1 means all the tokens are supervised). They analyze how this affects convergence dynamics, attention and MLP solutions, and generalization capabilities.
Critical strengths
- The experimental setup is clear and the analysis makes sense. Both the experimental setup and analysis are clearly related to the problem at hand. Additionally, the plots are well-made and understandable, and I can (for the most part) follow the train of thought in the results section.
Critical weaknesses
The primary weakness I identified is that it is unclear how generalizable these results are. Per my understanding (please correct me if I missed something, especially in the appendix): the study is done with i) 12-layer transformers and ii) one toy task. Thus, it is very unclear how these results hold to language / reasoning models more broadly, especially as you scale up parameters, layers, etc. Additionally, I couldn't find the number of seeds used, which further calls into question the generalizability (see Questions for Authors 1.). Some concrete suggestions that would improve the paper:
1. Run multiple seeds (again, refer to questions for authors and please correct me if I'm mistaken).
2. Evaluate on longer contexts. Currently, the paper argues for "long-context generalization" (line 152), but the ctx length x goes only up to 26. Can the authors clarify how long-context this setup actually is? Transformers today are scaled up to much bigger size and hundreds of thousands (even 1-M token context length), so the claim about long-context generalization is in doubt for me, especially with the title being about "transformer solutions", which is quite a broad claim.
3. Add an additional toy task. The task doesn't have to be huge, but this would certainly help with my worry about how generalizable these results are.
Questions for authors
-
Can the authors clarify the number of seeds they used? Lines 132 and 133 say "Splits are deterministic given a seed (450,000 training, 1,000 validation, and 49,000 test programs) and are the same in every run". Does this mean 450,000 different models / seeds? I do not see error bars on the plots in the main text, which is making me question the statistical significance of the results (e.g. it's hard to place in context what the difference between p=0 and p=0.25 is in Figures 2 and 3 means in Figures 2 and 3). If the authors can provide information on the number of seeds used / how many different models were trained, that would be helpful.
-
How strongly do these experimental results transfer to transformers more generally? The claim in the title is about "transformer solutions and long-context generalization", which is broad. However, the actual study is conducted on 12-layer transformers and one multi-hop task, which seems narrow.
3 comments
I thank the authors for the updates. I'm still concerned about the lack of scale, using only a 12-layer transformer is prohibitive to the study. Even going to 24 and 36 layers would be beneficial. Also this may be pedantic, but regarding weakness 2) I would refer to that as length generalization instead of long context. Long context typically means a huge # of tokens, but an increase over training tokens is more length generalization. I have updated to a borderline 3 with these concerns.
We would like to thank the reviewer for their time to evaluate our rebuttal, raising their score, and seeing value in our work. We are currently evaluating the possibility of scaling these results to bigger models in a collaboration with other institutions. Again, thank you for your time and suggestions that have improved the quality of our work.
We are grateful to the reviewer for their positive assessment of our work and for the time and effort devoted to its evaluation. We particularly appreciate the constructive observations and valuable suggestions provided, which have increased the quality of our work. In the following, we respond to each comment in detail and clarify the points raised.
Critical weakness 1: We apologize for this confusion. In fact, we had run more runs which all showed similar results, and omitted to add the aggregate results in the appendix. The revised version will also contain the averaged results from 3 seeds. Across the three seeds, the results hold: intermediate densities converge fastest, p = 0 generalizes best at step 300k in every seed, and at step 300k the solution is attention-dominated for p = 0 and MLP-dominated for p ≥ 0.25 (see Table 1 below).
Table 1. Mean accuracy ± sd over three seeds. Steps: first maximal accuracy model. LC: long-context accuracy (%), 18–26 lines × 5–8 hops. d_attn − d_mlp: direct contributions to the answer margin (positive: attention dominates).
Critical weakness 2: Evaluating on longer contexts, similar to those of frontier models (1M tokens), is prohibitively costly in our case. However, we do want to clarify that the condition “lines 26” implies a context length of 26*4+4 = 108 tokens, which is an increase of ~60% over the training input length.
Critical weakness 3: We thank the reviewer for this suggestion. To demonstrate the generalizability of our results, we added a second task, nested boxes: we extended the entity-tracking task [1], in templated natural language, to multi-hop containment (e.g., "the computer is in Box W, Box W is in Box N, …, Box N contains the" → "computer"), and reduced the objects from 85 to 10 to match the 10 digits of x=y. We replicated our findings with this natural language task: answer-only supervision generalizes best after continued training, and denser supervision yields more MLP-dominated solutions (see Table 2 below). We will add the results of this novel task in the revised version.
Table 2. Nested boxes results. Steps: first maximal accuracy model. LC: long-context accuracy (%), 18–26 clauses × 5–8 hops. d_attn − d_mlp: direct contributions to the answer margin (positive: attention dominates).
We will add the results of this novel natural language task to the revised version of the paper and thank again the reviewer for suggesting this addition which broadens the scope of our paper.
[1] Prakash, Nikhil, et al. "Fine-tuning enhances existing mechanisms: A case study on entity tracking." International Conference on Learning Representations. Vol. 2024. 2024.
Questions:
(i) We ran 3 seeds leading to 15 models trained. Our results remain unchanged and our revised version will contain error bars in the plots.
(ii) Evaluating this on broader models is, unfortunately, prohibitively costly for us, and we do not have access to the infrastructure to evaluate the generalizability of our results to bigger, state-of-the-art models. Note that our work requires training models from scratch. However, as suggested by the reviewer we did expand the scope of our results to another, natural language, tasks.