- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates how attention head activities are redistributed across layers when a serial task demands more steps, within a single forward pass of a fixed-depth Transformer with no intermediate tokens. Head's activity is defined by the L2 norm of its output into the residual stream, and a layer's concentration is the share held by its top-10% heads. The paper tracks this concentration as the step count k grows, and builds a signature to summarize the behavior of each model. Across four serial tasks and 17 open-weight models from four families, the results are reproducible: in most models, activity concentrates around the early-to-middle layers and then spreads in the deeper ones. Models also differ from one another: the back half of the smaller Qwen2.5 base models barely spreads, whereas the Llama models and OLMo-2 spread markedly.
Critical strengths
- The paper evaluates 17 models of different sizes from four families (Llama, Qwen2.5, OLMo-2, Mistral), and shows how active attention heads are redistributed across layers as the required number of steps k grows. This gives a useful picture of how attention head behavior differs across models when they perform different tasks.
- The paper conducts extensive experiments both in the main text and appendix to support reproducibility.
Critical weaknesses
- The motivation for the study is unclear. Most LLMs today are strong reasoners, and they carry out step-by-step inference inside a chain of thought, so the "no intermediate token" setting rarely occurs in practice. The question "how attention heads are distributed across layers in serial tasks" itself is a worthy topic to work on, but the authors just only compare models against one another. The second half of the paper is an elaborate numerical analysis of a fairly small set of experiment results. It repeatedly establishes that models differ, yet it never explains why. Model size, architecture (e.g. the number of heads per layer), training stage (e.g. whether the model has been through RL) and the content of the task are all left on the table. To be fair, the authors do report correlations between size, heads per layer, accuracy and the degree of spreading, and the Qwen2.5 range from 0.5B to 14B is a natural contrast, but the paper still stays at the level of correlation. It offers no mechanistic account, and it designs no controlled comparison that could separate size, head count, accuracy and family. That is precisely where the value of this question lies.
- The practical value of the paper is limited. The prompts cover only 4 task families plus a numeral control, and all of them are toy tasks. As an interpretability work, what a researcher can take away from this paper is largely the construction of the task setups, but it is difficult to see how the results could be used to improve LLMs in practice.
Questions for authors
- Sec. 3.3 defines the activity of a head as the L2 norm of what it writes into the residual stream. That is a level of activity, not a change with k. In serial tasks, I would expect the heads that matter to be the ones whose behavior shifts as the number of steps grows. Could the authors rank heads by the size of their response to k, select them on that basis, and repeat the same experiment? That would test the idea directly.
- In Sec. 3.4, the paper presents the model signature as a property of the model that carries over to other tasks, and cites r=0.93 as evidence that it is reliable. But the paper's own numbers are less encouraging: the same model on two different tasks correlates at only r=0.30, and different models on the same task correlate at r=0.22 (Line 212). The right panel of Figure 2 raises the similar doubt: when signatures are compared between models, pairs from different families and of different sizes can be more similar than two models within one family (e.g. L3.2-1B and Q0.5B are highly similar to each other, more so than several off-diagonal Qwen pairs). I am therefore skeptical that the signature measures what the paper claims it does. Is there a better way to define it?
2 comments
Reviewer MrzM, part 2 of 2
Q2, continued: Figure 2 (right). Signature similarity follows the pretraining run, not the developer. Pairs within a lineage (a base model and its post-trained versions) correlate 0.73 on average and separate pretraining runs −0.11, and base models from the same developer are only marginally more similar than chance (0.09 against −0.05, p = 0.06; Section 4.3). Two models from different families, such as Llama-3.2-1B and Qwen2.5-0.5B (r = 0.62), can therefore resemble each other more than two Qwen2.5 sizes do. The family contrast is a narrower claim about one feature of the profile, the back-half response. There, the three Llama base models separate completely from the four Qwen2.5 base models up to 7B in three of the four tasks (exact p = 1/35 each), now also with every model under the same attention kernel. In the revision we will define the signature as the reproducible profile on a task family, report its cross-task component separately with its effect size, and use the back-half and 50–70% responses as the interpretable summaries.
Weakness 1: motivation and controls. Chain of thought moves this question rather than removing it: each written step is still computed within one forward pass, models often fail to combine two facts they know without intermediate tokens (Press et al., 2023), and the second hop is often resolved late or not at all (Biran et al., 2024; Yang et al., 2024). How a fixed-depth network handles more composition within a pass therefore bears on multi-step reasoning and on circuit analyses, which are mostly done in this setting. We agree that the paper is observational. It does separate the factors in part. The Llama-3.1-70B and Qwen2.5-72B pair has the same number of layers and heads and still differs (Appendix D). Among the nine models with 32 heads per layer, model identity explains 30% of the non-shared variation (18% by chance). The Qwen2.5 series varies size within one family. For training stage, the OLMo-2 checkpoints after supervised fine-tuning, DPO and a final reinforcement-learning stage (RLVR, in the Instruct model) keep the base model's profile (signatures correlate 0.76–0.87 with the base), and the new matched analysis holds accuracy and answer uncertainty fixed (our response to Reviewer 5Bu5, Q2). Task content matters a great deal: most of what distinguishes a model on a task is specific to that task (Section 4.1). Separating size, head count and training data fully needs models that differ in one factor at a time. Intermediate training checkpoints, where architecture and data are fixed while ability grows, are the next step we are pursuing: they test whether the pattern changes before the ability does.
Weakness 2: practical value. The tasks are synthetic so that the number of steps can vary at constant prompt length, with balanced answers and independently verified solutions. The paper does not propose a way to improve models; its consequence is for interpretability. A circuit identified at one depth of composition need not describe a deeper one, and models differ in how concentrated the computation is, which carries over to held-out tasks as a difference in how much the answer depends on a few heads (our response to Reviewer 4dQU, Q1).
Reviewer MrzM, part 1 of 2
Thank you for the review and for two concrete questions, which we answer with new analyses; their code is in the anonymized supplementary material.
Q1: level of activity versus change with k. First a clarification: the quantity the paper studies is a change with k. For each layer we regress the top-10% share on k and standardize the slope against a permutation null (Section 3.4); the output norm only supplies the per-head activity from which the shares are computed. Your suggestion goes further, selecting heads by their own response to k. We have done the descriptive part: for each head, we regressed its share of the layer's activity on k and standardized the slope the same way.
Many heads respond. In back-half layers, 66% of heads change their share significantly with k (51% in the front half; about 5% expected by chance).
The most responsive heads are mostly not the most active ones, although the overlap exceeds chance: per layer, the q heads with the largest response overlap the q most active heads by 17% in the back half, against 12.5% by chance. Selecting heads by their response therefore picks largely different heads.
Repeating the ablation with heads selected by their response to k needs new GPU runs, and we consider it the most important next experiment. What the existing runs show is exploratory and inconclusive (see our responses to Reviewer 5Bu5, Q3, and Reviewer 4dQU, Q2).
Q2: what the signature measures. r = 0.93 is the item-split reliability of the signatures: the same models and tasks, recomputed on disjoint halves of the items. We report it to show that the measurement is reliable, not that signatures carry over across tasks. Carry-over across tasks is measured separately, and the paper reports it as modest: signatures from two tasks predict those from the other two at split-half r = 0.36 (chance 0.00, p < 0.001), and a model on two different tasks resembles itself (r = 0.30) more than different models resemble each other on the same task (r = 0.22). We agree that Sections 3.4 and 4.1 should say this plainly, and we will revise them.