acceptodds
Under review as a conference paper at ICLR 2027

How Language Models Differ in Redistributing Attention-Head Activity under Serial Demand

Abstract

The way a model distributes activity over each layer's attention heads offers a coarse view of how it routes information through depth; how this changes with the task is part of what a mechanistic account must explain. Holding prompt length fixed, we vary how many serial steps a task demands and measure, in every layer of 17 open-weight models, whether activity concentrates on a few heads or spreads across many as demand rises. Both occur: in most models, layers just before mid-depth concentrate activity and later layers spread it. Models differ in where and how strongly this happens. The Qwen2.5 base models from 0.5B to 7B, for example, spread less than the average model in every task and concentrate activity in parts of their second half, where Llama models from 1B to 8B and OLMo-2 spread; the contrast largely holds between Llama-3.1-70B and Qwen2.5-72B, which have the same number of layers and heads. These differences are reproducible, and post-trained models keep much of their base model's pattern. An ablation study suggests that, within a task, models whose activity is more concentrated on their top heads also depend more on those heads for the answer. Concentration and spreading across layers thus offer a new way to compare models, by how they route information through depth.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, attention heads, large language models, multi-step reasoning, serial computation, information flow, model comparison, universality

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MrzMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how attention head activities are redistributed across layers when a serial task demands more steps, within a single forward pass of a fixed-depth Transformer with no intermediate tokens. Head's activity is defined by the L2 norm of its output into the residual stream, and a layer's concentration is the share held by its top-10% heads. The paper tracks this concentration as the step count k grows, and builds a signature to summarize the behavior of each model. Across four serial tasks and 17 open-weight models from four families, the results are reproducible: in most models, activity concentrates around the early-to-middle layers and then spreads in the deeper ones. Models also differ from one another: the back half of the smaller Qwen2.5 base models barely spreads, whereas the Llama models and OLMo-2 spread markedly.

Critical strengths
  1. The paper evaluates 17 models of different sizes from four families (Llama, Qwen2.5, OLMo-2, Mistral), and shows how active attention heads are redistributed across layers as the required number of steps k grows. This gives a useful picture of how attention head behavior differs across models when they perform different tasks.
  2. The paper conducts extensive experiments both in the main text and appendix to support reproducibility.
Critical weaknesses
  1. The motivation for the study is unclear. Most LLMs today are strong reasoners, and they carry out step-by-step inference inside a chain of thought, so the "no intermediate token" setting rarely occurs in practice. The question "how attention heads are distributed across layers in serial tasks" itself is a worthy topic to work on, but the authors just only compare models against one another. The second half of the paper is an elaborate numerical analysis of a fairly small set of experiment results. It repeatedly establishes that models differ, yet it never explains why. Model size, architecture (e.g. the number of heads per layer), training stage (e.g. whether the model has been through RL) and the content of the task are all left on the table. To be fair, the authors do report correlations between size, heads per layer, accuracy and the degree of spreading, and the Qwen2.5 range from 0.5B to 14B is a natural contrast, but the paper still stays at the level of correlation. It offers no mechanistic account, and it designs no controlled comparison that could separate size, head count, accuracy and family. That is precisely where the value of this question lies.
  2. The practical value of the paper is limited. The prompts cover only 4 task families plus a numeral control, and all of them are toy tasks. As an interpretability work, what a researcher can take away from this paper is largely the construction of the task setups, but it is difficult to see how the results could be used to improve LLMs in practice.
Questions for authors
  1. Sec. 3.3 defines the activity of a head as the L2 norm of what it writes into the residual stream. That is a level of activity, not a change with k. In serial tasks, I would expect the heads that matter to be the ones whose behavior shifts as the number of steps grows. Could the authors rank heads by the size of their response to k, select them on that basis, and repeat the same experiment? That would test the idea directly.
  2. In Sec. 3.4, the paper presents the model signature as a property of the model that carries over to other tasks, and cites r=0.93 as evidence that it is reliable. But the paper's own numbers are less encouraging: the same model on two different tasks correlates at only r=0.30, and different models on the same task correlate at r=0.22 (Line 212). The right panel of Figure 2 raises the similar doubt: when signatures are compared between models, pairs from different families and of different sizes can be more similar than two models within one family (e.g. L3.2-1B and Q0.5B are highly similar to each other, more so than several off-diagonal Qwen pairs). I am therefore skeptical that the signature measures what the paper claims it does. Is there a better way to define it?
2 comments
AuthorsOpenReview ↗

Reviewer MrzM, part 1 of 2

Thank you for the review and for two concrete questions, which we answer with new analyses; their code is in the anonymized supplementary material.

Q1: level of activity versus change with k. First a clarification: the quantity the paper studies is a change with k. For each layer we regress the top-10% share on k and standardize the slope against a permutation null (Section 3.4); the output norm only supplies the per-head activity from which the shares are computed. Your suggestion goes further, selecting heads by their own response to k. We have done the descriptive part: for each head, we regressed its share of the layer's activity on k and standardized the slope the same way.

Many heads respond. In back-half layers, 66% of heads change their share significantly with k (51% in the front half; about 5% expected by chance).

The most responsive heads are mostly not the most active ones, although the overlap exceeds chance: per layer, the q heads with the largest response overlap the q most active heads by 17% in the back half, against 12.5% by chance. Selecting heads by their response therefore picks largely different heads.

Repeating the ablation with heads selected by their response to k needs new GPU runs, and we consider it the most important next experiment. What the existing runs show is exploratory and inconclusive (see our responses to Reviewer 5Bu5, Q3, and Reviewer 4dQU, Q2).

Q2: what the signature measures. r = 0.93 is the item-split reliability of the signatures: the same models and tasks, recomputed on disjoint halves of the items. We report it to show that the measurement is reliable, not that signatures carry over across tasks. Carry-over across tasks is measured separately, and the paper reports it as modest: signatures from two tasks predict those from the other two at split-half r = 0.36 (chance 0.00, p < 0.001), and a model on two different tasks resembles itself (r = 0.30) more than different models resemble each other on the same task (r = 0.22). We agree that Sections 3.4 and 4.1 should say this plainly, and we will revise them.

AuthorsOpenReview ↗

Reviewer MrzM, part 2 of 2

Q2, continued: Figure 2 (right). Signature similarity follows the pretraining run, not the developer. Pairs within a lineage (a base model and its post-trained versions) correlate 0.73 on average and separate pretraining runs −0.11, and base models from the same developer are only marginally more similar than chance (0.09 against −0.05, p = 0.06; Section 4.3). Two models from different families, such as Llama-3.2-1B and Qwen2.5-0.5B (r = 0.62), can therefore resemble each other more than two Qwen2.5 sizes do. The family contrast is a narrower claim about one feature of the profile, the back-half response. There, the three Llama base models separate completely from the four Qwen2.5 base models up to 7B in three of the four tasks (exact p = 1/35 each), now also with every model under the same attention kernel. In the revision we will define the signature as the reproducible profile on a task family, report its cross-task component separately with its effect size, and use the back-half and 50–70% responses as the interpretable summaries.

Weakness 1: motivation and controls. Chain of thought moves this question rather than removing it: each written step is still computed within one forward pass, models often fail to combine two facts they know without intermediate tokens (Press et al., 2023), and the second hop is often resolved late or not at all (Biran et al., 2024; Yang et al., 2024). How a fixed-depth network handles more composition within a pass therefore bears on multi-step reasoning and on circuit analyses, which are mostly done in this setting. We agree that the paper is observational. It does separate the factors in part. The Llama-3.1-70B and Qwen2.5-72B pair has the same number of layers and heads and still differs (Appendix D). Among the nine models with 32 heads per layer, model identity explains 30% of the non-shared variation (18% by chance). The Qwen2.5 series varies size within one family. For training stage, the OLMo-2 checkpoints after supervised fine-tuning, DPO and a final reinforcement-learning stage (RLVR, in the Instruct model) keep the base model's profile (signatures correlate 0.76–0.87 with the base), and the new matched analysis holds accuracy and answer uncertainty fixed (our response to Reviewer 5Bu5, Q2). Task content matters a great deal: most of what distinguishes a model on a task is specific to that task (Section 4.1). Separating size, head count and training data fully needs models that differ in one factor at a time. Intermediate training checkpoints, where architecture and data are fixed while ability grows, are the next step we are pursuing: they test whether the pattern changes before the ability does.

Weakness 2: practical value. The tasks are synthetic so that the number of steps can vary at constant prompt length, with balanced answers and independently verified solutions. The paper does not propose a way to improve models; its consequence is for interpretability. A circuit identified at one depth of composition need not describe a deeper one, and models differ in how concentrated the computation is, which carries over to held-out tasks as a difference in how much the answer depends on a few heads (our response to Reviewer 4dQU, Q1).

Reviewer_4dQUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a per-layer quantity that it calls spreading or concentration, and uses it to profile 17 open-weight models on several serial tasks. For the last token in the prompt, the measure tracks the norm of the attention-head output carried by the strongest 10% of the heads in any layer, and reflects the change in the relative norm as a function of k, the number of steps in the serial task. The measure is applied to four synthetic tasks (pointer chasing, modular arithmetic, variable binding, relational lookup) and a control task in which the prompt specifies k, but the serial task requires a constant number of steps. Note that each task is executed through a single forward pass with no chain of thought. For a given layer, a spread is identified when the relative norm decreases with k, while a concentration is identified when the relative norm increases with k. These trends are standardized by measuring the mean slope over permutations, and depth is standardized relative to the total number of layers in the respective model.

The main findings are that most models, including Llama and OLMo-2, concentrate just before mid-depth and spread at depths 50–80%. The small Qwen2.5 base models (up to 7B) exhibit no net spreading in the back (top) half of the networks. These profiles are stable when recomputed on different prompts. Fine-tuned models maintain their base model’s profile. Finally, mean-ablation of the top heads at 40–80% of depth significantly hurts most of the models substantially, but not the small Qwen models.

Critical strengths
  1. Novel network behavior measures. Profiling transformer networks by spreading/concentration is novel and potentially useful.

  2. Measurement rigor and reproducibility. Prompt length and answers are controlled, an independent solver verifies every item, and the per-model profiles replicate on disjoint items and on freshly generated ones. The results remain consistent across alternative statistics and settings (Table 7). These procedures make the results, including Qwen2.5's unique behavior, credible.

Critical weaknesses
  1. Lack of utility. The paper offers “a new way to compare models,” but it does not offer a mechanism, a function, or a use for such profiling.

  2. Head importance is not tracked. The measure is applied to the strongest 10% of the heads in a layer, but the behavior of individual heads, e.g., whether a specific head is dominant for some k and reduces or increases for k+1, is not tracked. Consequently, the two interpretations offered in Section 5 (reduced commitment vs. parallel composition) are left unresolved.

  3. Ablation. The ablation offers statistically significant behavior for the full set of models and tasks (ρ = 0.59, p < 0.001), but not if the Qwen2.5 ≤7B models are excluded (ρ = 0.18, p = 0.14) or within model families (ρ = 0.13, p = 0.17).

  4. Performance with k ≥ 3 does not exceed random. At k ≥ 3, for 16 of the 17 models, the mean accuracy over the four serial tasks is in the range between 0.15–0.24, compared to a mean accuracy of 0.23 achieved using random answers (Appendix B, Table 3), indicating failure in most of the longer tasks.

  5. Performance differs if additional heads are considered. The paper reports that Qwen2.5 and Llama/OLMo-2 behave consistently across the four tasks. This cross-task consistency, however, is shown for heads at the top-10% share and for a single-head share, but breaks at the top-25% and top-50 % share. For these statistics, only the two smallest Qwen models retain consistency across all tasks. The claim should therefore be restated at that narrower scope.

Questions for authors
  1. My main criticism involves the lack of use of the computed profiles. Can you demonstrate a use, e.g., predicting behavior on a held-out task?

  2. Can you demonstrate a k-dependent causal relation? For example, does ablating the layers or heads that spread most for a choice of k affect accuracy at larger k more than at smaller k?

1 comment

Thank you for the clear summary and the concrete suggestions. Both questions led to new analyses on existing captures; their code is in the anonymized supplementary material.

Q1 and Weakness 1: a use, predicting behavior on a held-out task. A model's top-10% share, measured on three tasks, predicts how much its answer depends on its most active heads on the held-out fourth task: Spearman ρ across models averages 0.59 over the four held-out tasks (0.76, 0.32, 0.69 and 0.57; permutation p < 0.001). This uses the level of concentration rather than its change with k, and because a model's share barely varies across tasks it is close to the within-task association of Section 4.4. Like that association, it rests mainly on the contrast between the Qwen2.5 models up to 7B and the other families (ρ = 0.17 without them). The use it points to is modest but concrete: forward passes alone may help anticipate whether a few heads carry a new task's answer, and so whether a sparse-circuit analysis will find them. The location of a model's response also replicates on fresh items: the decile of strongest spreading recurs in seven of the eight rerun models (2.5 expected, p = 0.004). We agree that a use of the change with k itself remains to be shown.

Q2 and Weakness 3: a k-dependent causal relation. We could test this only in part, and the answer is not yet conclusive. The 17-model ablation removed each layer's most active heads, separately in five 20% depth bands, and measured the answer margin; we use the margin rather than accuracy because accuracy at larger k is near chance and cannot fall further. Only in binding do most models (14 of 17) stay at least 15 points above chance at k = 2. There, removing the most active heads of the 60–80% band, where binding spreads most, costs more at k = 2 than at k = 1, beyond random heads, in 12 of 14 models (+0.19 logits per step, 95% CI [0.01, 0.38]), while the same removal in the first 40% of depth shows no change. The effect, however, comes mainly from the four OLMo-2 checkpoints: counting each lineage once it is +0.14 [−0.04, 0.31], and without OLMo-2 +0.05 [−0.10, 0.18]. We therefore treat it as exploratory. The direct test, ablating heads selected by their own response to k on held-out items and comparing the cost across k with that of equally active heads, is the next experiment.

Weakness 2: tracking individual heads. We added this analysis: each head's share of its layer's activity, regressed on k and standardized against a permutation null. In back-half layers, 66% of heads change their share significantly with k (about 5% expected), so the redistribution involves many heads, not only the top 10%. The heads that respond most are mostly not the most active ones, although the overlap exceeds chance (17% per layer, against 12.5%). On the two readings in Section 5: with items matched on correctness and answer entropy, about 60% of the back-half spreading remains (our response to Reviewer 5Bu5, Q2), so a falling commitment to the final answer, as measured by its entropy, does not account for all of it. Separating the rest from parallel composition needs interventions on intermediate states, which we have not done.

Weakness 4: accuracy near chance at k ≥ 3. The response is strongest at small k, where accuracy is highest: the back-half share falls 0.29 percentage points per step between k = 1 and 2 (95% CI [0.17, 0.42]), against 0.10 over the full range, and 0.12 per step over k ≤ 3 when items are matched on correctness and answer entropy. In binding, for instance, accuracy is 0.86, 0.61 and 0.44 at k = 1, 2 and 3, against 0.33 by chance. At a given k, head activity is nearly the same for correct and incorrect answers (Appendix B), and signatures computed from incorrect answers alone correlate 0.79 with those from all items. We will bring this into the main text.

Weakness 5: dependence on the statistic. Agreed; we will restate the claim at that scope. The same-direction pattern holds under the top-10% share (seven models) and the strongest-head share (nine). Under the top-25% and top-50% shares and the effective number of heads, only Qwen2.5-0.5B and 1.5B deviate in the same direction in every task under all three statistics. The relational separation of the Llama and Qwen2.5 models depends on the statistic the other way: it holds under the top-25% and top-50% shares and the effective number of heads, but not under the strongest-head share (p = 0.086).

Reviewer_5Bu5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study investigates how language models shift the work done by their attention heads as tasks demand more sequential reasoning steps. redistribute attention-head activity as tasks require increasingly many serial reasoning steps. The authors evaluate 17 language models across four synthetic tasks (pointer chasing, modular arithmetic, variable binding, and relational lookup), measuring how the share of attention-head output norms carried by the most active heads changes with task depth. Models show distinct, repeatable patterns of concentrating or dispersing activity across layers. Llama and OLMo models tend to spread activity more widely in later layers than the smaller Qwen2.5 models do. These patterns hold up partly from one task to the next and mostly survive post-training. The authors further note links between how activity is redistributed and task accuracy, and their ablation experiments indicate that models with more concentrated activity rely more heavily on their most active heads.

Critical strengths
  • The paper systematically compares 17 language models across four controlled synthetic tasks, covering multiple model families, sizes, and post-training stages.

  • The authors validate their findings through independent replications on newly generated examples, alternative concentration measures, permutation tests, and ablation experiments. These analyses strengthen confidence that the observed patterns are reproducible rather than artifacts of particular experimental choices.

Critical weaknesses
  • Serial demand is not fully disentangled from task difficulty and uncertainty. As the required number of steps increases, average accuracy rapidly approaches chance and answer uncertainty grows. Adjusting for answer entropy reduces the average back-half spreading effect by approximately 45%. Although spreading persists after this adjustment and correctness controls leave the signatures largely unchanged, the numeral control is not difficulty-matched and exhibits a different uncertainty trajectory. Consequently, the experiments do not establish how much of the observed redistribution is specific to serial computation beyond its associated difficulty and uncertainty.

  • The functional significance of the observed redistribution remains unclear. The proposed statistic measures the relative norms of attention-head outputs, which do not directly identify their contribution to predictions or the computations they implement. The shared average change is small—approximately 0.10 percentage points per additional step—and its behavioral significance is not established. The ablations primarily examine dependence on highly active heads, rather than the causal role of changes in concentration as serial demand increases. Moreover, the association between concentration and ablation sensitivity is largely driven by contrasts involving Qwen models and weakens substantially, becoming statistically nonsignificant, after removing family differences. These findings provide some functional grounding for concentration levels, but limited mechanistic explanation of their changes with serial demand.

  • The model-specific signatures have limited generality across tasks and prompts. Although the signatures are reproducible across item samples, the corrected variance decomposition attributes approximately 67% of non-shared variation to model–task interactions, and the average correlation between signatures estimated from complementary task halves is only 0.36. Transfer is particularly weak between tasks that explicitly state the step count and those that do not. Prompt wording also affects individual model–task profiles substantially, although task-averaged signatures remain more consistent. Equal-head-count and approximately size-matched comparisons provide partial controls, but architectural, training, and performance differences still limit explanations of the observed model contrasts. The evidence therefore supports reproducible profiles within the tested settings more strongly than broadly task-independent computational signatures.

  • I could not find a link to the code in the paper, which makes the experiments harder to reproduce and the correctness of the implementation difficult to assess. This negatively affects my score.

Questions for authors
  1. Appendix F reports that eager attention and SDPA agree on only 52–55% of predictions for the Qwen2.5-7B checkpoints, and that the measured spreading response approximately doubles under SDPA in the tested base-model condition. Do the principal family contrasts and post-training comparisons persist when these checkpoints are evaluated consistently with SDPA across all tasks? Please also clarify which models and conditions the 14-model SDPA robustness analysis covers.

  2. Can the authors provide a comparison that varies compositional demand while approximately matching accuracy and answer uncertainty, or independently increases difficulty without adding composition? Does the model-specific redistribution persist in this comparison?

  3. Can the authors show that intervening on heads or layers whose activity distribution changes with k affects the model’s ability to resolve additional steps, beyond the effects of intervening on generally active heads? In particular, is there evidence connecting the redistribution itself to a specific intermediate computation or behavioral consequence?

3 comments
AuthorsOpenReview ↗

Reviewer 5Bu5, part 1 of 2

Thank you for the close reading, particularly of the kernel and difficulty controls. We answer your three questions with new analyses on existing captures; their code is in the anonymized supplementary material.

Code. The supplementary material now contains the task generators, the capture and ablation code, the Colab notebooks, the full analysis pipeline, the figure scripts, and the scripts for the analyses below. The raw captures are not included because of their size; the README describes their layout.

Q1: all models under SDPA. The 17-model ablation run (Appendix G) recorded an unablated forward pass of every model under SDPA on the same test items as the main run (wording w0). We recomputed the statistic from these passes, with both Qwen2.5-7B checkpoints under SDPA in all four tasks, and compared it with the main run on the same items and wording.

Qwen2.5-7B spreads more under SDPA in every task. Its back-half mean z is −3.05 against −1.53 under eager attention in pointer chasing, −0.12 against +0.82 in arithmetic, −0.94 against −0.54 in binding and −0.45 against +0.20 in relational lookup, as the single-cell check in Appendix F suggested.

The family contrast holds. The three Llama base models separate completely from the four Qwen2.5 base models up to 7B in arithmetic, binding and relational lookup (exact p = 1/35 in each), but not in pointer chasing, exactly as under the paper's kernels. Six models deviate from the average model in the same direction in all four tasks under both kernels, the same six (2.0 and 2.4 expected by chance; p = 0.012 and 0.024). With one wording, Qwen2.5-7B is not among them under either kernel, because it spreads more than average in pointer chasing. We will therefore restrict "less than the average model in every task" to Qwen2.5-0.5B, 1.5B and 3B, and state three of four tasks for Qwen2.5-7B.

Post-training: with both Qwen2.5-7B checkpoints under SDPA, the Instruct model still spreads more than its base, but by less (−0.45, against −1.32 with mixed kernels on the same items). This agrees with the replication on fresh items, which ran both checkpoints under one kernel (−0.46). The other lineages do not change with the kernel: −0.35 against −0.34 for Llama-3.2-3B, −0.94 against −0.95 for Llama-3.1-8B, and +0.01 against +0.03 for OLMo-2.

For the 16 models whose main capture used eager attention, the per-layer profiles under the two kernels correlate r ≥ 0.98 in 13; the exceptions are Qwen2.5-0.5B, 1.5B and 7B (0.92, 0.90 and 0.76).

The 14-model SDPA analysis in Table 7 covers all models except the base models Llama-3.1-8B, Qwen2.5-7B and OLMo-2-1124-7B, which the main run captured under SDPA only in pointer chasing with wording w0 (the kernel check). For the 14, both kernels were captured on the same items in all five task families and both wordings, and their signatures correlate 0.98.

AuthorsOpenReview ↗

Reviewer 5Bu5, part 2 of 2

Q2: serial demand at matched difficulty. The outcome is each item's top-10% share averaged over back-half layers. We report its slope on k in percentage points (pp) per step, averaged over models, with 95% bootstrap intervals.

Varying k at matched accuracy and uncertainty. We compare items only within strata that agree on strict correctness and on the quintile of answer entropy over the admissible digits (a regression with stratum fixed effects). The share still falls: −0.059 pp per step [−0.081, −0.034], against −0.100 without matching (computed the same way), and it falls in 14 of 17 models (−0.055 with entropy deciles). The response is strongest at small k, where accuracy is highest: matched in the same way, the share falls 0.12 pp per step over k ≤ 3 [0.06, 0.19].

Raising difficulty without adding composition. In the numeral control, accuracy falls from 0.41 to 0.03 over k and answer entropy rises, but the back-half share does not fall: −0.013 pp per step unmatched and 0.000 [−0.047, 0.040] matched. Per model, the matched slope is lower in the serial tasks than in the control by 0.058 pp per step [0.012, 0.103] (13 of 17 models). As you note, the control is only a partial match: its entropy rises less than in the serial tasks, and at k ≤ 3 it also spreads somewhat (−0.08 against −0.20 pp per step). This is why we added the matched comparison.

The model-specific pattern survives the matching: matched and unmatched slopes correlate r = 0.85 across models, and the separation of the Llama and Qwen2.5 base models in arithmetic and relational lookup holds after matching. Together with Table 7, where signatures adjusted for entropy and correctness correlate 0.86 with the primary ones, this suggests that about 60% of the shared spreading, and most of the model differences, are not explained by correctness or by answer entropy (matched to quintiles). We will report this in Section 4.6 alongside the covariate result.

Q3: intervening on the layers and heads that redistribute. The existing runs answer this only in part, and not yet conclusively. The 17-model ablation removed each layer's most active heads, separately in five 20% depth bands, and measured the answer margin; we use the margin rather than accuracy, which at larger k is near chance and cannot fall further. Only in binding do most models (14 of 17) stay at least 15 points above chance at k = 2, so binding is the one task where a change with k can be measured. There, removing the most active heads of the 60–80% band, where binding spreads most, costs more at k = 2 than at k = 1, beyond as many random heads, in 12 of 14 models (+0.19 logits per step, [0.01, 0.38]); the same removal in the first 40% of depth shows no change with k. But the effect comes mainly from the four OLMo-2 checkpoints: counting each lineage once it is +0.14 [−0.04, 0.31], and without OLMo-2 +0.05 [−0.10, 0.18]. It is also one of several post hoc comparisons (six depth sets, two margin measures; as a fraction of the clean margin it is +0.08 [−0.005, 0.16]), so we treat it as exploratory. Two further limits apply. The heads were chosen by activity, and the heads that respond most to k are mostly different ones (17% overlap per layer, against 12.5% by chance; see our response to Reviewer MrzM). And we have not tied the redistribution to a specific intermediate computation, which would need interventions at earlier token positions. The direct test, ablating heads selected by their response to k on held-out items, is the next experiment.

Weaknesses 2 and 3: effect size and generality. The 0.10 pp per step averages over all models, tasks and values of k; between k = 1 and 2 it is 0.29, and single models reach 0.43 pp per step in single tasks. We agree that its size does not establish behavioral significance, and that the ablation association in Section 4.4 is mainly a contrast between families. We also agree that the profiles are reproducible within a task family more strongly than across tasks: prompt wording matters more than item sampling (single-task profiles agree across wordings at r = 0.54, against 0.84 across item halves), and transfer between tasks that state the step count and those that do not is weak (r = 0.03–0.20). We will scope the claims accordingly and name the contrast that holds across tasks, kernels and scale: the Llama and Qwen2.5 contrast in arithmetic and relational lookup, which holds under both kernels and in the 70B pair, which has the same number of layers and heads.

I thank the authors for their detailed rebuttal and the additional experiments.

The new experiments with the same attention kernel address my first concern. The analysis controlling for correctness and answer uncertainty is also useful, although I am not fully convinced that it separates serial demand from task difficulty.

Regarding my second concern, I still find the functional significance of the results unclear. In particular, the authors have not tested what happens when ablating the heads whose activity changes with k. The additional ablation results are interesting, but remain inconclusive. Similarly, the held-out-task experiment considers the level of concentration rather than how it changes with k, and its results seem to be largely driven by the Qwen models.

I also appreciate that the authors have acknowledged the limited generality of the signatures across tasks and have agreed to narrow some of their claims.

Overall, I think the rebuttal has strengthened the empirical results. However, my main concern remains that the paper is primarily observational, without clearly showing what the observed differences tell us about how models perform computations. I therefore decided to maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: new analyses and anonymized code

We thank the three reviewers for careful and constructive reviews. All three found the results reproducible; all three asked what the redistribution means for the computation, and two asked whether it reflects difficulty rather than serial demand. Since the reviews were posted we have run new analyses, all on captures we already had. The anonymized code, including the scripts for these analyses, is now in the supplementary material.

  1. One attention kernel for all models (5Bu5, Q1). The 17-model ablation run recorded an unablated forward pass of every model under SDPA, on the same items as the main run. With every model under SDPA, Qwen2.5-7B spreads more than under eager attention in all four tasks, and the three Llama base models still separate completely from the four Qwen2.5 base models up to 7B in arithmetic, binding and relational lookup (exact p = 1/35 each), as under the paper's kernels.

  2. Spreading at matched accuracy and uncertainty (5Bu5, Q2; 4dQU). Comparing only items that agree on correctness and on answer-entropy quintile, the back-half top-10% share still falls with k: −0.059 percentage points per step (95% CI [−0.081, −0.034]), against −0.100 without matching, and it falls in 14 of 17 models. In the numeral control, where difficulty rises with k but composition does not, it does not fall (0.000, [−0.047, 0.040]).

  3. Heads ranked by their own response to k (MrzM, Q1; 4dQU). In back-half layers, 66% of heads change their share of the layer's activity significantly with k (about 5% expected by chance), and the most responsive heads are mostly not the most active ones (17% overlap per layer, against 12.5% by chance).

  4. A held-out-task check (4dQU, Q1). A model's top-10% share measured on three tasks predicts how much its answer depends on its most active heads on the fourth (Spearman ρ = 0.59 averaged over held-out tasks, p < 0.001). Because the share barely varies across tasks, this is close to the within-task association of Section 4.4 and, like it, rests mainly on the Qwen2.5 contrast.

What we could not yet show is a k-dependent causal effect (4dQU, Q2; 5Bu5, Q3; MrzM, Q1). The existing ablation runs were not designed for it: they removed heads chosen by activity, and only in binding do most models stay well above chance at k = 2. In binding, removing the most active heads of the 60–80% band costs more at k = 2 than at k = 1 in 12 of 14 models, but the effect comes mainly from the OLMo-2 checkpoints and is not significant when each lineage counts once. We treat it as exploratory; the direct test, an ablation of heads selected by their response to k, is the next experiment.

Three corrections of scope follow, and we will make them in the paper. (i) Under SDPA, and with one wording under either kernel, Qwen2.5-7B spreads less than the average model in three of the four tasks, not all four; under the top-10% share, "in every task" holds for Qwen2.5-0.5B, 1.5B and 3B. (ii) Qwen2.5-7B's post-training shift is smaller with both checkpoints under one kernel (−0.45 on the same items and wording, against −1.32 with mixed kernels), though in the same direction. (iii) The component of the profiles shared across tasks is modest, so we will describe the profiles as reproducible within a task family and name the specific contrasts that hold across tasks. Two reviewers also found the paper dense; the revision will lead each section with its finding in words, define the measure before its formula, and move most statistics to the appendices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.