- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
With an inspiration from neuroscience, this study focus on how the same attention heads in a model work differently when different tasks are being processed. They use small 3-node graphs and keep the facts fixed while they change the tasks. They investigate the effect of nudging an attention head or ablating them to mean. They claim that same heads act differently under different tasks, the task can be switched from the internal states, and task states can help predicting responses.
Critical strengths
-
The use of opposite-answer graphs makes the claims and the experiments very clear.
-
Controls are strong (for magnitude). Multiple models have been tested. Weak results are also well reported.
Critical weaknesses
Further details of potential experiments that would address these concerns are listed in the “Questions for Authors”
- When the instruction changes the influence may also change regardless of the head’s role, there is no control to check if that plays a part in the results. One example could be paraphrasing conditions and noise floors for the angles reported.
- In the comparison of influence and importance the two profiles do not have similar size and units which makes it an unfair comparison.
- In the prediction test the map is fitted to the same prompts it predicts and the baseline is also weak.
- Copying the other instructions internal state for another task is expected to yield the same result as reading the first task. Random edits are not the best control since they do not change the behaviour and can’t rule this out.
Less importantly: 5. Natural tasks are needed to widen the setup, the current tests are too narrow. 6. The brain link is only an analogy and not further investigated.
Questions for authors
- How different are the influence profiles for different paraphrasings of a task?
- How would the results change in influence vs. importance if the sites and number of entries are the same?
- Have you tried new prompts for the prediction test?
- Would an edit that changes the behaviour but is not related to the task, like mismatched response keys, move the influence profile meaningfully?
Less importantly: 5. Could you test a non-graph task on the same wording list? 6. Have you tried other nudge sizes?
3 comments
Response key controls, counting, and perturbation size
Comment (2/2)
4. Reversing the response key
The task edit is a fixed mean-difference displacement, estimated on separate reference groups and added before layer 12. It does not copy the other instruction's state on each prompt. It switches fact-reading behavior while the written prompt stays fixed, allowing us to measure what the same heads then respond to. For the behavior-changing control, we reversed the internal A/B mapping at layer 12 while keeping the written prompt, graph, and requested fact fixed. This key edit flipped the answer on 512/512 opposite-answer prompts per model, as did the task edit, with mean margin drops matched to within 0.05 logits. The key edit used a longer span and a larger displacement than the task edit.
The key edit produced little change in the factual-response shares. Within either written key, it changed the mean share of squared head response to the alternative fact by at most 1.9 percentage points at H2 or H3. The task-minus-key differences at H2 and H3 were 79.5 and 97.2 points in Gemma and 68.1 and 92.3 in Qwen. All four simultaneous 95% intervals exclude zero.
The task edit moved the influence profile further toward the alternative-task profile. The key edit moved the six-link profile by 0.03734 in Gemma and 0.00577 in Qwen (L2 distance). On the opposite-answer prompts, the task edit reduced the squared distance to the unedited alternative-task profile by 92.4% in Gemma and 78.6% in Qwen. The key edit increased that distance to 27.7 times baseline in Gemma and reduced it by 51.5% in Qwen. The simultaneous 95% intervals for the task-minus-key difference were above zero in both models.
5. Counting with the same vocabulary
The counting task used the same names, presence terms, and A/B keys, and asked whether at least two of three listed items were present. Its paired control supplied the result while keeping the list, key, and required answer fixed. In Qwen, replacing each original head individually with its graph-calibrated mean left counting accuracy at 100% on 512 held-out pairs. The nudge effect on the True-minus-False logit gap was 0.1163 logits larger when the corresponding item could change the counting answer. The same split in the control gave 0.0148 logits (paired difference 0.1015, exploratory 95% CI [0.0784, 0.1237]).
6. Other nudge sizes
The Figure 2 test was repeated at 0.5, 1, 2, and 4 reference SD using the original prompts and sites. The task contrast is the difference between the two task influence profiles. The table gives its cosine with the 2 SD contrast from the same run. Each cell lists the results at 0.5, 1, and 4 SD, in that order.
| Model | Cosine to 2 SD |
|---|---|
| Gemma-9B | 0.999 / 1.000 / 1.000 |
| Qwen3-14B | 0.815 / 0.903 / 0.971 |
| Llama-8B | −0.401 / 0.416 / 0.249 |
| Falcon-7B | 0.463 / 0.899 / 0.962 |
| Falcon-1.5B | 0.889 / 0.879 / 0.997 |
Bold cosines have simultaneous 95% intervals above zero. In Gemma, Qwen, and both Falcon models, the point-estimate cosines at 1 and 4 SD ranged from 0.879 to 1.000. All 432 repeated blocks gave identical profiles.
The link to brain networks. Brain networks motivate the network view and the choice of measurements. Our empirical conclusions concern the language models tested here.
The revised Figure 4 results include the key-edit control, with details in Appendix D.5. Appendices H and B.7 describe the counting and nudge-size tests.
Thank you for running the requested analyses. The response-key control in particular is convincing for the factual-response shares.
- Prediction (E.4, Table S12). On new prompts, the transferred response map (the method described in Section 2.5) has higher error than a pooled map that ignores the task (0.0269 vs. 0.0177 in Gemma), and the label baseline is best in both models. The 21.4% / 2.8% figures come from ridge maps fitted directly to the factual interventions, not from the influence maps. The abstract and Section 3.4 still report only the within-prompt 53.6% / 5.0%.
- Influence vs. importance (B.4, Table S1). With matched sites the gap disappears in Gemma, and only 83/120 trios are positive. The main text cites the 749/749 result, which uses the unmatched profiles where the residual site makes up ~98–99.95% of importance.
- Paraphrases (B.6). By angle, task changes exceed rewording only in Gemma, which matters for the "redistribution" claim.
- Key edit and influence (Table S8). In Gemma the key edit moves the influence profile about 6× more than the task edit, and in Qwen it covers about half the distance to the target. Only the factual-share result appears in the main text.
The new experiments strengthen the factual-response findings, but they also show that the influence and prediction claims are weaker than currently stated, so I'm not increasing my score. For the paper to be sound as presented, the abstract, Figure 2C/Section 3.1, and Section 3.4 should be updated to reflect these results rather than leaving them only in the appendix.
Figure 2 robustness and prediction on new prompts
Comment (1/2)
We thank the reviewer for suggesting these validation and robustness checks.
1. Paraphrases and angle reliability
We reworded the task instruction line in four ways, keeping the task name fixed. With graphs and response keys matched, we compared 12 wording pairs within tasks and 12 across tasks. Each model used 16 vocabulary groups, except Falcon-1.5B, which used eight.
The first column of results gives the mean squared distance between influence profiles within and between tasks. The gap is the between-task mean minus the within-task mean, with simultaneous 95% intervals across models. The last column gives split-half median angles for the same task and wording and for different tasks, using equally sized, disjoint vocabulary halves.
By squared distance, task changes exceed rewording in every model. With the same wording pairs in both comparisons, the squared distance between profiles for different tasks was 1.7 to 7.5 times that for one task. Every gap interval is above zero. By angle, the gap was 23.55° [18.83, 27.88] in Gemma; in the other models its interval included zero. The median split-half angle between tasks exceeded the same-wording split-half angle in every model. Repeating identical inputs gave exactly the same profiles.
2. Matching the sites and the number of entries
We restricted influence to the three links among H1–H3 and importance to the mean-replacement effects at those same heads, so both profiles have three entries from the same sites. Appendix B.4 reports this comparison for the original trios, whose pointwise 95% intervals all included zero. We repeated it for all 120 eligible Figure 2C trios. The gap is the influence angle minus the importance angle. The median intervals use paired vocabulary resampling with adjustment across the four models.
The median gap is positive in all four models. Its interval is above zero in Qwen, Llama, and Falcon-7B, and the gap is positive in 83 of 120 trios. A uniform positive rescaling of either profile leaves its between-task angle unchanged. The residual accounts for 97.8–99.95% of the squared norm in the original four-site importance profiles.
3. Prediction on new prompts
We fit ridge maps on 32 vocabulary groups, selected their settings on 16 validation groups, and fixed them before testing on 64 new groups per model. Each prediction used the measured H2 displacement on the test prompt. As a stronger baseline, we fit and calibrated a pooled map on the same data. The task-specific maps reduced mean squared error relative to this baseline by 21.4% in Gemma and 2.8% in Qwen.
The lowest error came from the label baseline, which lets the map vary with the task label and pre-intervention H2 state. It had the same inputs and parameter count as the task-specific map. Its error was 0.00534 in Gemma and 0.09372 in Qwen, against 0.01766 and 0.10307 for the pooled map. It also outperformed the task-specific map in both models. Both context-dependent predictors improved on the pooled map on new prompts.
The revised appendix includes the head-only comparison in B.4, the paraphrase comparison in B.6, and prediction on new prompts in E.4.