- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that LLMs under-use in-context knowledge because attention is misallocated. Comparing each prompt region's attention with its gradient saliency, it finds that the system region takes over 60% of attention but contributes little, while the knowledge region takes about 16% yet drives predictions in mid-to-late layers. KASR therefore moves part of the attention on the top system tokens to knowledge tokens, in layers selected on a validation set. It improves over the base model for all 10 models on five benchmarks, by 3.83 points on average over the strongest competitor.
Critical strengths
- KASR is training-free, needs no extra forward pass, and improves on the base model in all 50 settings of Table 5 (about +6 points on average).
- The attention, saliency and intervention studies span many models, layers and heads. Some findings, such as the collapse caused by fully masking system attention (Figure 6), are useful in practice.
Critical weaknesses
- The main finding is largely the known attention sink. <|im_start|> alone takes 0.32 of the attention (Table 7). Eq. 2 ignores the sum-to-one constraint and thus understates sink tokens. Masking system attention costs 17.6–21.8 points (Figures 25 and 30), so the system region is far from marginal.
- The gains are not shown to come from knowledge. Amplifying system (+5.44) or query (+3.77) attention helps as much as amplifying knowledge attention (+4.18; Figure 26). MMLU, which has no knowledge region, also improves by 4.4–6.5 points. A generic sink-weakening effect explains these results equally well.
- The evaluation is weak. The test sets are tiny (103 DXY and 31 MMLU-Medicine questions) and untested for significance. The MMLU test and validation splits appear swapped (Table 4), and the value of β in the main experiments is not reported. ACT and PASTA fall to near chance on GMD (7.11 and 8.37), which suggests misconfigured baselines.
- The motivating setting is untested. No knowledge-conflict or noisy-retrieval benchmark is evaluated, although this is where boosting context attention unconditionally is most likely to hurt.
Questions for authors
- Does the imbalance hold without the first token, or with a saliency that respects the sum-to-one constraint?
- Which MMLU split was used for testing, and how do the results change on the full official test splits with significance tests?
- How does KASR compare with redirecting the same attention to random or query tokens, and how does it behave with counterfactual knowledge (e.g., NQ-Swap)?
- How were ACT and PASTA configured, given their near-chance results on GMD?
6 comments
R-W2: It is important to clarify that KASR focuses on identifying redundant attention within the System region that can be redistributed, rather than assuming that the entire System region is useless or attempting to completely suppress its attention. The System region contains important information, such as task instructions, role definitions, and output constraints. Therefore, appropriately increasing attention to this region may help models better follow instructions and improve generation quality. However, this does not mean that redundant attention does not exist within the region. The overall importance of a region and the appropriateness of its internal attention allocation are two distinct issues. KASR aims to identify and redistribute relatively low-saliency attention, rather than simply reducing the overall importance of a particular functional region.
As shown in Figures 25–34, although increasing attention to the System, Query, and Knowledge regions can all potentially improve model performance, interventions on different regions exhibit different levels of stability. In particular, increasing System attention can also lead to performance degradation under certain intervention settings, whereas increasing Knowledge attention more frequently improves performance. This suggests that the effectiveness of attention interventions depends not only on the overall importance of the target region but also on the specific intervention layers, token selection, and their relevance to prediction. Therefore, KASR does not simply transfer attention from one region to another. Instead, it employs saliency-guided selection to identify positions that are more suitable for attention redistribution. Furthermore, the experiments jointly increasing attention to the Knowledge and Query regions in Section 4.3 also demonstrate the potential value of coordinated attention adjustment across different task-relevant regions.
We acknowledge the phenomenon observed by the reviewer in the MMLU experiments. Although MMLU does not contain an explicit external Knowledge region, KASR still achieves performance improvements of 4.4–6.5 percentage points. This does not directly demonstrate improved external knowledge utilization; rather, it indicates that KASR's underlying mechanism is not dependent on any specific input region. Its core principle is to identify redundant attention allocation across different functional input regions based on Attention–Saliency Imbalance and adjust it accordingly. In knowledge-augmented tasks, we focus on redistributing redundant attention from the System region to the Knowledge region. In other tasks, the same mechanism can also be applied to adjust attention allocation across other task-relevant regions. This further demonstrates the potential applicability of KASR as a general-purpose attention analysis and intervention mechanism.
R-W3:
a. We thank the reviewer for pointing out the issues regarding test set size and statistical reliability. We agree that the test sets of DXY and MMLU-Medicine are relatively small. We conducted a significance test comparing the Base and KASR results on MMLU-MED using Llama 3.1-8B and obtained p=0.125. This indicates that although KASR achieves a notable performance improvement on MMLU-MED, the current sample size is insufficient to provide strong statistical evidence. We will clarify this limitation in the revised manuscript.
In addition, although the initial manuscript reported results on datasets with relatively larger numbers of samples, including DXY, GMD, MMLU-LAW, and StrategyQA, we further incorporated NeQA and the moral_scenarios and professional_psychology subsets of MMLU to address the limited evaluation data. The dataset statistics are presented in Table 2, and the results are reported in Table 3. KASR achieves strong performance across these datasets, providing further evidence of its effectiveness.
Table 2. Statistics of the additional datasets
| Dataset | Language | #Val | #Test |
|---|---|---|---|
| MMLU-moral_scenarios | English | 895 | 100 |
| MMLU-professional_psychology | English | 612 | 69 |
| NeQA | English | 398 | 53 |
Table 3. KASR performance on the additional datasets
| Model | Method | moral_scenarios | professional_psychology |
|---|---|---|---|
| Qwen2.5-3B | Base | 37.00 | 69.57 |
| KASR (Ours) | 43.00 | 73.91 |
| Model | Method | NeQA |
|---|---|---|
| Qwen1.5-8B | Base | 39.62 |
| KASR (Ours) | 49.06 |
b. We used two subsets of the MMLU dataset. Since we needed to probe the contributions of different input regions to the final predictions on the validation set, and the official validation split contains too few samples to support stable saliency probing and layer selection, we used the official test split for these analyses and development, while reserving the official validation split for final performance evaluation. We will clarify this procedure in the revised manuscript.
c. Regarding the setting of , although Section 4.3 already analyzes the effects of different values through ablation experiments, we indeed omitted its specific selection procedure from the main experimental settings. Since the influence of knowledge context on model predictions may vary across tasks, we first evaluated three candidate values of on the validation set, namely , and selected the best-performing value for evaluation on the test set.
d. We described the settings of ACT and PASTA in Sections 4.1 and B.0.1, adopting the same settings as those in the original papers.
ACT identifies sink tokens whose attention ratios exceed (5/N) and redistributes 60% of their attention to the remaining tokens in the selected attention heads. It consists of two stages:
- Profiling: ACT is applied to each attention head individually on the training set. The accuracy following each intervention is recorded, and all heads that yield an accuracy higher than the baseline are selected.
- Testing: ACT is simultaneously applied to all selected attention heads, and the resulting performance is evaluated on the test set.
The effectiveness of intervening on individual heads does not necessarily imply that simultaneous intervention on multiple heads will also be effective. Since ACT does not constrain the attention components or input regions being modified, it substantially alters the model's attention patterns, which may explain its limited performance on many tasks. We also discuss this issue in Section 4.2.
PASTA reallocates attention from non-instruction tokens to instruction tokens. Similar to ACT, it consists of two stages. First, attention redistribution is individually applied to different attention heads on the validation set, and the top 1% of heads are retained for evaluation on the test set. Since PASTA modifies a relatively smaller amount of attention and does not substantially disrupt the model's structure, it performs better than ACT.
We re-evaluated all implementations. The ACT results were verified to be correct. For PASTA, the implementation on the InternLM2.5-1.8B model with the GMD dataset was also correct, but an issue with the execution environment affected the results. The correct result is 48.95, and we have corrected it accordingly.
Thank you for taking the time and effort to review our manuscript and for providing valuable suggestions to improve its quality. We will address your comments point by point.
R-W1: a. As shown in Table 7, the initial token (
<|im_start|>) indeed receives a relatively high attention weight. However, other tokens in the System region, beyond the initial token, also receive substantial attention. To verify that the Attention–Saliency Imbalance is not limited to the initial token, we calculated the attention and saliency of each token in the System and Knowledge regions with respect to the target prediction token, as well as the ratio of saliency share to attention share. The formula for this ratio is given in Eq. (1). Taking Qwen1.5-4B on the GMD dataset as an example, we found that, apart from<|im_start|>(attention: 0.2229), which has relatively low saliency (0.0044) and a saliency-to-attention share ratio (0.7674), other tokens in the System Region, such as\n, also exhibit low ratios (0.5642). In contrast, although the maximum attention value of tokens in the Knowledge region is only 0.00049, the saliency-to-attention share ratio of every token exceeds 1.5. This indicates that their contributions substantially exceed the attention allocated to them.Furthermore, to investigate the impact of the initial token on model performance, we conducted an additional experiment using Qwen1.5-1.8B on the DXY dataset, in which we redistributed only the attention allocated to the initial token to the Knowledge region. Although this intervention improved performance over the Base model, the accuracy reached only 55.34%, remaining below that of KASR. This indicates that modifying the attention of the initial token alone is insufficient to reproduce KASR's performance gains, and that attention allocation to other tokens in the System region also deserves consideration. Therefore, our study does not focus solely on the attention sink associated with the initial token, but rather on the broader attention allocation imbalance across different functional input regions.
SARi(N)=∑j∈IallAN,jAN,i∑j∈IallSN,jSN,i(1)
b. We clarify that Eq. (2), based on a first-order Taylor expansion, defines the saliency of each element in the Attention Matrix. This metric estimates the first-order contribution of an individual attention connection to the change in loss, rather than representing an attention probability. Therefore, unlike Softmax-normalized attention weights, saliency values are not required to sum to 1. Accordingly, Eq. (4) in our manuscript measures the contributions of different input regions. For a given target prediction token, the normalized contribution ratios across all input regions sum to 1.
c. As explained in Section 3.2.2, masking all input functional regions severely disrupts the model's input structure and introduces additional biases, leading to substantial performance degradation. We believe that the System region contains important task instructions and structural information. Therefore, completely masking its attention would severely disrupt the original input structure and result in a significant drop in model performance. However, this does not imply that the System region contains no redundant attention. Rather than removing the entire System region, our goal is to identify relatively low-value attention allocations within it and redistribute them to the Knowledge region.
Furthermore, the additional experiments in Figures 25–34, which involve reducing and increasing attention in different functional regions, provide further evidence. Reducing attention to the Knowledge region more frequently leads to performance degradation. In the middle and later layers, increasing attention to the Knowledge region yields more substantial performance gains than increasing attention to the System region. These results are consistent with our analysis that the Knowledge region receives insufficient attention, while part of the attention allocated to the System region can be redistributed.