acceptodds
Under review as a conference paper at ICLR 2027

Knowledge-Aware Attention Steering and Redistribution

Abstract

Large language models often fail to faithfully utilize the knowledge provided in context, producing answers that contradict the given evidence. To bridge this gap, we systematically investigate the attention allocation across different functional input regions and its causal contribution to information propagation and final predictions via information-flow analysis and extensive attention interventions. We trace this unfaithfulness to a striking mechanistic root: attention heavily goes where information is not. Through large-scale analysis, we reveal three core mechanistic insights: (1) Attention–Saliency Imbalance. The system region absorbs over 60% of attention (up to 80% in LLaMA) yet contributes marginally to prediction, whereas the knowledge region drives information fusion and prediction in mid-to-late layers with merely 16% of attention. (2) Redundant Attention Bias. A substantial portion of system attention functions as a structural bias rather than informative computation, and scaling it down is harmless and even yields performance gains. (3) Attentional Alignment. Both causal interventions and supervised fine-tuning trajectories confirm that elevating the knowledge attention in mid-to-late layers causally improves knowledge utilization and accuracy. Motivated by these findings, we propose Knowledge-Aware Attention Steering and Redistribution (KASR). During inference, KASR dynamically intercepts the redundant attention from the system region at critical layers and redistributes it to knowledge tokens, thereby amplifying the latent factual signals. KASR is training-free and serves as a plug-and-play inference approach. Extensive experiments on 10 models across 5 knowledge-intensive benchmarks demonstrate that KASR yields substantial and consistent improvements, raising accuracy by 4.18% on average with negligible overhead.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Model, Knowledge-Aware Attention Redistribution

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qKNZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that LLMs under-use in-context knowledge because attention is misallocated. Comparing each prompt region's attention with its gradient saliency, it finds that the system region takes over 60% of attention but contributes little, while the knowledge region takes about 16% yet drives predictions in mid-to-late layers. KASR therefore moves part of the attention on the top system tokens to knowledge tokens, in layers selected on a validation set. It improves over the base model for all 10 models on five benchmarks, by 3.83 points on average over the strongest competitor.

Critical strengths
  1. KASR is training-free, needs no extra forward pass, and improves on the base model in all 50 settings of Table 5 (about +6 points on average).
  2. The attention, saliency and intervention studies span many models, layers and heads. Some findings, such as the collapse caused by fully masking system attention (Figure 6), are useful in practice.
Critical weaknesses
  1. The main finding is largely the known attention sink. <|im_start|> alone takes 0.32 of the attention (Table 7). Eq. 2 ignores the sum-to-one constraint and thus understates sink tokens. Masking system attention costs 17.6–21.8 points (Figures 25 and 30), so the system region is far from marginal.
  2. The gains are not shown to come from knowledge. Amplifying system (+5.44) or query (+3.77) attention helps as much as amplifying knowledge attention (+4.18; Figure 26). MMLU, which has no knowledge region, also improves by 4.4–6.5 points. A generic sink-weakening effect explains these results equally well.
  3. The evaluation is weak. The test sets are tiny (103 DXY and 31 MMLU-Medicine questions) and untested for significance. The MMLU test and validation splits appear swapped (Table 4), and the value of β in the main experiments is not reported. ACT and PASTA fall to near chance on GMD (7.11 and 8.37), which suggests misconfigured baselines.
  4. The motivating setting is untested. No knowledge-conflict or noisy-retrieval benchmark is evaluated, although this is where boosting context attention unconditionally is most likely to hurt.
Questions for authors
  1. Does the imbalance hold without the first token, or with a saliency that respects the sum-to-one constraint?
  2. Which MMLU split was used for testing, and how do the results change on the full official test splits with significance tests?
  3. How does KASR compare with redirecting the same attention to random or query tokens, and how does it behave with counterfactual knowledge (e.g., NQ-Swap)?
  4. How were ACT and PASTA configured, given their near-chance results on GMD?
6 comments

Thank you for taking the time and effort to review our manuscript and for providing valuable suggestions to improve its quality. We will address your comments point by point.

R-W1: a. As shown in Table 7, the initial token (<|im_start|>) indeed receives a relatively high attention weight. However, other tokens in the System region, beyond the initial token, also receive substantial attention. To verify that the Attention–Saliency Imbalance is not limited to the initial token, we calculated the attention and saliency of each token in the System and Knowledge regions with respect to the target prediction token, as well as the ratio of saliency share to attention share. The formula for this ratio is given in Eq. (1). Taking Qwen1.5-4B on the GMD dataset as an example, we found that, apart from <|im_start|> (attention: 0.2229), which has relatively low saliency (0.0044) and a saliency-to-attention share ratio (0.7674), other tokens in the System Region, such as \n, also exhibit low ratios (0.5642). In contrast, although the maximum attention value of tokens in the Knowledge region is only 0.00049, the saliency-to-attention share ratio of every token exceeds 1.5. This indicates that their contributions substantially exceed the attention allocated to them.

Furthermore, to investigate the impact of the initial token on model performance, we conducted an additional experiment using Qwen1.5-1.8B on the DXY dataset, in which we redistributed only the attention allocated to the initial token to the Knowledge region. Although this intervention improved performance over the Base model, the accuracy reached only 55.34%, remaining below that of KASR. This indicates that modifying the attention of the initial token alone is insufficient to reproduce KASR's performance gains, and that attention allocation to other tokens in the System region also deserves consideration. Therefore, our study does not focus solely on the attention sink associated with the initial token, but rather on the broader attention allocation imbalance across different functional input regions.

SARi(N)=SN,i∑j∈IallSN,jAN,i∑j∈IallAN,j(1){SA\\_R}_i(N) = \frac{\frac{S_{N,i}}{\sum_{j\in\mathcal{I}_{\mathrm{all}}} S_{N,j}}}{\frac{A_{N,i}}{\sum_{j\in\mathcal{I}_{\mathrm{all}}} A_{N,j}}} \qquad (1)

b. We clarify that Eq. (2), based on a first-order Taylor expansion, defines the saliency of each element in the Attention Matrix. This metric estimates the first-order contribution of an individual attention connection to the change in loss, rather than representing an attention probability. Therefore, unlike Softmax-normalized attention weights, saliency values are not required to sum to 1. Accordingly, Eq. (4) in our manuscript measures the contributions of different input regions. For a given target prediction token, the normalized contribution ratios across all input regions sum to 1.

c. As explained in Section 3.2.2, masking all input functional regions severely disrupts the model's input structure and introduces additional biases, leading to substantial performance degradation. We believe that the System region contains important task instructions and structural information. Therefore, completely masking its attention would severely disrupt the original input structure and result in a significant drop in model performance. However, this does not imply that the System region contains no redundant attention. Rather than removing the entire System region, our goal is to identify relatively low-value attention allocations within it and redistribute them to the Knowledge region.

Furthermore, the additional experiments in Figures 25–34, which involve reducing and increasing attention in different functional regions, provide further evidence. Reducing attention to the Knowledge region more frequently leads to performance degradation. In the middle and later layers, increasing attention to the Knowledge region yields more substantial performance gains than increasing attention to the System region. These results are consistent with our analysis that the Knowledge region receives insufficient attention, while part of the attention allocated to the System region can be redistributed.

R-W2: It is important to clarify that KASR focuses on identifying redundant attention within the System region that can be redistributed, rather than assuming that the entire System region is useless or attempting to completely suppress its attention. The System region contains important information, such as task instructions, role definitions, and output constraints. Therefore, appropriately increasing attention to this region may help models better follow instructions and improve generation quality. However, this does not mean that redundant attention does not exist within the region. The overall importance of a region and the appropriateness of its internal attention allocation are two distinct issues. KASR aims to identify and redistribute relatively low-saliency attention, rather than simply reducing the overall importance of a particular functional region.

As shown in Figures 25–34, although increasing attention to the System, Query, and Knowledge regions can all potentially improve model performance, interventions on different regions exhibit different levels of stability. In particular, increasing System attention can also lead to performance degradation under certain intervention settings, whereas increasing Knowledge attention more frequently improves performance. This suggests that the effectiveness of attention interventions depends not only on the overall importance of the target region but also on the specific intervention layers, token selection, and their relevance to prediction. Therefore, KASR does not simply transfer attention from one region to another. Instead, it employs saliency-guided selection to identify positions that are more suitable for attention redistribution. Furthermore, the experiments jointly increasing attention to the Knowledge and Query regions in Section 4.3 also demonstrate the potential value of coordinated attention adjustment across different task-relevant regions.

We acknowledge the phenomenon observed by the reviewer in the MMLU experiments. Although MMLU does not contain an explicit external Knowledge region, KASR still achieves performance improvements of 4.4–6.5 percentage points. This does not directly demonstrate improved external knowledge utilization; rather, it indicates that KASR's underlying mechanism is not dependent on any specific input region. Its core principle is to identify redundant attention allocation across different functional input regions based on Attention–Saliency Imbalance and adjust it accordingly. In knowledge-augmented tasks, we focus on redistributing redundant attention from the System region to the Knowledge region. In other tasks, the same mechanism can also be applied to adjust attention allocation across other task-relevant regions. This further demonstrates the potential applicability of KASR as a general-purpose attention analysis and intervention mechanism.

R-W3:

a. We thank the reviewer for pointing out the issues regarding test set size and statistical reliability. We agree that the test sets of DXY and MMLU-Medicine are relatively small. We conducted a significance test comparing the Base and KASR results on MMLU-MED using Llama 3.1-8B and obtained p=0.125. This indicates that although KASR achieves a notable performance improvement on MMLU-MED, the current sample size is insufficient to provide strong statistical evidence. We will clarify this limitation in the revised manuscript.

In addition, although the initial manuscript reported results on datasets with relatively larger numbers of samples, including DXY, GMD, MMLU-LAW, and StrategyQA, we further incorporated NeQA and the moral_scenarios and professional_psychology subsets of MMLU to address the limited evaluation data. The dataset statistics are presented in Table 2, and the results are reported in Table 3. KASR achieves strong performance across these datasets, providing further evidence of its effectiveness.

Table 2. Statistics of the additional datasets

DatasetLanguage#Val#Test
MMLU-moral_scenariosEnglish895100
MMLU-professional_psychologyEnglish61269
NeQAEnglish39853

Table 3. KASR performance on the additional datasets

ModelMethodmoral_scenariosprofessional_psychology
Qwen2.5-3BBase37.0069.57
KASR (Ours)43.0073.91
ModelMethodNeQA
Qwen1.5-8BBase39.62
KASR (Ours)49.06

b. We used two subsets of the MMLU dataset. Since we needed to probe the contributions of different input regions to the final predictions on the validation set, and the official validation split contains too few samples to support stable saliency probing and layer selection, we used the official test split for these analyses and development, while reserving the official validation split for final performance evaluation. We will clarify this procedure in the revised manuscript.

c. Regarding the setting of β\beta, although Section 4.3 already analyzes the effects of different β\beta values through ablation experiments, we indeed omitted its specific selection procedure from the main experimental settings. Since the influence of knowledge context on model predictions may vary across tasks, we first evaluated three candidate values of β\beta on the validation set, namely β∈{0.25,0.5,0.75}\beta \in \{0.25, 0.5, 0.75\}, and selected the best-performing value for evaluation on the test set.

d. We described the settings of ACT and PASTA in Sections 4.1 and B.0.1, adopting the same settings as those in the original papers.

ACT identifies sink tokens whose attention ratios exceed (5/N) and redistributes 60% of their attention to the remaining tokens in the selected attention heads. It consists of two stages:

  • Profiling: ACT is applied to each attention head individually on the training set. The accuracy following each intervention is recorded, and all heads that yield an accuracy higher than the baseline are selected.
  • Testing: ACT is simultaneously applied to all selected attention heads, and the resulting performance is evaluated on the test set.

The effectiveness of intervening on individual heads does not necessarily imply that simultaneous intervention on multiple heads will also be effective. Since ACT does not constrain the attention components or input regions being modified, it substantially alters the model's attention patterns, which may explain its limited performance on many tasks. We also discuss this issue in Section 4.2.

PASTA reallocates attention from non-instruction tokens to instruction tokens. Similar to ACT, it consists of two stages. First, attention redistribution is individually applied to different attention heads on the validation set, and the top 1% of heads are retained for evaluation on the test set. Since PASTA modifies a relatively smaller amount of attention and does not substantially disrupt the model's structure, it performs better than ACT.

We re-evaluated all implementations. The ACT results were verified to be correct. For PASTA, the implementation on the InternLM2.5-1.8B model with the GMD dataset was also correct, but an issue with the execution environment affected the results. The correct result is 48.95, and we have corrected it accordingly.

Reviewer_NY5JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why large language models fail to fully utilize contextual knowledge in knowledge-grounded question answering. The authors partition inputs into functional regions and compare their attention allocation, gradient-based saliency, and contributions to residual-stream updates. They identify an attention–saliency imbalance: the system region receives substantial attention, whereas the knowledge region exhibits relatively high predictive saliency in middle-to-late layers despite receiving less attention. Motivated by these observations, the authors propose Knowledge-Aware Attention Steering and Redistribution (KASR). KASR selects half of the Transformer layers using saliency-to-attention ratios computed on a labeled validation set. During inference, it reduces attention to selected highly attended system tokens and proportionally redistributes the released attention to knowledge tokens. The method requires no parameter updates and is evaluated on ten models from three model families across five benchmarks. Recomputing Table 5 confirms an average improvement of approximately 3.83 percentage points over the strongest baseline in each model–task combination.

Critical strengths
  1. A coherent combination of analysis and intervention. The paper goes beyond attention visualization by combining gradient-based saliency, residual-stream contributions, and interventions at the region, layer, and attention-head levels. This provides a stronger empirical basis than attention distributions alone and establishes a clear motivation for the proposed inference-time intervention.

  2. A simple method with broadly positive reported results. KASR requires neither parameter updates nor an additional model forward pass. In Table 5, it improves over the unmodified model in all 50 model–task combinations and exceeds the strongest baseline in 45 combinations. These results indicate potential practical value even though the mechanistic interpretation requires further validation.

Critical weaknesses
  1. The specific contribution of saliency-guided layer selection is not sufficiently isolated. The attention interventions provide evidence that modifying regional attention can improve accuracy. However, they do not establish whether saliency-guided selection provides an advantage over simpler layer-selection strategies. Figure 8(b) explores intervention starting layers but does not report comparisons against random or fixed middle-to-late layer selection with the same number of intervened layers and identical intervention settings. Such controls are important for evaluating the central methodological contribution.The SFT analysis provides supporting correlational evidence, while the increase in knowledge attention in Figure 8(a) is directly induced by KASR and is not an independent measure of improved knowledge utilization. The broader mechanistic interpretation should therefore be qualified.

  2. Baseline ambiguities and limited statistical evidence weaken the comparative claims. The paper describes CAD as contrasting correct and perturbed knowledge, whereas the original CAD contrasts predictions with and without context. It is unclear whether this is a description error or an implementation difference. Some ACT results also deteriorate dramatically—for example, Llama3.1-8B on GMD drops from 61.09% to 7.11%—and warrant implementation and calibration checks. Table 4 reports only 31 test examples for MMLU-MED, so a single correct prediction corresponds to approximately 3.23 percentage points in an individual run. Without per-example results or uncertainty estimates, it is difficult to assess the reliability of the reported improvements. Averaging three runs does not replace an assessment of uncertainty across test examples.

Questions for authors
  1. Baseline implementation: Section 4.1 describes CAD as contrasting correct and perturbed knowledge, whereas the original method contrasts predictions with and without context. Which formulation was implemented? Please also clarify how ACT was calibrated and whether its substantial degradation persists after implementation checks.

  2. Contribution of saliency guidance: With the same number of intervened layers and identical β, λ, and redistribution rules, does saliency-guided selection outperform random selection or fixed middle-to-late layer selection?

  3. Evaluation protocol and reliability: Please identify the exact MMLU dataset and splits used, and clarify whether the citation to MMLU-Pro is a citation error or reflects the actual dataset. How was β selected, what varied across the three runs, and can you provide uncertainty estimates for the reported gains, particularly on the 31-example MMLU-MED test set?

4 comments

Thank you for taking the time and effort to review our manuscript and for providing valuable suggestions to improve its quality. We will address your comments point by point.

R-W1: a. Thank you for your comments, which are very helpful in improving the comprehensiveness of our experimental evaluation. We conducted additional experiments on the DXY dataset using Qwen2.5-3B and Qwen2.5-7B with random and fixed middle-to-late layer selection. Each setting was run three times, and the average results are reported. In each setting, 50% of the model's Transformer layers were selected, with identical attention intervention configurations. The results in Table 1 lead to the following observations:

  1. Random selection performs relatively poorly. On the two models, KASR outperforms Random by 3.88 and 2.91 percentage points, respectively. This indicates that arbitrarily selecting a set of layers for attention intervention does not necessarily yield comparable improvements.

  2. KASR outperforms or matches fixed-layer selection. On Qwen2.5-3B, KASR achieves gains of 0.97 and 2.91 percentage points, respectively. On Qwen2.5-7B, KASR matches both fixed-layer settings. We also explicitly examined the layers selected by KASR. For Qwen2.5-3B, the selected intervention layers are L27, L26, L25, L29, L30, L33, L20, L23, L21, L22, L31, L28, L24, L19, L32, L18, L17, and L15. For Qwen2.5-7B, they are L20, L18, L19, L21, L24, L22, L16, L17, L27, L26, L25, L15, L23, and L14. Most selected layers are located in the middle-to-late stages of the models, consistent with the findings in Section 3.2 of our paper. The fixed-layer settings also cover the middle-to-late layers. These results indicate that different Transformer layers do not contribute equally to knowledge utilization. The Saliency-to-Attention Ratio provides an effective criterion for identifying layers suitable for attention intervention and achieves competitive performance across different models. We will include these experiments in the revised manuscript.

Table1. Comparison results on the DXY dataset using Qwen2.5-3B and Qwen2.5-7B with random and fixed middle-to-late layer selection

ModelLayerAcc.
Qwen2.5-3BBase41.75
Random44.66
From Layer 17 to Layer 35 (layer indices)47.57
From Layer 18 to Layer 35 (layer indices)45.63
KASR (Ours)48.54
Qwen2.5-7BBase49.51
Random55.34
From Layer 13 to Layer 27 (layer indices)58.25
From Layer 14 to Layer 27 (layer indices)58.25
KASR (Ours)58.25

b. Regarding the mechanistic interpretation of knowledge utilization: Through statistical analyses, various intervention settings, and SFT experiments, we have preliminarily established the relationship among attention, saliency, and model performance. However, we also agree that the increase in Knowledge Region attention in Figure 8(a) is itself a direct result of the KASR intervention and cannot independently serve as evidence of improved knowledge utilization. In Section 4.3, Figure 8(b), we compare model performance when KASR redistributes attention from the System region to Iknow\mathcal{I}_{\mathrm{know}}, Iqry\mathcal{I}_{\mathrm{qry}}, Iknow+qry\mathcal{I}_{\mathrm{know+qry}}. The best performance obtained by allocating attention to Iknow\mathcal{I}_{\mathrm{know}} is approximately 4% and 1% higher than that obtained by allocating it to Iqry\mathcal{I}_{\mathrm{qry}} and Iknow+qry\mathcal{I}_{\mathrm{know+qry}}, respectively. This suggests that KASR can promote knowledge utilization in knowledge-driven tasks by reallocating redundant attention to the Knowledge region.

Furthermore, we are considering investigating the impact of KASR on knowledge utilization through the model's internal states, specifically examining whether the input representations in the hidden states before prediction become closer to the knowledge representations and the correct answer representations after applying KASR.

R-W2: Thank you for your comments. We address each point below.

a. We thank the reviewer for pointing out the potential ambiguity in our description of CAD in the original manuscript. We described the experimental settings in Sections 4.1 and B.0.1, and will provide further clarification. The original CAD indeed performs contrastive decoding by comparing model predictions with and without context. However, subsequent studies [1], [2] also describe contrastive learning with correct and distracting knowledge to help models learn knowledge utilization, and we followed this description in our manuscript. We acknowledge that this may cause confusion. Our actual implementation uses contrastive decoding based on the presence or absence of knowledge context, rather than directly contrasting correct knowledge with perturbed knowledge. Specifically, we adjust the model's prediction distribution using the following formula. CAD(y)=(1+α)logP(y∣x,k)−αlogP(y∣x)CAD(y) = (1 + α) log P(y | x, k) − α log P(y | x) We set α∈[0,1]\alpha \in [0,1], with a step size of 0.05, resulting in 21 candidate values, and select the best-performing result as the baseline. We will provide a more detailed description of this setting in the revised manuscript.

b. We described the settings of ACT and PASTA in Sections 4.1 and B.0.1. We adopted the same settings as those in the original papers. ACT identifies sink tokens whose attention ratios exceed (5/N) and redistributes 60% of their attention to the remaining tokens in the selected attention heads. It consists of two stages:

  • Profiling: ACT is applied to each attention head individually on the training set. The accuracy following each intervention is recorded, and all heads that yield an accuracy higher than the baseline are selected.
  • Testing: ACT is simultaneously applied to all selected attention heads, and the resulting performance is evaluated on the test set.

The effectiveness of intervening on individual heads does not necessarily imply that simultaneous intervention on multiple heads will also be effective. Since ACT does not constrain the attention components or input regions being modified, it substantially alters the model's attention patterns, which may explain its limited performance on many tasks. We also discuss this issue in Section 4.2.

c. We described the datasets in Section 4.1. For MMLU, we used selected subsets of the original dataset but incorrectly cited them as MMLU-Pro. We chose the professional_law and professional_medicine subsets of MMLU primarily because they evaluate models' knowledge in specialized domains such as law and medicine. Moreover, these English-language datasets provide a more direct assessment of domain knowledge. All sample counts are from the original MMLU dataset. Although MMLU-MED contains relatively few samples, DXY, GMD, MMLU-LAW, and StrategyQA provide larger evaluation sets, which together offer substantial evidence of KASR's performance. In addition, we included NeQA and the moral_scenarios and professional_psychology subsets of MMLU. Their statistics are presented in Table 2, and the results are reported in Table 3.

Furthermore, we acknowledge that the MMLU-MED test set contains only 31 samples, meaning that a single prediction can change the accuracy by approximately 3.23 percentage points. The average results over three independent runs reflect variation across runs but are insufficient to independently quantify statistical uncertainty at the sample level. We conducted a significance test comparing the Base and KASR results on MMLU-MED using Llama 3.1-8B and obtained p=0.125. This indicates that although KASR achieves a notable accuracy improvement on MMLU-MED, the current sample size is insufficient to provide strong statistical evidence. We will clarify this limitation in the revised manuscript.

Table 2. Statistics of the additional datasets

DatasetLanguage#Val#Test
MMLU-moral_scenariosEnglish895100
MMLU-professional_psychologyEnglish61269
NeQAEnglish39853

Table 3. KASR performance on the additional datasets

ModelMethodMMLU-moral_scenariosMMLU-professional_psychology
Qwen2.5-3BBase37.0069.57
KASR (Ours)43.0073.91
ModelMethodNeQA
Qwen1.5-8BBase39.62
KASR (Ours)49.06

R-W2: d. We discussed the ablation study of β\beta in Section 4.3, but its selection procedure was indeed missing from the experimental settings. Since the influence of knowledge context on model predictions may vary across tasks, we first evaluated three candidate values of (\beta) on the validation set, namely β∈{0.25,0.5,0.75}\beta \in \{0.25, 0.5, 0.75\}, and selected the best-performing value for evaluation on the test set.

[1] Huang L, Feng X, Ma W, et al. Improving contextual faithfulness of large language models via retrieval heads-induced optimization[C]//Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025: 16896-16913.

[2] Favero A, Zancato L, Trager M, et al. Multi-modal hallucination control by visual information grounding[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 14303-14312.

We sincerely appreciate your time and effort in reviewing our responses and engaging in this discussion. If you feel that our responses have addressed your concerns, we would be grateful if you would consider revising your score accordingly. If you have any further questions or concerns, we would be more than happy to continue the discussion and provide additional clarification.

Reviewer_QfwpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates attention allocation in LLMs and finds that the system region receives substantial attention despite its limited contribution to final predictions. Based on these observations, the paper proposes KASR, a training-free method that improves knowledge utilization by redistributing redundant attention from the system region to knowledge tokens at critical layers. Experiments across multiple models and benchmarks demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The proposed method outperforms the baselines across multiple datasets and different model families.

  2. The observations provide new insights into the attention behavior of LLMs.

Critical weaknesses
  1. Limited generalizability to modern attention architectures. The observation experiments are conducted exclusively on models with full attention, whereas recent LLMs adopt hybrid attention architectures. It remains unclear whether the observations generalize to these different architectures.

  2. Potential region length bias in the used metrics of Section 3.2.1. The metrics defined in Eq. (4), particularly when aggregated at the region level, may be confounded by the number of tokens in each region. Regions containing more tokens may naturally accumulate larger attention or saliency scores, introducing potential biases into the analysis.

  3. Unclear normalization of regional contributions. The metric defined in Eq. (5) for measuring regional contributions to residual stream updates does not guarantee that the contributions across regions sum to 1. However, Figure 2(c) appears to present normalized contributions within this range.

  4. Insufficient experimental details and presentation issues. The SFT setup is not sufficiently described. Additionally, the caption of Figure 6 contains typos.

  5. Limited evaluation on more novel models. The evaluated models are relatively outdated compared with recent LLM architectures. Including more recent models would strengthen the empirical evidence and provide a more convincing assessment of the method's generalizability.

Questions for authors
  1. Provide more analysis on large language models with hybrid attention architectures.

  2. Provide more evaluation results on more novel models.

6 comments

Thank you for taking the time and effort to review our manuscript and for providing valuable suggestions to improve its quality. We will address your comments point by point.

R-W1: KASR systematically investigates the relationships among the attention allocated to different input regions, their contributions to generated predictions, and model performance under various intervention and SFT. To facilitate the direct examination of attention and contribution, our current probing and experimental analyses primarily use full-attention models, including Llama 3.1/3.2, Qwen 1.5/2/2.5, and InternLM 2/2.5.

We fully agree with the reviewer that experiments under hybrid attention architectures would help demonstrate the generalizability of our method. However, in existing hybrid attention architectures, such as Qwen3.5-9B, which combines linear and full attention, it is difficult to calculate the attention and contribution ratios of different input regions in linear attention layers using Eqs. (1)–(5) in the paper. We are therefore attempting to adapt our method and code and are currently testing them on the DXY dataset. We will report the results once they become available. If we are ultimately unable to obtain a workable implementation, we will acknowledge the current method's reliance on full-attention architectures as a limitation.

R-W2: Thank you very much for this valuable suggestion. First, we would like to clarify that KASR considers the cumulative attention and cumulative saliency contribution of different input regions, rather than the average contribution per token. Both saliency metrics in Eq. (4) are calculated by summing the saliency scores of tokens within a region and then dividing by the total saliency across all visible tokens. This region-level aggregation approach has also been adopted in prior work. For example, Kang et al. [1] measure the "visual non-sink ratio" by aggregating attention weights over subsets of visual tokens, without explicitly normalizing by the number of tokens in each subset. This metric characterizes the total attention resources allocated to "informative visual tokens."

Nevertheless, we acknowledge that cumulative region-level metrics may be affected by differences in token counts and cannot fully isolate the importance of individual tokens. To address this concern, we have added a length-normalized analysis. Specifically, on the DXY dataset, we calculate the per-token attention and per-token saliency for each functional input region at every layer, along with the corresponding Saliency-to-Attention Ratio, using Eqs. (1)–(4). The results are presented in Table 1.

The results show that the System region indeed receives a relatively large attention allocation, with its cumulative attention proportion substantially exceeding that of the Knowledge region. Meanwhile, although the per-token saliency of the Knowledge region is not higher than that of the System region, its per-token attention is reduced much more substantially. This is related to the original attention allocated to the Knowledge region, as illustrated by the examples in Table 7 of the paper. Consequently, the Knowledge region still exhibits a higher Saliency-to-Attention Ratio, indicating a more pronounced mismatch between its predictive relevance and the attention resources it receives, particularly in the middle and later layers of the model. These findings further support our observation of the Attention–Saliency Imbalance.

We will incorporate the per-token analysis into the revised manuscript and explicitly distinguish cumulative region-level metrics from per-token averages to avoid ambiguity in interpreting regional contributions.

Aˉl(N,I)=1∣I∣∑j∈IAN,jl(1)\bar A_l(N,\mathcal I)=\frac{1}{|\mathcal I|} \sum_{j\in\mathcal I}A^l_{N,j} \qquad (1)

= \frac{1}{|\mathcal{I}|} \sum_{j\in\mathcal{I}} A^{l}_{\mathcal{I}_{\mathrm{qry}},j} \qquad (2)$$ $$\bar S_l(N,\mathcal{I}) = \frac {1}{|\mathcal{I}|}S_l(N,\mathcal{I}) \qquad (3)$$ $$\bar{S}_l(\mathcal{I}_{\mathrm{qry}},\mathcal{I}) = \frac{1}{|\mathcal{I}|} S_l(\mathcal{I}_{\mathrm{qry}},\mathcal{I}) \qquad (4)$$

R-W2: Table 1-1. Comparison of cumulative and per-token Attention and Saliency, as well as Saliency-to-Attention Ratios, across different input regions.

LayersCumulative System(Attention%/Saliency%)Cumulative Knowledge(Attention%/Saliency%)Pre-token System(Attention%/Saliency%)Pre-token Knowledge(Attention%/Saliency%)S/A SystemS/A Knowledge
06.2/5.255.2/49.11.4/0.91.9/1.30.0150.016
114.7/13.047.1/35.52.9/1.81.4/0.80.0340.029
233.0/12.230.8/26.95.9/1.20.8/0.40.0180.043
349.4/22.226.2/23.912.9/2.91.0/0.50.0180.037
457.1/23.420.1/22.615.0/2.80.8/0.40.0170.047
559.1/20.56.0/9.211.3/1.90.2/0.10.0200.088
659.1/37.46.0/8.210.6/4.40.2/0.10.0400.087
773.2/32.23.4/6.618.0/3.20.1/0.10.0260.116
853.0/31.24.2/5.48.6/3.60.1/0.10.0500.111
967.9/46.93.3/5.516.5/7.70.1/0.10.0700.172
1070.5/48.63.8/6.020.0/9.40.2/0.20.0740.171
1162.2/43.54.8/8.512.5/6.50.1/0.20.1170.301

R-W2: Table 1-2. Comparison of cumulative and per-token Attention and Saliency, as well as Saliency-to-Attention Ratios, across different input regions.

LayersCumulative System(Attention%/Saliency%)Cumulative Knowledge(Attention%/Saliency%)Pre-token System(Attention%/Saliency%)Pre-token Knowledge(Attention%/Saliency%)S/A SystemS/A Knowledge
1260.4/38.08.1/15.121.4/10.50.4/0.60.1300.385
1361.1/30.710.1/19.518.9/7.00.5/0.70.1150.443
1451.8/32.016.5/35.718.1/12.30.9/2.10.1880.659
1567.4/41.010.2/27.225.8/13.70.6/1.40.1320.577
1662.3/34.78.3/19.619.9/8.00.4/0.70.1020.432
1761.9/45.616.8/33.927.4/23.31.1/2.60.1310.359
1842.5/29.519.4/36.911.6/9.60.8/1.80.1040.284
1965.4/33.612.9/29.626.0/11.20.8/1.50.0470.211
2058.4/23.014.6/26.814.3/3.20.5/0.60.0340.161
2169.4/44.213.2/29.025.7/12.10.7/1.20.0480.166
2253.5/25.923.1/36.516.2/5.31.1/1.10.0400.130
2348.6/16.719.9/28.011.1/2.40.7/0.60.0220.091

[1] Kang S, Kim J, Kim J, et al. See what you are told: Visual attention sink in large multimodal models[C]//International Conference on Learning Representations. 2025, 2025: 87676-87703.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal

Dear AC and reviewers,

We sincerely thank the three reviewers for their careful evaluation and constructive feedback. The reviewers generally recognize the core research value and contributions of this work. And their comments have helped us further clarify the motivation, methodological design, and experimental scope of KASR (Knowledge-Aware Attention Steering and Redistribution). During the rebuttal, we provided additional theoretical explanations, experimental analyses, and implementation details, and conducted several supplementary experiments. We summarize our main responses and improvements below.

1. Further Clarification of the Attention–Saliency Imbalance

We further clarified the Attention–Saliency Imbalance identified through computational analyses, intervention experiments, and SFT in the initial submission, as well as the important role of the Knowledge region in middle-to-late layers (Section 3).

We also extended the analysis from the region level to the token level. Through length-normalized regional Attention and Saliency measurements, together with token-level Saliency-to-Attention Ratios, we further demonstrated that System tokens receive a disproportionately large share of attention relative to their predictive contributions, whereas Knowledge tokens exhibit relatively high contributions despite receiving less attention.

Furthermore, we distinguished our findings from the conventional initial-token Attention Sink phenomenon. Building on the analysis in Appendix B.2 of the initial submission, we incorporated token-level Saliency analysis to further differentiate the Attention–Saliency Imbalance from the initial-token Attention Sink. We also conducted an additional experiment that redistributes only the attention allocated to the initial token toward the Knowledge region. Although this intervention improves performance over the Base model, it remains inferior to KASR. This provides further evidence that KASR's effectiveness cannot be fully attributed to the initial-token Attention Sink.

2. Further Validation of Layer Selection and Knowledge-Aware Attention Intervention

In Section 4 of the initial submission, we conducted extensive comparative, ablation, and analytical experiments across 10 models and five benchmarks. During the rebuttal, we further validated the effectiveness of our method in response to the reviewers' comments.

In particular, Reviewer NY5J questioned whether Saliency-guided Layer Selection is more effective than random selection or fixed middle-to-late layer selection. To address this concern, we conducted additional controlled experiments on Qwen2.5-3B and Qwen2.5-7B. Under identical intervention settings, we compared KASR with random and fixed middle-to-late layer selection strategies and reported the average results over three runs. The results show that KASR outperforms random selection on both models and achieves performance comparable to or better than fixed-layer selection. These findings further support the effectiveness of the proposed Saliency-guided Layer Selection strategy.

3. Model Adaptability and Evaluation on Additional Datasets

To further examine the applicability of KASR to newer models and different model scales, we added experiments on Qwen3-4B/8B/14B. The results show that KASR outperforms the Base model on all three models.

In addition, we conducted experiments on NeQA and NQ-Swap to investigate KASR's performance under different knowledge conditions. For NQ-Swap, KASR achieved improvements of 3.33 and 2.90 percentage points under the original-knowledge and counterfactual-knowledge conditions, respectively. These results indicate that a KASR configuration determined using training data with original knowledge can still yield performance gains when the input knowledge changes.

4. Clarification of Baseline Settings and Datasets

Sections 4.1 and B.1 of the initial submission described the datasets and baselines used in our experiments. However, some details were not sufficiently clear due to space limitations. In response to the reviewers' concerns about experimental settings, we supplemented the SFT training configurations and the implementation details of CAD, ACT, and PASTA. We also further clarified the use of the official MMLU test and validation splits.

We carefully examined the concern regarding the small size of the MMLU-MED test set and its limited statistical power. We further added experiments on four new datasets beyond the original five benchmarks to broaden the evaluation and provide additional evidence of KASR's performance.

The applicability of KASR to hybrid attention architectures remains an open direction for further investigation. Apart from this aspect, we have addressed the reviewers' major concerns. We hope that our detailed rebuttal and supplementary experiments further demonstrate the research value of KASR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.