- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates evidence resistance in language models, where language models preserve their original answer preferences despite conflicting external evidence. The authors introduce Contextual Shift Attribution (CSA) to analyze the contributions of the components to answer preference changes based on their directional contribution. CSA uses repeated assertions of an alternative answer to detect components that resist preference revision. The detected components are then patched with their outputs from evidence-free prompts. The experiments conducted on six language models and three knowledge-conflict benchmarks show higher evidence-following rates than previous steering approaches. Besides, the paper investigates the overlap and transferability of the components identified across benchmarks.
Critical strengths
- CSA considers not only the change in the component output but also its contribution to preference changes. It allows us to separate the components resisting preference revision from the ones supporting it.
- The intervention results are impressive. In Table 3, the results show that targeted patching substantially outperforms random patching with the same component budget in all six language models and three benchmarks.
- The results across benchmarks are useful. Table 4 shows substantial overlap among identified attention heads, while Table 5 shows that components identified on one benchmark can be transferred to others with little performance loss.
Critical weaknesses
- The evidence-following rate is defined by comparing the probabilities of the evidence-supported answer and the model's original answer (Appendix A.3). If P(d) > P(c0) in an example, it counts as successful, even if another option has a higher probability than d. Such a metric is appropriate for quantifying preference revision; however, it does not capture how often the evidence-supported answer is chosen by the model. Reporting this measure would make Table 1 more informative.
- Components are selected and intervention budgets are tuned on the same benchmarks used for evaluation. As explained in Appendix A.2, CSA is computed across all the questions in the benchmarks, and the budgets are selected based on the average performance on the three benchmarks. Thus, the reported results may be inflated by using the evaluation data in the component selection and tuning process. Although Table 5 demonstrates the transfer of the components across benchmarks without reselection, the budgets are optimized for all three benchmarks.
- The necessity of repeating assertions to identify components remains unclear. According to the process illustrated in Appendix A.4.2, CSA repeats a short assertion regarding an alternative answer 1,000 times, whereas the actual test uses natural evidence passages. These are two completely distinct types of inputs, and it is unclear why such a large number of repetitions is needed for identifying resistance-related components. A comparison using fewer repetitions or the original evidence passages would help justify this choice.
Questions for authors
- Is it possible to show the percentage of cases where the evidence-supported answer has the highest probability among all four options?
- Is it possible to conduct an assessment of the method using separate calibration and test sets where intervention budgets are not determined based on test set performance? With respect to the cross-benchmark transfer case, is it possible to determine budgets using only the source benchmark?
- How sensitive is component selection to the number of repetitions? Does it matter if we use a single assertion or the evidence passage itself to determine components?
4 comments
Weakness 2 and Question 2
Weakness 2 and Question 2: Performance Comparison under Calibration/Test Splits
We compare our method with the baselines on held-out test splits of all three benchmarks and assess cross-benchmark transfer across different source–target pairs in Table 2 and Table 3 respectively. Specifically, we follow the protocol in Table 1 to select components on each source benchmark’s calibration set and evaluate performance on held-out test sets within and across benchmarks.
Table 2: Within Benchmark Performance Across Settings and Baselines
| model | bench | dense | CAD | RCA | PASTA | JUICE | Ours |
|---|---|---|---|---|---|---|---|
| Qwen3-8B | faitheval | 21.4 | 37.3 | 44.4 | 79.6 | 87.3 | |
| Qwen3-8B | KRE | 56.6 | 72.4 | 74.3 | 91.9 | 96.8 | |
| Qwen3-8B | Farm | 65.7 | 77.0 | 76.3 | 78.3 | 95.9 | |
| Qwen3-4B | faitheval | 17.1 | 25.2 | 31.9 | 60.6 | 78.7 | |
| Qwen3-4B | KRE | 52.6 | 70.6 | 71.3 | 89.7 | 95.8 | |
| Qwen3-4B | Farm | 61.5 | 72.4 | 58.6 | 68.3 | 92.0 | |
| Qwen3-1.7B | faitheval | 17.3 | 25.1 | 31.5 | 39.0 | 81.2 | |
| Qwen3-1.7B | KRE | 31.0 | 38.3 | 52.1 | 66.5 | 90.2 | |
| Qwen3-1.7B | Farm | 38.5 | 46.5 | 52.4 | 55.7 | 80.1 | |
| Mean | 40.2 | 51.6 | 54.8 | 73.1 | 88.1 |
Table 3: Cross Benchmark Performance Across Settings
| model | test benchmark | faitheval components | KRE components | Farm components |
|---|---|---|---|---|
| Qwen3-8B | faitheval | 87.3 | 89.6 | 87.1 |
| Qwen3-8B | KRE | 95.8 | 96.8 | 94.7 |
| Qwen3-8B | Farm | 96.8 | 97.2 | 95.9 |
| Qwen3-4B | faitheval | 78.7 | 78.1 | 67.7 |
| Qwen3-4B | KRE | 95.5 | 95.8 | 93.8 |
| Qwen3-4B | Farm | 92.1 | 92.7 | 92.0 |
| Qwen3-1.7B | faitheval | 75.8 | 73.9 | 67.5 |
| Qwen3-1.7B | KRE | 91.4 | 90.2 | 88.8 |
| Qwen3-1.7B | Farm | 82.0 | 80.0 | 80.1 |
As shown in Table 2, our method outperforms all baselines by at least 7.5 percentage** on average aligned with the results in Table 1. These results indicate that the effectiveness of the identified resistance-associated components extends beyond the calibration samples, supporting the generalization of our component localization approach to unseen queries within the same benchmarks. Furthermore, the results in Table 3 exhibit a mean performance reduction of 1.1 percentage relative to within-benchmark evaluation, corroborating the generalization results reported in the manuscript. All the results in Tables 2 and 3 demonstrating strong cross-benchmark generalization and superior performance under this stricter independent evaluation protocol.
Weakness 3 and Question 3
Weakness 3 and Question 3: Necessity of Sufficient Repetition Counts
We report performance across repetition counts and evidence formats in Table 4 and component convergence at N=1000 in Table 5. Our conclusion is that achieving strong and stable intervention performance requires a sufficiently large repetition budget, with theoretical grounding in the convergence guarantee established in Appendix D. Specifically, we compare intervention performance across assertion repetitions and natural evidence passages using the same component selection and intervention protocol. We additionally report layerwise convergence rates for attention and FFN contributions at (N=1000), requiring maximum tail variation to remain within 5% or 10% of the total displacement.
Table 4: Intervention Performance Across Repetition Counts and Evidence Formats
| model | bench | dense | natural reason | x1 | x10 | x100 | x1000 |
|---|---|---|---|---|---|---|---|
| Qwen3-8B | faitheval | 21.5 | 75.6 | 58.4 | 64.4 | 70.9 | 77.9 |
| Qwen3-8B | KRE | 56.0 | 88.6 | 78.6 | 77.5 | 77.4 | 95.1 |
| Qwen3-8B | Farm | 64.8 | 83.6 | 80.6 | 77.9 | 75.0 | 86.2 |
| Qwen3-4B | faitheval | 16.4 | 75.4 | 70.0 | 72.0 | 68.4 | 63.3 |
| Qwen3-4B | KRE | 51.8 | 97.1 | 90.7 | 88.0 | 89.9 | 91.1 |
| Qwen3-4B | Farm | 60.2 | 85.7 | 81.1 | 82.1 | 82.4 | 83.9 |
| Qwen3-1.7B | faitheval | 15.7 | 47.3 | 64.6 | 74.4 | 70.6 | 67.4 |
| Qwen3-1.7B | KRE | 30.7 | 91.6 | 86.6 | 81.2 | 83.7 | 84.3 |
| Qwen3-1.7B | Farm | 36.8 | 76.8 | 72.2 | 69.6 | 72.4 | 71.1 |
| Mean | 39.3 | 80.2 | 75.9 | 76.3 | 76.7 | 80.0 |
Table 5: Layerwise Convergence Rate Across Models and Benchmarks.
| model | bench | layers | attn a=0.1 | FFN a=0.1 | attn a=0.05 | FFN a=0.05 |
|---|---|---|---|---|---|---|
| Qwen3-8B | faitheval | 36 | 0.996 | 0.996 | 0.975 | 0.971 |
| Qwen3-8B | KRE | 36 | 0.999 | 0.999 | 0.981 | 0.981 |
| Qwen3-8B | Farm | 36 | 0.997 | 0.995 | 0.970 | 0.968 |
| Qwen3-4B | faitheval | 36 | 0.982 | 0.973 | 0.967 | 0.947 |
| Qwen3-4B | KRE | 36 | 0.998 | 0.991 | 0.988 | 0.971 |
| Qwen3-4B | Farm | 36 | 0.993 | 0.987 | 0.973 | 0.965 |
| Qwen3-1.7B | faitheval | 28 | 0.818 | 0.795 | 0.749 | 0.714 |
| Qwen3-1.7B | KRE | 28 | 0.874 | 0.865 | 0.811 | 0.797 |
| Qwen3-1.7B | Farm | 28 | 0.858 | 0.846 | 0.789 | 0.765 |
| Mean | 0.946 | 0.939 | 0.911 | 0.898 |
As shown in Table 4, increasing repetitions from to improves mean performance by 4.1 percentage, demonstrating the necessity of sufficient repetition counts. Moreover, natural evidence achieves comparable average performance but ranges from 47.3% to 97.1% across model–benchmark settings. This variability is consistent with the absence of a convergence guarantee for single-passage inputs, whereas repetition stabilizes component contributions as empirically demonstrated in Table 5. Collectively, these results demonstrate the importance of repetition count and evidence format for strong and stable intervention performance and substantiate the rationale and effectiveness of our experimental configuration.
Response to the Rebuttal
Regarding W1/Q1, the results in Table 1 indeed show that CSA also achieves competitive top-1 accuracy compared to other methods. Regarding W2/Q2, the results in the rebuttal show no noticeable performance drop, and the transfer performance remains stable. Regarding W3/Q3, using 1,000 repetitions does lead to a noticeable improvement over 1, 10, or 100 repetitions, but the advantage over natural evidence passages is less clear. Nevertheless, using repeated assertions of randomly selected alternative answers is more consistent with the paper's formulation than directly using natural evidence passages.
I thank the authors for their thorough rebuttal, and my concerns are largely resolved. I will maintain my score.
Weakness 1 and Question 1
We thank the reviewer for the comment. We further report top-1 accuracy over the four answer options across all three benchmarks using disjoint calibration and test splits for component selection and evaluation. Specifically, we partition each benchmark into a 100-question calibration set and a held-out test set at an approximate ratio of 1:9, using the former for component selection according to the protocol in the manuscripts and the latter for performance comparison with the baselines in Qwen models. The results are shown in Table 1.
Table 1: Evidence-Supported Answer Ranked Highest Among Four Options Arcoss Settings
As shown in Table 1, our method outperforms all baselines by at least 8.6 percentage on average in four-option top-1 accuracy. These results demonstrate that the gains reflect improved evidence-supported answer selection rather than inflated performance estimates induced by a particular evaluation metric.