acceptodds
Under review as a conference paper at ICLR 2027

Misled by the Few: A Mechanistic Analysis of Demonstration Conflicts in In-Context Rule Inference

Abstract

In-context learning enables large language models to perform novel tasks through few-shot demonstrations, including in real deployments where models must infer underlying rules from examples alone. However, demonstrations can naturally contain conflicting examples, making this capability vulnerable. Thus, understanding the dynamics under such conflicts is essential for reliable ICL deployment. In this work, we investigate this question on tasks requiring genuine demonstration reliance. We find that models suffer substantial performance degradation from a single demonstration with a corrupted rule. This systematic misleading behavior motivates our investigation of how models process conflicting evidence internally. Using linear probes and logit lens analysis, we discover that under corruption, models encode both correct and incorrect rules in intermediate layers but develop prediction confidence only in late layers, revealing a two-phase computational structure. We then identify attention heads for each phase underlying the reasoning failures: Vulnerability Heads in early-to-middle layers exhibit positional attention bias with high sensitivity to corruption, while Susceptible Heads in late layers significantly reduce support for correct predictions when exposed to the corrupted evidence. Targeted ablation validates our findings, with masking a small number of identified heads improving accuracy by up to 11.12% relative to the corrupted baseline.

Primary area:
interpretability and explainable AI
Keywords:
In Context Learning, Rule Inference, Conflict Resolution, LLM Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8MstOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors test the model on two tasks, Operator Induction and Fake Word Inference, on 4 open-source models, and finds significant performance drop after corruption. This paper discovers that under corruption, models encode both correct and incorrect rules in intermediate layers but develop prediction confidence only in late layers. And masking specific head would significant mitigate this issue!

Critical strengths
    1. This paper finds an interesting and important problem, where the model's performance would significantly drops after some simple corruptions, such as conflicting demonstrations,
    1. This work designs two interesting test tasks, Operator Induction and Fake Word Inference, to test the robustness of LLMs.
    1. The authors find out two kinds of attention heads, which are Vulnerability Heads and Susceptible Heads, specifically related to the degrading performance. And experiments shows that masking them would lead to significant performance improvements.
Critical weaknesses
    1. The limited scope of tested LLMs. The authors only tested the model up to 8B and limited to two small model families (Qwen and Llama). Expanding to more open source and even proprietary models would significantly strengthen the claim.
    1. The presentation of the paper is not straightforward and hard to comprehend, especially for diagrams and figures!
    1. The performance was reported on single tasks, and not on some public benchmarks or compared with baselines. The performance on the public benchmarks like GSM8K or MATH would significantly strengthen the generalizability of the claim.
Questions for authors
    1. Halawi et al. 2023 also mentioned the similar findings that some attention head is highly relevant to the worse performance with low-quality demonstrations. Though the paper cites the work, and claim that Halawi et al. 2023 mainly focuses on text QA and classification. I'm curious what's the difference between the submitted work with the related work other than task difference.
    1. For table 1, I'm curious the performance without prompt corruption!
    1. There are major inconsistencies in the paper, such as mismatch between Sec. 5.2 and Fig.12. The example in Sec. 5.2 for top susceptible heads are L30H29 和 L31H5, but in Fig. 12 are L34H18 and L33H31. -4. There are a lot of grammar errors in the paper.
3 comments
AuthorsOpenReview ↗

Authors Comment

Dear Reviewer 8Mst,

Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Thanks for the detailed rebuttal! It addresses most of my concerns but I still remain concerns about the generalization of the method. Therefore I would like to keep my score. Best wishes for the paper!

AuthorsOpenReview ↗

Authors Comment

Dear Reviewer 8Mst,

Thank you again for your constructive suggestions and kind words! We are glad that our response addresses most of your concerns. Your suggestions have made the paper clearer and its mechanistic analysis more complete.

We agree that broader generalization is important, and Section 6 names larger models and richer rule-inference settings as future work. As an initial mechanistic analysis, the paper finds the same failure on two tasks and traces it to the same two head families in four models from two families, with consistent evidence from probes, the logit lens and targeted ablation. We hope this work encourages the community to test the mechanism in larger models and more tasks.

Best regards,

The Authors

Reviewer_h8PGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how LLMs process conflicting demonstrations in in-context rule inference. The authors show that a single corrupted demonstration can substantially degrade performance even when the remaining demonstrations support the correct rule. Based on linear probing and logit-lens analyses, the paper proposes a two-phase account in which competing rules are encoded in intermediate layers and resolved in later layers. It further identifies Vulnerability Heads and Susceptible Heads associated with these two stages. Targeted ablation of the identified heads improves accuracy under corruption, providing causal evidence for their relevance.

Critical strengths
  1. The paper studies a clear and practically relevant failure mode of in-context learning, and the motivation is well established.
  2. The behavioral effect is strong: models can be significantly misled by a single corrupted demonstration despite a clear majority of correct examples.
  3. The mechanistic analysis is reasonably comprehensive, combining probes, logit-lens analysis, head-level localization, and targeted ablations.
  4. The ablation results are convincing: targeted head removal improves performance in 23/24 reported configurations and consistently outperforms random ablation.
  5. The analysis is conducted across multiple model families and includes some cross-task validation.
Critical weaknesses
  1. The proposed two-phase interpretation is somewhat stronger than the evidence directly establishes. The temporal difference between linear-probe decodability and logit-lens readability supports different representational stages, but does not by itself demonstrate a sequential “conflict creation → conflict resolution” computation.
  2. The positional-bias argument assumes that demonstrations provide equivalent evidence. This assumption is not fully guaranteed in Operator Induction, where individual examples can differ in how strongly they disambiguate the underlying operator. Therefore, part of the position-dependent corruption effect may be confounded with demonstration informativeness.
Questions for authors

see weaknesses

1 comment
AuthorsOpenReview ↗

Authors Comment

Dear Reviewer h8PG,

Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Reviewer_T8WaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how large language models handle conflicting demonstrations during in-context rule inference. Using Operator Induction and Fake Word Inference, the authors show that a single corrupted demonstration can substantially reduce prediction accuracy even when most demonstrations support the correct rule. Through linear probing and logit lens analysis, they propose a two-phase mechanism, competing rules become encoded in intermediate layers, while the final prediction preference emerges in later layers. They identify two groups of attention heads a) Vulnerability Heads associated with positional sensitivity and b) Susceptible Heads associated with reduced support for the correct rule.

Critical strengths
  1. Understanding how LLMs handle contradictory demonstrations is relevant to the reliability of in-context learning. The focus on rule-inference tasks, where models must rely on demonstrations rather than simply retrieve memorized answers, provides a useful experimental setting.

  2. Systematic mechanistic investigation. The paper combines behavioral experiments, linear probing, logit lens analysis, attention-head attribution, and targeted ablation. This provides multiple perspectives on the observed failures rather than relying exclusively on accuracy measurements.

  3. Evidence of causal relevance. Table 1 shows that targeted head ablation generally improves performance under corruption, whereas random head ablation degrades it. The cross-task experiments also provide preliminary evidence that some identified components may capture mechanisms shared across different rule-inference tasks.

Critical weaknesses
  1. Limited novelty relative to existing mechanistic studies. Halawi et al. (2024) already investigate false demonstrations using intermediate-layer predictions, identify late-layer attention heads responsible for misleading behavior, and demonstrate improvements through head ablation. The present paper extends this analysis to minority corruption in rule-inference tasks and introduces early-layer Vulnerability Heads. However, the extent to which these represent fundamentally new mechanisms remains insufficiently established.

  2. Limited task coverage and uncertain generalizability. The experiments rely on two relatively simple, synthetic tasks. Although Operator Induction originates from the established VL-ICL Bench, the paper does not sufficiently clarify whether it follows the original task construction and evaluation protocol. The assumption that all demonstrations provide equally informative evidence also warrants examination. For example, in Operator Induction, different operations can produce identical outputs for particular inputs. Consequently, performance differences attributed to demonstration position in Figure 2 may partly reflect differences in example informativeness unless demonstration content is carefully controlled.

  3. The proposed two-phase mechanism is not conclusively established. Linear probes demonstrate that information about competing rules is decodable from intermediate representations, but this does not establish how the model uses that information. Similarly, confidence emerging in later layers through the logit lens does not necessarily demonstrate an explicit conflict-resolution process. Section 4.2 examines prompts asking models to identify the underlying rule rather than compute the final answer, potentially introducing a mismatch between the mechanistic analysis and the original prediction task. Stronger causal interventions would help substantiate the proposed interpretation.

  4. Limited evidence supporting the effectiveness of identified heads. Although Table 1 demonstrates positive effects from targeted ablation, many improvements are relatively modest, and the reported gains are relative percentages rather than percentage-point improvements.

Missing prior work:

  • Todd et al., Function Vectors in Large Language Models (ICLR 2024).
  • Yin and Steinhardt, Which Attention Heads Matter for In-Context Learning? (ICML 2025).
  • Gao et al., On the Noise Robustness of In-Context Learning for Text Generation (NeurIPS 2024).
Questions for authors
  1. How does the proposed two-phase mechanism differ from the late-layer overthinking and false induction heads identified by Halawi et al. (2024)? Can the authors establish whether Vulnerability and Susceptible Heads are functionally distinct from previously identified ICL-related heads?

  2. Does the positional-bias analysis control for demonstration-specific informativeness by permuting the same examples across positions? How are ambiguous operator examples, where multiple rules produce identical outputs, handled?

  3. Could the authors clarify how the rule-identification analysis relates to the original answer-prediction behavior?

1 comment
AuthorsOpenReview ↗

Authors Comment

Dear Reviewer T8Wa,

Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttals and Paper Revisions

We thank the three reviewers for their careful reviews. We have uploaded a revised PDF with the revisions marked in blue.

Main results in the responses

  1. The position effect holds with demonstration content fixed (T8Wa, h8PG). We rotated fixed sets of four demonstrations so that each one is corrupted once at every position. In every model, position moves accuracy more than the choice of corrupted demonstration, with p < 0.001. Demonstration informativeness therefore does not explain the position effect.
  2. The interventions support the order of the two phases (h8PG, T8Wa). Targeted ablation of either head family improves accuracy under corruption in 23 of 24 configurations while random ablation lowers it, and ablating the early Vulnerability Heads lowers the susceptibility of the late Susceptible Heads. An intervention on the early phase thus changes the late phase, as the order from conflict creation to conflict resolution predicts.
  3. Our conflict differs from that of Halawi et al., and so does the mechanism (T8Wa, 8Mst). Their models often solve the tasks zero-shot, whereas ours are near chance without demonstrations, so the model must arbitrate among the demonstrations themselves. Since the answer is computed from new operands, copying a label cannot explain why 71% to 81% of the flipped answers equal the corrupted operator's result. The models therefore apply the corrupted rule itself.
  4. Heads scored on the rule prompt matter for the final answer (T8Wa). Susceptible Heads are scored on the prompt that asks for the rule, yet ablating them raises the accuracy of the final answers in every configuration of Table 1. In percentage points, the best targeted ablation gains about 2 to 7 points, while random ablation loses accuracy.
  5. The failure persists at larger scale, and our head score beats simpler rankings (8Mst). In Qwen3-32B, one corrupted demonstration still lowers accuracy by up to 44.4 points, and our score yields significant ablation effects in more configurations than attention or sensitivity alone, as Appendices A.4 and E.2 report.

Changes to the paper

  • The abstract states that the results support a two-phase account.
  • Section 2 contrasts our inter-context conflict with the context-memory conflict of Halawi et al. and discusses the three works T8Wa suggested.
  • Section 3 and the new Appendix A.5 add the rotation control, with both effects shown under Figure 2.
  • Section 4.3 reports the ablation gains in percentage points, converted in Appendix E.1, and the effect of Susceptible Heads on the final answer.
  • Section 5.2 corrects the head indices, and the new Section 5.3 sets out the evidence for the order of the two phases.
  • Figure 1 marks which panel shows conflict creation and which shows conflict resolution, the other captions are clearer, and the grammar is corrected.

The scatter plot of attention allocation against corruption sensitivity moved from Appendix E.2 into Section 4.3.1 as Figure 7, so former Figures 7 to 9 are now Figures 8 to 10. Every number our responses cite is unchanged.

We thank the reviewers again and would be glad to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.