acceptodds
Under review as a conference paper at ICLR 2027

When the Wrong Key Wins: Understanding and Detecting Hallucinations in LLMs

Abstract

Large language models can hallucinate even when the knowledge required for a correct answer is already available. We study this failure through a latent-key view of inference, where answer selection depends on competition among associations acquired during pretraining. We show that model predictions can be highly sensitive to individual query keywords, that these influential keywords exhibit entity-specific binding, and that their effects are systematically shaped by pretraining frequency. Multiple bindings can also compete and exhibit higher-order interactions within the same query. Based on this mechanism, we introduce a two-stage keyword-perturbation method for hallucination detection. By removing influential keywords and measuring how the model reorganizes its prediction, the method distinguishes errors caused by misleading key associations from correct decisions supported by diagnostic evidence. Across multiple models and benchmarks, perturbation provides a strong and transferable detection signal, reaching AUROC on probe-known ScientistQA. Finally, we extend the same probabilistic framework to four hallucination regimes: knowledge deficit, wrong knowledge, context distraction, and unstable inference. Their distributions across benchmarks help explain why different detector families succeed in different settings.

Primary area:
interpretability and explainable AI
Keywords:
LLM Hallucination, Hallucination Detection, Perturbation Analysis, Causal Evidence Localization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kdVTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates why LLMs can produce hallucinations even when the knowledge required to answer correctly is available, through the lens of latent-key competition. The authors suggest that competing entity–keyword associations acquired during pretraining can lead models to select incorrect answers, and provide empirical analyses of these effects. Based on these observations, they propose a two-stage keyword perturbation method for hallucination detection, which identifies influential keywords and examines how model predictions change after their removal.

Critical strengths

S1. Interesting perspective on the mechanisms underlying hallucinations.

Rather than focusing solely on detecting hallucinations, this paper attempts to explain why models produce incorrect answers despite having access to the relevant knowledge. The latent-key perspective offers an interesting interpretation of hallucinations through competition among entity–keyword associations. The authors further investigate this hypothesis through keyword perturbation, controlled frequency exposure, and analyses of interactions among multiple keywords.

S2. Strong empirical performance in hallucination detection.

The proposed two-stage keyword perturbation method achieves competitive or superior performance compared to existing uncertainty-based and representation-based approaches across several benchmarks, including ScientistQA, TriviaQA, and DROP. In particular, it achieves an AUROC of 0.910 on probe-known ScientistQA and demonstrates promising generalization to unseen entity domains, supporting the effectiveness of the proposed detection signals.

Critical weaknesses

W1. Insufficient evidence that latent-key competition is the underlying cause of hallucinations.

The paper shows that entity–keyword associations influence answer selection, but this does not necessarily establish them as the underlying cause of hallucinations. For example, the Eric Clapton case could also result from a failure to properly handle the explicit negative constraint. Moreover, Table 1 shows strong bindings in both correct and incorrect responses, while Table 3 does not fully distinguish associative bias from logical reasoning failures. Further experiments separating these factors would strengthen the central claim.

W2. Limited connection between the limitations of existing approaches and the proposed method.

The introduction highlights limitations of existing hallucination detectors, but it remains unclear how the proposed method directly addresses them. Although the experimental results demonstrate strong detection performance, there is limited analysis of whether the method succeeds specifically in cases where existing approaches fail. More targeted comparisons would strengthen the motivation.

W3. Insufficient justification for using model-wide mean embeddings.

The method replaces keyword embeddings with the model-wide mean embedding, but it is unclear whether this selectively removes keyword associations or introduces unintended perturbation artifacts. Would a more context-dependent replacement, such as the mean embedding of the original query, be more appropriate? Comparisons with alternative strategies, such as query-specific mean embeddings or token deletion, would help validate this design choice.

W4. Limited applicability to black-box LLMs.

The proposed method requires access to the generator's internal embeddings, hidden states, and candidate likelihoods, which limits its applicability to black-box LLMs accessible only through APIs. This dependency may restrict its practical deployment in real-world settings.

Questions for authors

Q1. How sensitive is the proposed detector to the choice of alternative answer?

The method relies on relative preference scores between the selected and alternative answers, but the choice of alternative may affect detection performance. Although Appendix C.3 evaluates a Pred-only variant, could the authors clarify how sensitive the full method is to different alternative candidates, particularly in open-ended settings?

4 comments

Thank you for the response. I acknowledge that the interventions in Tables 8–9 support the contribution of competing associations, and that Table 22 demonstrates the added value of perturbation features. I also view the internal-access requirement as a limitation of applicability rather than soundness.

The following concerns remain:

  1. Presentation and logical connection: Although Table 8 is summarized in Table 2, the key findings from Tables 9 and 22 also warrant inclusion in the main text. Keyword associations influencing answer selection, their competition causing errors, and the detector exploiting that competition are distinct claims. Their connections remain insufficiently established, making the transition from the mechanistic account to the detection method somewhat unclear.

  2. Evaluation and mechanism: Correctness-based AUROC alone cannot distinguish errors despite accessible knowledge from errors due to missing knowledge. Probe-known ScientistQA partially addresses this, but an evaluation metric or protocol that explicitly separates these cases is needed to assess the central claim.

  3. Perturbation validity: I understand the concern that query-specific mean embeddings may retain contextual information, but this should be tested empirically. Do the findings hold under token deletion, query-specific mean replacement, and random vectors matched to the embedding scale and variance? These controls would help distinguish semantic suppression from perturbation-specific artifacts.

  4. Alternative-answer sensitivity: Independent construction of alternatives does not establish robustness to their choice. With the question and generated answer fixed, how much do detection scores and AUROC vary across different plausible alternatives? The Pred-only comparison tests detection without an alternative, rather than sensitivity to different alternatives.

We thank the reviewer for detailed feedback. We address the concerns below.

Presentation and logical connection

We will revise Sections 2 and 3 to present a clearer progression: (1) Keyword associations influence answer selection, supported by the controlled frequency–dose experiment (Table 8, Table 2); (2) Competing associations can contribute to hallucinations, supported by the directional intervention results in Table 9, where neutralizing wrong-supporting keywords increases correct-answer generation, while neutralizing right-supporting keywords decreases it; and (3) These perturbation responses provide useful detection signals, supported by the matched ablation in Table 22, which shows improvements from adding keyword-perturbation features, particularly under cross-domain transfer. We will move the key findings from Tables 9 and 22 into the main text and strengthen the transition from Section 2 to Section 3, explicitly explaining how the observed competition motivates our two-stage perturbation detector.

Evaluation and mechanism

We agree that correctness-based AUROC alone cannot explicitly distinguish errors made despite accessible knowledge from those where knowledge availability is not established. To directly address this concern, we stratified the full ScientistQA evaluation using independent closed-book knowledge probes. An item is classified as knowledge- accessible only when both binary fact accuracy and pairwise owner accuracy exceed 0.5; otherwise, it is classified as missing knowledge.

Using the same unified detector, the same fixed 0.5 threshold, and the same right-person-grouped out-of-fold predictions for both strata, the detector recalls 81.7% of knowledge-accessible errors and 71.3% of missing knowledge errors. Thus, its performance is not driven primarily by detecting questions for which the model lacks accessible knowledge.

These results directly evaluate detection performance across knowledge-availability conditions and support our central hypothesis: keyword perturbation is particularly effective at identifying errors that occur despite accessible knowledge, rather than merely detecting failures caused by unavailable knowledge.

Perturbation validity.

To examine whether our findings depend on the choice of perturbation operator, we conducted a controlled experiment on 170 questions, covering 521 attribute spans from 101 independent person/source groups.

We compared our vocabulary-mean replacement with (i) physical token deletion followed by retokenization, (ii) query-specific mean embedding replacement computed from non-target tokens, and (iii) random replacement vectors sampled from a diagonal Gaussian matched to the embedding table's coordinate-wise mean and variance, averaging five independent draws per span. Evaluation used three repetitions of group-aware five-fold cross-validation, with only perturbation-effect features and no unperturbed answer margin.

The likelihood-error AUROCs were 0.644 for token deletion, 0.673 for query-specific mean replacement, and 0.579 for matched-random replacement, compared with 0.794 for our original vocabulary-mean-based detector on full ScientistQA (Table 4).

We further examined whether neutralizing one answer-associated keyword increases attention to the competing keyword. Attention was measured from teacher-forced answer-token positions to the competing keyword span. Relative to width- and position-matched placebo interventions, neutralizing the wrong-answer keyword increased attention to the correct-answer keyword by 1.56×10−41.56\times10^{-4}. Symmetrically, neutralizing the correct-answer keyword increased attention to the wrong-answer keyword by 1.37×10−41.37\times10^{-4}. These results indicate that targeted neutralization redirects model processing toward competing contextual evidence rather than merely introducing nonspecific embedding noise.

These results suggests that targeted keyword perturbations capture information beyond nonspecific embedding noise and supports the effectiveness of our proposed perturbation strategy.

Alternative-answer sensitivity

We performed a matched sensitivity experiment on probe-known ScientistQA. For each response, we replaced the original competing candidate with the alternative from a semantically matched question; the replacement was inserted into the displayed option set, and pairs were formed within the same correctness stratum. We then reran the complete two-stage perturbation pipeline. The three swapped alternatives realization yielded an AUROC of 0.886, 0.875, and 0.870. Although this represents a decrease from the original performance of approximately 0.90, it remains competitive with existing detection methods. These results demonstrate that the perturbation signal remains informative under substantial changes in alternative selection.


We appreciate the reviewer’s suggestions. If you find the clarifications address the concerns, we kindly request reconsideration of the score.

Thank you for the detailed response and the additional experiments. The directional interventions provide useful evidence that competing keyword associations can influence answer selection.

My main concern remains the scope of the mechanistic claim. The experiments show how keyword associations can affect answers at inference time, while their origins in the evaluated models’ pretraining remain unclear. I am also still unsure how much keyword competition explains errors when the relevant knowledge is accessible, or whether the detector specifically captures that mechanism.

I appreciate the authors’ follow-up and will keep my original rating.

We respectfully yet firmly object to the reviewer’s latest assessment, as well as their decision to maintain the ratings of Soundness 1 and Clear Rejection (1).

1. We have thoroughly addressed and resolved all four specific concerns raised by the reviewer in the previous round, yet the reviewer failed to provide any substantive evaluation of our responses. In our previous rebuttal, we addressed each of the reviewer’s four concerns with clear explanations and new experimental evidence:

  • Connection between the mechanism and the detection method: Through Tables 8, 9, and 22, we explicitly established the logical link between keyword associations, the impact of competing associations on hallucinations, and our perturbation-based detection method.
  • Knowledge accessibility: Through independent knowledge probing and stratified evaluation, we verified the detector’s ability to identify errors that occur when knowledge is accessible.
  • Perturbation validity: As requested by the reviewer, we conducted experimental comparisons across three alternative perturbation methods and added an attention reallocation analysis.
  • Sensitivity to alternative answers: We performed three independent alternative-replacement experiments, achieving AUROCs of 0.886, 0.875, and 0.870, thereby providing clear evidence of robustness. We believe that all four concerns were addressed thoroughly and directly. However, in their latest response, the reviewer neither evaluated any of these newly added experiments nor pointed out any technical flaws in our rebuttal. Instead, they shifted to further demands regarding the origins of original pretraining associations and the explanatory coverage of the mechanism—specific requirements that were not explicitly raised in the initial review. We strongly object to this reviewing practice, which avoids engaging in scientific discussion based on the authors' actual responses and instead continually shifts the focus of discussion and introduces new demands for proof simply to maintain an initial negative evaluation.

2. The reviewer’s further questions exceed the actual claims of the paper and disregard existing controlled experimental evidence. What we propose is an empirically supported probabilistic explanatory framework, rather than a claim that we can precisely trace every association in the original pretraining data or explain all hallucinations. Table 8 demonstrates, via controlled training-frequency interventions, that association frequency systematically influences answer preferences; Table 9 demonstrates, via targeted keyword interventions, that competing associations affect the generation of correct answers. These are controlled intervention experiments, not mere inference-time correlational observations. Meanwhile, our two-stage detection method has been independently validated through systematic ablation studies and cross-domain transfer experiments. The effectiveness of the algorithm does not require proving that keyword competition is the sole cause of all hallucinations, or that the detector can only capture this single mechanism.

3. These concerns are entirely insufficient to justify ratings of Soundness 1 and Clear Rejection (1). The reviewer had previously explicitly acknowledged the experimental evidence in Tables 8–9 and 22, and we subsequently provided comprehensive responses to all four of their specific requests. Nevertheless, the reviewer has still not identified any theoretical errors, invalid experiments, or fundamental methodological flaws sufficient to refute the paper’s main conclusions. We strongly object to the reviewer maintaining the lowest ratings based on further demands that go beyond the paper's actual claims, without providing a substantive evaluation of the new evidence offered in response to their original concerns. We believe that such an extremely negative evaluation lacks scientific justification commensurate with its severity and fails to fairly reflect the paper's validated mechanistic evidence, theoretical analysis, and algorithmic contributions.

We are deeply concerned that the reviewer has maintained the lowest possible ratings without substantively engaging with our detailed responses and additional experimental evidence. We believe this raises serious questions about the fairness and scientific rigor of the review process. We will therefore bring these concerns to the Area Chair and request an independent assessment of the review’s quality and justification.

Reviewer_xKSGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses the important hallucination detection problem. It uses keyword removal to detect hallucinations. The idea is that a two-stage keyword perturbation method can detect hallucinations. Keywords are parts of the prompt that removing them can change the model's answer. This work shows higher result compered to other known hallucination detection methods. In addition, this work defines a hallucination taxonomy in relation to the model’s knowledge.

Critical strengths
  1. This work shows interesting advantages of keyword removal to detect hallucination compared to other methods such as certainty methods.
  2. Define a hallucination taxonomy based on knowledge.
Critical weaknesses
  1. The claims lack support and ablation. E.g in Section 2 throughout. See comments below.
  2. In section 2.1 the paper defines knowledge but LLM knowledge is a broad field. The paper lacks comparison and evaluation against other knowledge defenitions, which is later used in the paper (e.g. Gekhman et al., 2025 Inside-Out: Hidden Factual Knowledge in LLMs, Simhi et al., 2025 - Trust me, I’m wrong: LLMs hallucinate with certainty despite knowing the answer)
  3. The paper defines a hallucination taxonomy, however hallucination texonomy is a broad field. but the paper lacks a broad related work section and comparison with hallucination taxonomies from the literature (e.g (Ji et al., 2023; Sharma et al., 2023; Kalai and Vempala, 2023) and even a knowledge-specific hallucination taxonomy (Simhi et al., 2025) Ji et al., 2023 - Survey of hallucination in natural language generation Sharma et al., 2023 - Towards understanding sycophancy in language models Kalai and Vempala, 2023 - Calibrated language models must hallucinate Simhi et al., 2025 - Trust me, I’m wrong: LLMs hallucinate with certainty despite knowing the answer
Questions for authors
  1. Table 1 seems to use a model to calculate likelihood of preferred answers, yet I do not see any specification of the model or setup.
  2. Table 1, how did the authors evaluate responses? Greedy generation? What length? What model?
  3. Delta f is explained in 210, but the table discussion is already in 198.
  4. The delta f and u value from Table 1 need some calibration. E.g is 0.319 vs 0.337 a large difference or not? The authors seem to claim it is a large difference, but the raw numbers do not seem highly different.
  5. Line 228 the definition of prior is missing. Which is crucial to understand this part.
  6. Which version of the models is used? Which size?
  7. Section 2.1III misses key information about the experiment, such as the specific models, why the last four transformer blocks for training? Why only 50 training examples? Also, how should we read the slope?
  8. Section 2.1III: What constitutes a large slope? Ablation is missing.
  9. Section 2.1III - a comparison to the literature is missing (e.g. Kang & Choi (2023)). Training changes model preference to answer. How is it novel compared to the literature?
  10. Line 258 “Real prompts” - what are Real prompts? I do not see a definition.
  11. Line 264 discusses types of keyword interactions but missing a definition for Redundancy, synergy, or Competition.
  12. Table 3- it seems that most keyword interactions are “other”; what does this mean for the paper's conclusions?
Reviewer_RfsXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper focus on why LLMs produce incorrect answers even when the required knowledge is available. The authors propose a latent-key framework to explain how competing entity–keyword associations affect answer selection. Based on this analysis, they introduce a two-stage keyword perturbation method for hallucination detection. Experiments across multiple models and benchmarks demonstrate promising detection performance and generalisation ability.

Critical strengths
  1. The paper studies hallucinations from the perspective of competing knowledge associations. The motivation is clear and meaningful.
  2. The paper connects theoretical analysis, empirical observations, and hallucination detection in a coherent framework.
  3. The experiments results are promising.
Critical weaknesses
  1. The experiments support the proposed explanation, but do not directly verify the existence or competition of latent keys. The authors should clarify the scope of their theoretical claims.
  2. Table 22 shows relatively small improvements from perturbation features for Llama and Mistral. The authors should better clarify the unique contribution of the proposed perturbation mechanism. Or do more supporting experiments.
  3. The main detector requires an alternative answer, which may not be available in real-world hallucination detection. Although the appendix includes a candidate-free variant, its performance is lower on several benchmarks.
  4. The paper discusses existing perturbation-based hallucination detectors, such as Shaking to Reveal, but does not directly compare against them. Such comparisons would help establish the effectiveness of the proposed method. In addition, since the method focuses on entity–keyword associations, the authors are encouraged to discuss recent related studies, such as LUMINA, EAEV, and HalluEntity, to better clarify the novelty and differences of their approach.

[1] LUMINA: Detecting hallucinations in RAG system with context–knowledge signals. (2026)

[2] Evidence-Aligned Entity Verification for Hallucination Detection in Retrieval-Augmented Generation. (2026)

[3] Halluentity: Benchmarking and understanding entity-level hallucination detection. (2025)

Questions for authors

See my Weakness.

2 comments

Thank you for the detailed response. The results in Tables 8–9 and the transfer results in Table 22 provide some additional support for the proposed method. However, most of my concerns remain unresolved, particularly regarding the evidence for the latent-key mechanism and the sensitivity to alternative answer selection. I still encourage the authors to discuss the relevant studies in more detail, even if they choose not to conduct direct experimental comparisons. For now, I will maintain my original score.

Thank you for your follow-up and for acknowledging the additional evidence provided by Tables 8–9 and 22. We appreciate your constructive feedback and would like to provide two further experimental results addressing your remaining concerns.


Additional evidence for keyword competition.

To further investigate the proposed mechanism, we conducted an additional attention analysis examining whether neutralizing one answer-associated keyword increases attention to its competing keyword.

Specifically, we measured attention from teacher-forced answer-token positions to the competing keyword span, using width- and position-matched placebo interventions as controls. Neutralizing the wrong-supporting keyword increased attention to the correct-supporting keyword by 1.56×10−41.56\times10^{-4} relative to the matched control. Conversely, neutralizing the correct-supporting keyword increased attention to the wrong-supporting keyword by 1.37×10−41.37\times10^{-4}.

These directionally consistent attention changes provide additional evidence that suppressing one influential association can increase the model's attention to competing contextual evidence.

Together with the controlled frequency interventions (Table 8) and the opposite effects on answer generation (Table 9), these findings provide complementary behavioral and attention-based evidence consistent with keyword competition. We acknowledge that they do not establish the existence of discrete latent keys as identifiable neural structures, and we will clarify this theoretical boundary in the revised manuscript.

Additional alternative-answer sensitivity experiments.

We conducted a controlled alternative-swapping experiment on probe-known ScientistQA responses. For each response, we held the original question and generated answer fixed while replacing the competing candidate with an alternative from a semantically matched question within the same correctness stratum. We then reran the complete two-stage perturbation pipeline.

Three independently constructed alternative-swap realizations yielded AUROCs of 0.886, 0.875, and 0.870, respectively. These results demonstrate that the perturbation signal remains informative across different alternative selections, rather than depending on a single competing candidate.

Related work.

We also appreciate your suggestion regarding related studies. We will expand the discussion of Shaking to Reveal, LUMINA, EAEV, and HalluEntity, emphasizing the distinctions between representation sensitivity, evidence-based entity verification, entity-level hallucination detection, and our directional keyword-competition framework.


We thank you again for your careful consideration. We hope these additional experiments and planned clarifications help address your remaining concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.