- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates why LLMs can produce hallucinations even when the knowledge required to answer correctly is available, through the lens of latent-key competition. The authors suggest that competing entity–keyword associations acquired during pretraining can lead models to select incorrect answers, and provide empirical analyses of these effects. Based on these observations, they propose a two-stage keyword perturbation method for hallucination detection, which identifies influential keywords and examines how model predictions change after their removal.
Critical strengths
S1. Interesting perspective on the mechanisms underlying hallucinations.
Rather than focusing solely on detecting hallucinations, this paper attempts to explain why models produce incorrect answers despite having access to the relevant knowledge. The latent-key perspective offers an interesting interpretation of hallucinations through competition among entity–keyword associations. The authors further investigate this hypothesis through keyword perturbation, controlled frequency exposure, and analyses of interactions among multiple keywords.
S2. Strong empirical performance in hallucination detection.
The proposed two-stage keyword perturbation method achieves competitive or superior performance compared to existing uncertainty-based and representation-based approaches across several benchmarks, including ScientistQA, TriviaQA, and DROP. In particular, it achieves an AUROC of 0.910 on probe-known ScientistQA and demonstrates promising generalization to unseen entity domains, supporting the effectiveness of the proposed detection signals.
Critical weaknesses
W1. Insufficient evidence that latent-key competition is the underlying cause of hallucinations.
The paper shows that entity–keyword associations influence answer selection, but this does not necessarily establish them as the underlying cause of hallucinations. For example, the Eric Clapton case could also result from a failure to properly handle the explicit negative constraint. Moreover, Table 1 shows strong bindings in both correct and incorrect responses, while Table 3 does not fully distinguish associative bias from logical reasoning failures. Further experiments separating these factors would strengthen the central claim.
W2. Limited connection between the limitations of existing approaches and the proposed method.
The introduction highlights limitations of existing hallucination detectors, but it remains unclear how the proposed method directly addresses them. Although the experimental results demonstrate strong detection performance, there is limited analysis of whether the method succeeds specifically in cases where existing approaches fail. More targeted comparisons would strengthen the motivation.
W3. Insufficient justification for using model-wide mean embeddings.
The method replaces keyword embeddings with the model-wide mean embedding, but it is unclear whether this selectively removes keyword associations or introduces unintended perturbation artifacts. Would a more context-dependent replacement, such as the mean embedding of the original query, be more appropriate? Comparisons with alternative strategies, such as query-specific mean embeddings or token deletion, would help validate this design choice.
W4. Limited applicability to black-box LLMs.
The proposed method requires access to the generator's internal embeddings, hidden states, and candidate likelihoods, which limits its applicability to black-box LLMs accessible only through APIs. This dependency may restrict its practical deployment in real-world settings.
Questions for authors
Q1. How sensitive is the proposed detector to the choice of alternative answer?
The method relies on relative preference scores between the selected and alternative answers, but the choice of alternative may affect detection performance. Although Appendix C.3 evaluates a Pred-only variant, could the authors clarify how sensitive the full method is to different alternative candidates, particularly in open-ended settings?
4 comments
We thank the reviewer for detailed feedback. We address the concerns below.
Presentation and logical connection
We will revise Sections 2 and 3 to present a clearer progression: (1) Keyword associations influence answer selection, supported by the controlled frequency–dose experiment (Table 8, Table 2); (2) Competing associations can contribute to hallucinations, supported by the directional intervention results in Table 9, where neutralizing wrong-supporting keywords increases correct-answer generation, while neutralizing right-supporting keywords decreases it; and (3) These perturbation responses provide useful detection signals, supported by the matched ablation in Table 22, which shows improvements from adding keyword-perturbation features, particularly under cross-domain transfer. We will move the key findings from Tables 9 and 22 into the main text and strengthen the transition from Section 2 to Section 3, explicitly explaining how the observed competition motivates our two-stage perturbation detector.
Evaluation and mechanism
We agree that correctness-based AUROC alone cannot explicitly distinguish errors made despite accessible knowledge from those where knowledge availability is not established. To directly address this concern, we stratified the full ScientistQA evaluation using independent closed-book knowledge probes. An item is classified as knowledge- accessible only when both binary fact accuracy and pairwise owner accuracy exceed 0.5; otherwise, it is classified as missing knowledge.
Using the same unified detector, the same fixed 0.5 threshold, and the same right-person-grouped out-of-fold predictions for both strata, the detector recalls 81.7% of knowledge-accessible errors and 71.3% of missing knowledge errors. Thus, its performance is not driven primarily by detecting questions for which the model lacks accessible knowledge.
These results directly evaluate detection performance across knowledge-availability conditions and support our central hypothesis: keyword perturbation is particularly effective at identifying errors that occur despite accessible knowledge, rather than merely detecting failures caused by unavailable knowledge.
Perturbation validity.
To examine whether our findings depend on the choice of perturbation operator, we conducted a controlled experiment on 170 questions, covering 521 attribute spans from 101 independent person/source groups.
We compared our vocabulary-mean replacement with (i) physical token deletion followed by retokenization, (ii) query-specific mean embedding replacement computed from non-target tokens, and (iii) random replacement vectors sampled from a diagonal Gaussian matched to the embedding table's coordinate-wise mean and variance, averaging five independent draws per span. Evaluation used three repetitions of group-aware five-fold cross-validation, with only perturbation-effect features and no unperturbed answer margin.
The likelihood-error AUROCs were 0.644 for token deletion, 0.673 for query-specific mean replacement, and 0.579 for matched-random replacement, compared with 0.794 for our original vocabulary-mean-based detector on full ScientistQA (Table 4).
We further examined whether neutralizing one answer-associated keyword increases attention to the competing keyword. Attention was measured from teacher-forced answer-token positions to the competing keyword span. Relative to width- and position-matched placebo interventions, neutralizing the wrong-answer keyword increased attention to the correct-answer keyword by . Symmetrically, neutralizing the correct-answer keyword increased attention to the wrong-answer keyword by . These results indicate that targeted neutralization redirects model processing toward competing contextual evidence rather than merely introducing nonspecific embedding noise.
These results suggests that targeted keyword perturbations capture information beyond nonspecific embedding noise and supports the effectiveness of our proposed perturbation strategy.
Alternative-answer sensitivity
We performed a matched sensitivity experiment on probe-known ScientistQA. For each response, we replaced the original competing candidate with the alternative from a semantically matched question; the replacement was inserted into the displayed option set, and pairs were formed within the same correctness stratum. We then reran the complete two-stage perturbation pipeline. The three swapped alternatives realization yielded an AUROC of 0.886, 0.875, and 0.870. Although this represents a decrease from the original performance of approximately 0.90, it remains competitive with existing detection methods. These results demonstrate that the perturbation signal remains informative under substantial changes in alternative selection.
We appreciate the reviewer’s suggestions. If you find the clarifications address the concerns, we kindly request reconsideration of the score.
Thank you for the detailed response and the additional experiments. The directional interventions provide useful evidence that competing keyword associations can influence answer selection.
My main concern remains the scope of the mechanistic claim. The experiments show how keyword associations can affect answers at inference time, while their origins in the evaluated models’ pretraining remain unclear. I am also still unsure how much keyword competition explains errors when the relevant knowledge is accessible, or whether the detector specifically captures that mechanism.
I appreciate the authors’ follow-up and will keep my original rating.
We respectfully yet firmly object to the reviewer’s latest assessment, as well as their decision to maintain the ratings of Soundness 1 and Clear Rejection (1).
1. We have thoroughly addressed and resolved all four specific concerns raised by the reviewer in the previous round, yet the reviewer failed to provide any substantive evaluation of our responses. In our previous rebuttal, we addressed each of the reviewer’s four concerns with clear explanations and new experimental evidence:
- Connection between the mechanism and the detection method: Through Tables 8, 9, and 22, we explicitly established the logical link between keyword associations, the impact of competing associations on hallucinations, and our perturbation-based detection method.
- Knowledge accessibility: Through independent knowledge probing and stratified evaluation, we verified the detector’s ability to identify errors that occur when knowledge is accessible.
- Perturbation validity: As requested by the reviewer, we conducted experimental comparisons across three alternative perturbation methods and added an attention reallocation analysis.
- Sensitivity to alternative answers: We performed three independent alternative-replacement experiments, achieving AUROCs of 0.886, 0.875, and 0.870, thereby providing clear evidence of robustness. We believe that all four concerns were addressed thoroughly and directly. However, in their latest response, the reviewer neither evaluated any of these newly added experiments nor pointed out any technical flaws in our rebuttal. Instead, they shifted to further demands regarding the origins of original pretraining associations and the explanatory coverage of the mechanism—specific requirements that were not explicitly raised in the initial review. We strongly object to this reviewing practice, which avoids engaging in scientific discussion based on the authors' actual responses and instead continually shifts the focus of discussion and introduces new demands for proof simply to maintain an initial negative evaluation.
2. The reviewer’s further questions exceed the actual claims of the paper and disregard existing controlled experimental evidence. What we propose is an empirically supported probabilistic explanatory framework, rather than a claim that we can precisely trace every association in the original pretraining data or explain all hallucinations. Table 8 demonstrates, via controlled training-frequency interventions, that association frequency systematically influences answer preferences; Table 9 demonstrates, via targeted keyword interventions, that competing associations affect the generation of correct answers. These are controlled intervention experiments, not mere inference-time correlational observations. Meanwhile, our two-stage detection method has been independently validated through systematic ablation studies and cross-domain transfer experiments. The effectiveness of the algorithm does not require proving that keyword competition is the sole cause of all hallucinations, or that the detector can only capture this single mechanism.
3. These concerns are entirely insufficient to justify ratings of Soundness 1 and Clear Rejection (1). The reviewer had previously explicitly acknowledged the experimental evidence in Tables 8–9 and 22, and we subsequently provided comprehensive responses to all four of their specific requests. Nevertheless, the reviewer has still not identified any theoretical errors, invalid experiments, or fundamental methodological flaws sufficient to refute the paper’s main conclusions. We strongly object to the reviewer maintaining the lowest ratings based on further demands that go beyond the paper's actual claims, without providing a substantive evaluation of the new evidence offered in response to their original concerns. We believe that such an extremely negative evaluation lacks scientific justification commensurate with its severity and fails to fairly reflect the paper's validated mechanistic evidence, theoretical analysis, and algorithmic contributions.
We are deeply concerned that the reviewer has maintained the lowest possible ratings without substantively engaging with our detailed responses and additional experimental evidence. We believe this raises serious questions about the fairness and scientific rigor of the review process. We will therefore bring these concerns to the Area Chair and request an independent assessment of the review’s quality and justification.
Thank you for the response. I acknowledge that the interventions in Tables 8–9 support the contribution of competing associations, and that Table 22 demonstrates the added value of perturbation features. I also view the internal-access requirement as a limitation of applicability rather than soundness.
The following concerns remain:
Presentation and logical connection: Although Table 8 is summarized in Table 2, the key findings from Tables 9 and 22 also warrant inclusion in the main text. Keyword associations influencing answer selection, their competition causing errors, and the detector exploiting that competition are distinct claims. Their connections remain insufficiently established, making the transition from the mechanistic account to the detection method somewhat unclear.
Evaluation and mechanism: Correctness-based AUROC alone cannot distinguish errors despite accessible knowledge from errors due to missing knowledge. Probe-known ScientistQA partially addresses this, but an evaluation metric or protocol that explicitly separates these cases is needed to assess the central claim.
Perturbation validity: I understand the concern that query-specific mean embeddings may retain contextual information, but this should be tested empirically. Do the findings hold under token deletion, query-specific mean replacement, and random vectors matched to the embedding scale and variance? These controls would help distinguish semantic suppression from perturbation-specific artifacts.
Alternative-answer sensitivity: Independent construction of alternatives does not establish robustness to their choice. With the question and generated answer fixed, how much do detection scores and AUROC vary across different plausible alternatives? The Pred-only comparison tests detection without an alternative, rather than sensitivity to different alternatives.