- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose to split "hallucinations" into two types: those that occur due to lack of knowledge in the model. The authors define these two types. They then train a model L', based on a model L, the exhibit verifiable deceptive behaviors and conduct various analysis to understand the underlying mechanism for these behaviors.
Critical strengths
-
The problem is interesting and important: understanding why models may sometimes generate incorrect answers even though they internally "know" the answer.
-
The L' construction is interesting and novel (though I have reservations regarding its validity for some of the conclusions - see below)
Critical weaknesses
-
The problem setting is already defined and explored in at least one prior work: "Distinguishing Ignorance from Error in LLM Hallucinations". As far as I can see, it includes the exact same definitions and problem layout.
-
"CL(x) = 1 ⇐⇒ KL(x) = 1" - this is a strong assumption, that is most likely incorrect, since if it is correct, then there are no deceptions in L.
-
Using DPO to introduce specific behavior perhaps does not remove existing knowledge, but it may introduce new internal knowledge mechanisms/flows, and hence the trained L' may not reflect natural LLM behavior for hallucinations/deceptions but a synthetic one, due to the specific training.
-
The focus on training L' on altered correct answers is not addressing the cases in which L answers incorrectly for various reasons. Thus, there may be internal patterns that are completely missed just because L' is inspected only on successful answers under L.
-
I found sec 5 and 6 hard to follow. I didn't understand the motivations and the underlying insights. For example, in sec 6's takeway: "the failure to recover hallucinated outputs under the same intervention highlights a fundamental mechanism-level distinction, indicating that hallucination arises from knowledge absence (K = 0) and cannot be resolved through behavior-level control alone" - it seems obvious, since the def of hallucination is lack of knowledge, and any steering that does not introduce new knowledge cannot resolve it.
Questions for authors
-
What is the delta between this work and "Distinguishing Ignorance from Error in LLM Hallucinations"? Are there any mehtods introduced in this prior work (and perhaps others the cite it) that may be as good as the method proposed in this paper?
-
How would you verify that the insights you have in L' are transferrable to L?
5 comments
This distinction is reflected in our experimental design and analyses. Starting from a pretrained Base model, we apply DPO to construct different behavioral regimes under initially verified knowledge conditions. We then combine bottleneck representation analysis, SAE-based feature decomposition, and angular activation editing to investigate their internal organization and causal controllability. Our results reveal a dominant representational boundary associated with knowledge availability, alongside additional sparse and entity-dependent structures associated with behavioral expression. Furthermore, knowledge-preserving deceptive behaviors can be partially reversed through targeted activation editing, whereas hallucinated behaviors are not comparably recoverable under the same intervention. These findings provide complementary representational and causal evidence for the mechanistic differences between the two types of failures.
Regarding whether existing methods could serve as alternatives, we believe that these approaches address complementary research objectives. WACK and its associated classifiers are designed for error identification, while the approaches in [2–6] investigate specific aspects of deception, knowledge awareness, truth representations, or hallucination mechanisms. Our approach complements these methods by integrating controlled behavioral construction, sparse feature analysis, and causal activation intervention within a unified framework. We will further clarify these distinctions and acknowledge the contributions of these closely related studies in the revised manuscript.
[1] A. Simhi et al. Distinguishing Ignorance from Error in LLM Hallucinations. arXiv, 2024.
[2] W. Yang and G. Buzsáki. Interpretability of LLM Deception: Universal Motif. ICLR, 2025.
[3] J. Ferrando et al. Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models. ICLR, 2025.
[4] S. Marks and M. Tegmark. The Geometry of Truth: Emergent Linear Structure in LLM Representations of True/False Datasets. COLM, 2024.
[5] L. Yu et al. Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations. Findings of EMNLP, 2024.
[6] P. Suresh et al. From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers. NeurIPS, 2025.
R2. Knowledge Verification and Experimental Assumptions
The criterion serves as an operational definition for establishing a knowledge-verified subset in our controlled setting, rather than a universal claim that output correctness and latent knowledge are equivalent for arbitrary inputs. Specifically, we consider single-hop, entity-centric factual QA tasks, where the base model's ability to produce a factually correct answer provides positive behavioral evidence that the corresponding knowledge is accessible under the given input conditions. This criterion allows us to establish a controlled starting point for investigating how subsequent preference optimization affects knowledge expression. We acknowledge that output correctness alone is not an infallible measurement of latent knowledge. Therefore, we further verify knowledge accessibility after DPO instead of simply assuming that the original knowledge is necessarily preserved.
The absence of deception in the original Base model is intentional, as our objective is to induce and investigate knowledge-preserving behavioral suppression through a controlled intervention, rather than to identify naturally occurring deception in . We first construct from questions that answers correctly, thereby establishing the initial knowledge-verified condition. We then apply DPO to induce two distinct behaviors on this subset: evading the correct answer () and producing an intentionally incorrect answer (). Importantly, we do not infer knowledge preservation solely from the original correctness of . Instead, we independently apply multiple jailbreak strategies to without introducing new factual information, retaining only samples for which the model can recover the correct answer. As reported in the paper, the combined verification strategies achieve a 75.4% success rate on the evasion candidate pool. This post-training accessibility verification directly addresses the possibility that DPO may have altered or eliminated the relevant knowledge.
We also clarify that an incorrect response from does not, by itself, establish that the corresponding knowledge is genuinely absent. In our experimental construction, is used to identify samples outside the initially knowledge-verified subset, rather than to establish a universally valid characterization of the model's latent knowledge. Accordingly, the corresponding hallucination cases should be interpreted under this operational criterion. We will make this distinction more explicit in the revised manuscript to avoid overinterpreting incorrect base-model outputs. This clarification does not affect our controlled construction of and , which is based on initially verified knowledge and independently validated knowledge accessibility after DPO.
R3. Validity of the DPO-based Construction
The DPO-based construction provides a controlled experimental setting for investigating how preference-based post-training reshapes behavioral expression under initially verified knowledge conditions, rather than an attempt to reproduce the natural distribution of deception in deployed LLMs. We deliberately start from the LLaMA-3.1-8B Base checkpoint, without prior instruction tuning or RLHF, to isolate the effects of our explicit preference-based intervention from potentially confounding post-training histories. DPO serves as a concrete and controllable instance of preference-based post-training, allowing us to systematically alter response preferences while keeping the initial model and underlying factual questions fixed. This design is also motivated by existing studies showing that preference-based alignment can induce unintended behavioral changes, including strategic misrepresentation, evasion, and manipulation [7–9]. Therefore, our construction is not an arbitrary manipulation introduced solely for analysis, but a controlled investigation of behavior changes arising from preference optimization.
Importantly, we do not assume that DPO leaves the model's internal knowledge representations or computational pathways unchanged. Preference optimization can affect model generalization, response diversity, and the distribution of generated outputs [10, 11]. Such changes are precisely why we independently verify knowledge accessibility after training and subsequently investigate the resulting internal representations and causal intervention responses. In other words, our objective is not to assume that retains exactly the same internal mechanisms as , but to examine how its internal organization changes when preference optimization induces knowledge-preserving behavioral suppression. The resulting representations and intervention effects are empirical findings of the study, rather than properties assumed by the construction.
Regarding the transferability of our findings, we distinguish the scope of our empirical claims from the broader significance of the proposed knowledge–behavior framework. We do not claim that the specific SAE features or activation-steering directions identified in can be directly transferred to arbitrary models, nor that all naturally occurring deception follows the same mechanism. Instead, our experiments establish that, under a controlled preference-based intervention, knowledge-preserving behavioral suppression and knowledge-unverified hallucination exhibit distinguishable representational structures and intervention responses. This provides mechanistic evidence for the proposed distinction and motivates further investigation across different models and post-training procedures. We believe that requiring the specific internal mechanisms discovered in one controlled setting to transfer directly to all naturally trained LLMs would impose a substantially broader generalization requirement than the scientific objective of this study.
Our experimental construction also includes a separate hallucination comparison group, rather than restricting the analysis to behaviors induced on initially correct answers. Specifically, we construct from the initial knowledge-unverified subset , while , , and originate from . The final dataset contains 900 normal cases, 822 knowledge-verified evasion cases, 1,391 knowledge-verified incorrect-answer cases, and 900 hallucination comparison cases. This enables us to compare knowledge-preserving behavioral suppression with errors arising outside the initially verified knowledge subset. We acknowledge that our current study focuses on single-hop, entity-centric factual QA across four entity types, and does not aim to cover every possible failure mode, such as reasoning failures or uncertainty expression when knowledge is unavailable. We will clarify this scope and the operational interpretation of the hallucination comparison group in the revised manuscript.
We sincerely thank the reviewer NuuF for the careful assessment and constructive feedback. We particularly appreciate the reviewer NuuF’s recognition of the importance of understanding why LLMs may produce incorrect answers despite retaining the relevant knowledge, as well as the novelty of our controlled construction of L'. These observations align with the central motivation of our work.
We understand the reviewer NuuF’s concerns as revolving around four main issues:
(1) Novelty and distinction from prior work: Whether our problem formulation and contributions substantially differ from prior studies on distinguishing ignorance from error, particularly Simhi et al.
(2) Knowledge verification and experimental assumptions: Whether the criterion C_L(x) = 1 iff K_L(x) = 1 is sufficiently justified and whether it leads to an overly restrictive characterization of knowledge and deception.
(3) Validity of the DPO-based construction: Whether the behaviors induced in L' provide a meaningful setting for studying knowledge–behavior dissociation, whether the resulting findings are informative beyond the specific intervention, and whether our experimental construction overlooks other failure cases.
(4) Mechanistic insights and causal interpretation: Whether the representation analysis and activation interventions in Sections 5 and 6 reveal substantive mechanistic differences, rather than merely confirming consequences implied by our definitions.
We would like to emphasize that the central contribution of our work is not simply to distinguish two types of hallucinations. Rather, we establish a controlled framework to investigate the relationship between knowledge availability and behavioral expression, and further characterize their distinct internal representations and causal controllability. This objective goes beyond the classification and detection of knowledge-related errors in prior work. We address the reviewer NuuF’s concerns below by clarifying the distinctions from closely related studies, establishing the rationale for our experimental design, and explaining the insights obtained from our representation and intervention analyses.
R1. Novelty and Distinction from Prior Work
The fundamental novelty of our work lies in moving beyond distinguishing knowledge absence from knowledge-preserving errors to investigating how knowledge availability and behavioral expression are internally represented and causally controllable under a controlled DPO intervention. We appreciate the reviewer NuuF for pointing out the related work by Simhi et al. [1], which makes an important contribution by distinguishing hallucinations caused by insufficient knowledge (HK−) from incorrect responses despite possessing relevant knowledge (HK+). While this distinction is closely related to the conceptual foundation of our work, the two studies pursue different research objectives. Simhi et al. primarily focus on constructing the WACK dataset, identifying these error types through hidden-state classification, and exploring prompt-based mitigation. In contrast, our work investigates the internal organization and causal controllability of different knowledge–behavior states.
Several recent studies have also made important contributions to the mechanistic understanding of knowledge representation, deception, and hallucination. Yang and Buzsáki [2] investigate the internal representations of induced deceptive behaviors and their causal relevance through activation interventions. Ferrando et al. [3] identify SAE features associated with entity-level knowledge awareness, while Marks and Tegmark [4] study the geometric structure of truth-related representations. Yu et al. [5] and Suresh et al. [6] further examine the internal mechanisms underlying hallucinations. We appreciate these advances, which provide important foundations for mechanistic investigations of LLM behaviors. Our work differs in jointly investigating knowledge availability and behavioral expression across multiple failure states, rather than examining knowledge awareness, deception, or hallucination in isolation.