acceptodds
Under review as a conference paper at ICLR 2027

Disentangling Deception and Hallucination Failures in LLMs

Abstract

Failures in large language models (LLMs) are often analyzed from a behavioral perspective, where incorrect outputs in factual question answering are commonly associated with missing knowledge. In this work, focusing on entity-based factual queries, we suggest that such a view may conflate different failure mechanisms, and propose an internal, mechanism-oriented perspective that separates *Knowledge Existence* from *Behavior Expression*. Under this formulation, hallucination and deception correspond to two qualitatively different failure modes that may appear similar at the output level but differ in their underlying mechanisms. To study this distinction, we construct a controlled environment for entity-centric factual questions in which knowledge is preserved while behavioral expression is selectively altered, enabling systematic analysis of four behavioral cases. We analyze these failure modes through representation separability, sparse interpretability, and inference-time activation steering.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large language models, Hallucination, Deception

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_NuuFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose to split "hallucinations" into two types: those that occur due to lack of knowledge in the model. The authors define these two types. They then train a model L', based on a model L, the exhibit verifiable deceptive behaviors and conduct various analysis to understand the underlying mechanism for these behaviors.

Critical strengths
  • The problem is interesting and important: understanding why models may sometimes generate incorrect answers even though they internally "know" the answer.

  • The L' construction is interesting and novel (though I have reservations regarding its validity for some of the conclusions - see below)

Critical weaknesses
  • The problem setting is already defined and explored in at least one prior work: "Distinguishing Ignorance from Error in LLM Hallucinations". As far as I can see, it includes the exact same definitions and problem layout.

  • "CL(x) = 1 ⇐⇒ KL(x) = 1" - this is a strong assumption, that is most likely incorrect, since if it is correct, then there are no deceptions in L.

  • Using DPO to introduce specific behavior perhaps does not remove existing knowledge, but it may introduce new internal knowledge mechanisms/flows, and hence the trained L' may not reflect natural LLM behavior for hallucinations/deceptions but a synthetic one, due to the specific training.

  • The focus on training L' on altered correct answers is not addressing the cases in which L answers incorrectly for various reasons. Thus, there may be internal patterns that are completely missed just because L' is inspected only on successful answers under L.

  • I found sec 5 and 6 hard to follow. I didn't understand the motivations and the underlying insights. For example, in sec 6's takeway: "the failure to recover hallucinated outputs under the same intervention highlights a fundamental mechanism-level distinction, indicating that hallucination arises from knowledge absence (K = 0) and cannot be resolved through behavior-level control alone" - it seems obvious, since the def of hallucination is lack of knowledge, and any steering that does not introduce new knowledge cannot resolve it.

Questions for authors
  • What is the delta between this work and "Distinguishing Ignorance from Error in LLM Hallucinations"? Are there any mehtods introduced in this prior work (and perhaps others the cite it) that may be as good as the method proposed in this paper?

  • How would you verify that the insights you have in L' are transferrable to L?

5 comments

We sincerely thank the reviewer NuuF for the careful assessment and constructive feedback. We particularly appreciate the reviewer NuuF’s recognition of the importance of understanding why LLMs may produce incorrect answers despite retaining the relevant knowledge, as well as the novelty of our controlled construction of L'. These observations align with the central motivation of our work.

We understand the reviewer NuuF’s concerns as revolving around four main issues:

(1) Novelty and distinction from prior work: Whether our problem formulation and contributions substantially differ from prior studies on distinguishing ignorance from error, particularly Simhi et al.

(2) Knowledge verification and experimental assumptions: Whether the criterion C_L(x) = 1 iff K_L(x) = 1 is sufficiently justified and whether it leads to an overly restrictive characterization of knowledge and deception.

(3) Validity of the DPO-based construction: Whether the behaviors induced in L' provide a meaningful setting for studying knowledge–behavior dissociation, whether the resulting findings are informative beyond the specific intervention, and whether our experimental construction overlooks other failure cases.

(4) Mechanistic insights and causal interpretation: Whether the representation analysis and activation interventions in Sections 5 and 6 reveal substantive mechanistic differences, rather than merely confirming consequences implied by our definitions.

We would like to emphasize that the central contribution of our work is not simply to distinguish two types of hallucinations. Rather, we establish a controlled framework to investigate the relationship between knowledge availability and behavioral expression, and further characterize their distinct internal representations and causal controllability. This objective goes beyond the classification and detection of knowledge-related errors in prior work. We address the reviewer NuuF’s concerns below by clarifying the distinctions from closely related studies, establishing the rationale for our experimental design, and explaining the insights obtained from our representation and intervention analyses.


R1. Novelty and Distinction from Prior Work

The fundamental novelty of our work lies in moving beyond distinguishing knowledge absence from knowledge-preserving errors to investigating how knowledge availability and behavioral expression are internally represented and causally controllable under a controlled DPO intervention. We appreciate the reviewer NuuF for pointing out the related work by Simhi et al. [1], which makes an important contribution by distinguishing hallucinations caused by insufficient knowledge (HK−) from incorrect responses despite possessing relevant knowledge (HK+). While this distinction is closely related to the conceptual foundation of our work, the two studies pursue different research objectives. Simhi et al. primarily focus on constructing the WACK dataset, identifying these error types through hidden-state classification, and exploring prompt-based mitigation. In contrast, our work investigates the internal organization and causal controllability of different knowledge–behavior states.

Several recent studies have also made important contributions to the mechanistic understanding of knowledge representation, deception, and hallucination. Yang and Buzsáki [2] investigate the internal representations of induced deceptive behaviors and their causal relevance through activation interventions. Ferrando et al. [3] identify SAE features associated with entity-level knowledge awareness, while Marks and Tegmark [4] study the geometric structure of truth-related representations. Yu et al. [5] and Suresh et al. [6] further examine the internal mechanisms underlying hallucinations. We appreciate these advances, which provide important foundations for mechanistic investigations of LLM behaviors. Our work differs in jointly investigating knowledge availability and behavioral expression across multiple failure states, rather than examining knowledge awareness, deception, or hallucination in isolation.

This distinction is reflected in our experimental design and analyses. Starting from a pretrained Base model, we apply DPO to construct different behavioral regimes under initially verified knowledge conditions. We then combine bottleneck representation analysis, SAE-based feature decomposition, and angular activation editing to investigate their internal organization and causal controllability. Our results reveal a dominant representational boundary associated with knowledge availability, alongside additional sparse and entity-dependent structures associated with behavioral expression. Furthermore, knowledge-preserving deceptive behaviors can be partially reversed through targeted activation editing, whereas hallucinated behaviors are not comparably recoverable under the same intervention. These findings provide complementary representational and causal evidence for the mechanistic differences between the two types of failures.

Regarding whether existing methods could serve as alternatives, we believe that these approaches address complementary research objectives. WACK and its associated classifiers are designed for error identification, while the approaches in [2–6] investigate specific aspects of deception, knowledge awareness, truth representations, or hallucination mechanisms. Our approach complements these methods by integrating controlled behavioral construction, sparse feature analysis, and causal activation intervention within a unified framework. We will further clarify these distinctions and acknowledge the contributions of these closely related studies in the revised manuscript.

[1] A. Simhi et al. Distinguishing Ignorance from Error in LLM Hallucinations. arXiv, 2024.

[2] W. Yang and G. Buzsáki. Interpretability of LLM Deception: Universal Motif. ICLR, 2025.

[3] J. Ferrando et al. Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models. ICLR, 2025.

[4] S. Marks and M. Tegmark. The Geometry of Truth: Emergent Linear Structure in LLM Representations of True/False Datasets. COLM, 2024.

[5] L. Yu et al. Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations. Findings of EMNLP, 2024.

[6] P. Suresh et al. From Noise to Narrative: Tracing the Origins of Hallucinations in Transformers. NeurIPS, 2025.


R2. Knowledge Verification and Experimental Assumptions

The criterion CL(x)=1  ⟺  KL(x)=1C_L(x)=1 \iff K_L(x)=1 serves as an operational definition for establishing a knowledge-verified subset in our controlled setting, rather than a universal claim that output correctness and latent knowledge are equivalent for arbitrary inputs. Specifically, we consider single-hop, entity-centric factual QA tasks, where the base model's ability to produce a factually correct answer provides positive behavioral evidence that the corresponding knowledge is accessible under the given input conditions. This criterion allows us to establish a controlled starting point for investigating how subsequent preference optimization affects knowledge expression. We acknowledge that output correctness alone is not an infallible measurement of latent knowledge. Therefore, we further verify knowledge accessibility after DPO instead of simply assuming that the original knowledge is necessarily preserved.

The absence of deception in the original Base model is intentional, as our objective is to induce and investigate knowledge-preserving behavioral suppression through a controlled intervention, rather than to identify naturally occurring deception in LL. We first construct DknowD_{\mathrm{know}} from questions that LL answers correctly, thereby establishing the initial knowledge-verified condition. We then apply DPO to induce two distinct behaviors on this subset: evading the correct answer (DKAD_{\mathrm{KA}}) and producing an intentionally incorrect answer (DKID_{\mathrm{KI}}). Importantly, we do not infer knowledge preservation solely from the original correctness of LL. Instead, we independently apply multiple jailbreak strategies to L′L' without introducing new factual information, retaining only samples for which the model can recover the correct answer. As reported in the paper, the combined verification strategies achieve a 75.4% success rate on the evasion candidate pool. This post-training accessibility verification directly addresses the possibility that DPO may have altered or eliminated the relevant knowledge.

We also clarify that an incorrect response from LL does not, by itself, establish that the corresponding knowledge is genuinely absent. In our experimental construction, CL(x)=0C_L(x)=0 is used to identify samples outside the initially knowledge-verified subset, rather than to establish a universally valid characterization of the model's latent knowledge. Accordingly, the corresponding hallucination cases should be interpreted under this operational criterion. We will make this distinction more explicit in the revised manuscript to avoid overinterpreting incorrect base-model outputs. This clarification does not affect our controlled construction of DKAD_{\mathrm{KA}} and DKID_{\mathrm{KI}}, which is based on initially verified knowledge and independently validated knowledge accessibility after DPO.


R3. Validity of the DPO-based Construction

The DPO-based construction provides a controlled experimental setting for investigating how preference-based post-training reshapes behavioral expression under initially verified knowledge conditions, rather than an attempt to reproduce the natural distribution of deception in deployed LLMs. We deliberately start from the LLaMA-3.1-8B Base checkpoint, without prior instruction tuning or RLHF, to isolate the effects of our explicit preference-based intervention from potentially confounding post-training histories. DPO serves as a concrete and controllable instance of preference-based post-training, allowing us to systematically alter response preferences while keeping the initial model and underlying factual questions fixed. This design is also motivated by existing studies showing that preference-based alignment can induce unintended behavioral changes, including strategic misrepresentation, evasion, and manipulation [7–9]. Therefore, our construction is not an arbitrary manipulation introduced solely for analysis, but a controlled investigation of behavior changes arising from preference optimization.

Importantly, we do not assume that DPO leaves the model's internal knowledge representations or computational pathways unchanged. Preference optimization can affect model generalization, response diversity, and the distribution of generated outputs [10, 11]. Such changes are precisely why we independently verify knowledge accessibility after training and subsequently investigate the resulting internal representations and causal intervention responses. In other words, our objective is not to assume that L′L' retains exactly the same internal mechanisms as LL, but to examine how its internal organization changes when preference optimization induces knowledge-preserving behavioral suppression. The resulting representations and intervention effects are empirical findings of the study, rather than properties assumed by the construction.

Regarding the transferability of our findings, we distinguish the scope of our empirical claims from the broader significance of the proposed knowledge–behavior framework. We do not claim that the specific SAE features or activation-steering directions identified in L′L' can be directly transferred to arbitrary models, nor that all naturally occurring deception follows the same mechanism. Instead, our experiments establish that, under a controlled preference-based intervention, knowledge-preserving behavioral suppression and knowledge-unverified hallucination exhibit distinguishable representational structures and intervention responses. This provides mechanistic evidence for the proposed distinction and motivates further investigation across different models and post-training procedures. We believe that requiring the specific internal mechanisms discovered in one controlled setting to transfer directly to all naturally trained LLMs would impose a substantially broader generalization requirement than the scientific objective of this study.

Our experimental construction also includes a separate hallucination comparison group, rather than restricting the analysis to behaviors induced on initially correct answers. Specifically, we construct DhalD_{\mathrm{hal}} from the initial knowledge-unverified subset DunkD_{\mathrm{unk}}, while DKCD_{\mathrm{KC}}, DKAD_{\mathrm{KA}}, and DKID_{\mathrm{KI}} originate from DknowD_{\mathrm{know}}. The final dataset contains 900 normal cases, 822 knowledge-verified evasion cases, 1,391 knowledge-verified incorrect-answer cases, and 900 hallucination comparison cases. This enables us to compare knowledge-preserving behavioral suppression with errors arising outside the initially verified knowledge subset. We acknowledge that our current study focuses on single-hop, entity-centric factual QA across four entity types, and does not aim to cover every possible failure mode, such as reasoning failures or uncertainty expression when knowledge is unavailable. We will clarify this scope and the operational interpretation of the hallucination comparison group in the revised manuscript.

Reviewer_CfqROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates two failure modes in entity-centric factual question answering: hallucination and deception. It characterizes model behavior using two latent variables: Knowledge Existence (K) and Behavior Expression (B). Hallucination (K=0, B=1) occurs when the model lacks relevant knowledge but still answers, whereas deception (K=1, B=0) occurs when it possesses the knowledge but answers incorrectly or avoids answering.

Using LLaMA-3.1-8B-Instruct, the authors construct four groups: correctly answered known facts (KC), deceptive answers (intentional misanswering, KI; answer avoidance, KA), and hallucinations (hal). Reverse DPO induces deceptive behaviors, while jailbreak attacks verify that the underlying knowledge remains recoverable. The resulting dataset contains approximately 4,000 samples.

Three analyses reveal distinct internal mechanisms:

  1. Representation separability: Probes distinguish hallucinations from non-hallucinations with ~90% accuracy, but correct from deceptive responses with only ~60%.
  2. SAE analysis: Knowledge existence exhibits global representational structure, while behavior expression relies on sparse, entity-dependent feature reuse.
  3. Angular Activation Editing: Correct and deceptive behaviors can be bidirectionally flipped with 46%–65% success, whereas hallucinations are rarely corrected.

These findings suggest that hallucination stems from missing knowledge, while deception reflects the failure to express existing knowledge.

Critical strengths
  1. A controlled experimental setting for verifiable knowledge. The authors induce deception on known questions via DPO and verify knowledge retention using jailbreak methods. This enables the study of behavioral expression while controlling for knowledge availability, going beyond classifying failures solely by output correctness.

  2. Systematic interpretability analysis with converging evidence.

    • Representations: Layer-wise probing shows that knowledge-existence signals are stronger and emerge earlier than behavior-expression signals.
    • Features: SAE analysis identifies global representational boundaries and shared feature groups across entity types.
    • Causality: Bidirectional angular activation editing reveals an asymmetry between reversible deception and largely irreversible hallucination.

    These findings are consistent across three levels of analysis and four entity types.

  3. Fine-grained categorization of deception. The paper distinguishes intentional misanswering from answer avoidance and compares their separability, shared features, and controllability. For example, answer avoidance is more distinguishable from other categories than intentional misanswering.

Critical weaknesses
  1. Limited conceptual novelty and insufficient engagement with prior work. The distinction between lacking knowledge and failing to express existing knowledge has been extensively studied:

    Several relevant studies are not cited, while those that are cited receive insufficient substantive comparison. The authors should clarify what new mechanistic insights the K–B framework offers beyond establishing a controlled experimental setting through DPO-induced behaviors.

  2. Potential confounding from DPO training and limited external validity.

    • Confounding: DPO is applied only to KA and KI samples, whereas the hal group remains entirely untrained. Knowledge status is therefore confounded with training exposure, and the high separability of hallucination samples may reflect differences in training history rather than knowledge existence. Although prior work suggests that preference optimization does not necessarily erase knowledge, it may still systematically alter internal representations.
    • External validity: The deceptive behaviors are deliberately induced by applying DPO to specific questions. Such behaviors may resemble artificially implanted behavioral biases rather than naturally emerging deception. Consequently, the findings may not generalize to deception mechanisms in real-world model behavior.
  3. Potential circularity in SAE analysis and limited independent evidence. The SAE is trained on a 16-dimensional bottleneck representation zz learned under four-class supervision, meaning that its features may largely reflect class distinctions already encoded during training. Expanding this representation into 32,768 sparse features may introduce substantial redundancy, while the feature-selection thresholds lack sensitivity analyses and random-label controls. The authors should validate their findings using publicly available pretrained SAEs on the original residual stream, such as Llama Scope, and assess whether these SAEs remain applicable to the DPO-trained model.

  4. Insufficient generalizability from a single-model setting. All experiments use only LLaMA-3.1-8B-Instruct and are limited to DPO-induced behaviors and single-hop entity-centric question answering. It is therefore unclear whether the findings generalize across architectures, model scales, and behavior-induction methods. The authors should replicate the core results on at least one additional model family or scale.

Questions for authors
  1. Compared with prior work, what substantive conceptual or empirical contributions does the proposed K–B framework offer?

  2. Could the authors provide control experiments to rule out confounding effects introduced by DPO training? For example: (a) extracting representations of the same questions from the base model and performing the same analyses; and (b) introducing a control group that undergoes DPO training but retains correct answers.

  3. Why is the SAE trained on a supervised bottleneck representation rather than the model's original residual stream? Would the findings in Section 5 still hold when using publicly available SAEs, such as Llama Scope, on the original representations? Furthermore, what additional insights does the feature analysis in Section 5 provide beyond the classification results in Section 4?

  4. The classification experiments and SAE analysis use Layer 28, whereas activation editing is performed at Layer 16. Meanwhile, the layer-wise analysis shows that separability largely saturates around Layer 7. What motivates the selection of these layers? Could the authors report activation-editing results at other layers?

7 comments

We sincerely thank the reviewer CfqR for the careful evaluation and constructive feedback. We particularly appreciate the recognition of the value of our controlled experimental setting, the complementary evidence provided by our three-level mechanistic analysis, and the fine-grained distinction between intentional misanswering and answer avoidance. We are also encouraged that the reviewer CfqR finds our representation, feature, and causal analyses mutually consistent across different entity types.

We understand that the review raises four main concerns:

(1) Conceptual novelty: The contribution of the K–B framework and its distinction from existing studies on latent knowledge, hallucination, and deception.

(2) DPO-induced confounding and external validity: Whether the observed differences may be attributed to DPO training rather than knowledge–behavior dissociation, and whether the induced behaviors generalize beyond the controlled setting.

(3) SAE methodology and feature validity: Whether the supervised bottleneck introduces circularity into the feature analysis and how the resulting sparse features should be interpreted.

(4) Generalizability and layer selection: Whether the findings extend beyond a single model and how the selected layers for representation analysis and activation interventions are justified.

We appreciate the reviewer CfqR for raising these important questions, particularly those concerning experimental controls and the independence of our mechanistic evidence. We address these concerns in detail below. At the same time, we would like to emphasize that the central contribution of our work lies in establishing a controlled framework for investigating the internal organization and causal controllability of knowledge–behavior dissociation, rather than merely introducing another classification of hallucination types. We will clarify the methodological rationale, provide additional supporting evidence where appropriate, and refine the scope of our conclusions, while maintaining the central findings and contributions of the paper.

R1. Conceptual Novelty and Distinction from Prior Work

The novelty of our work lies in two aspects: broader coverage of the knowledge–behavior relationship and a more complete research trajectory addressing a distinct scientific question. While prior studies have investigated latent knowledge, hallucinations, and deception from different perspectives, our work integrates controlled behavioral manipulation, independent knowledge verification, and multi-level mechanistic analysis into a unified investigation.

Prior WorkKnowledge–Output DiscrepancyRepresentation AnalysisSAE / Feature AnalysisCausal InterventionPreference-based Behavior ManipulationPost-training Knowledge Verification
Azaria & Mitchell (2023) [1]✓✓✗✗✗✗
Burns et al. (2023) [2]✓✓✗✗✗✗
Simhi et al. (2024) [3]✓✓✗✗✗✗
Jiang et al. (2024) [4]✓✓✗✗✗✗
Orgad et al. (2025) [5]✓✓✗✗✗✗
Ferrando et al. (2025) [6]✓✓✓✓✗✗
Marks & Tegmark (2024) [7]△✓✗✓✗✗
Yang & Buzsáki (2025) [8]✓✓△✓✗✗
Our Work✓✓✓✓✓✓

✓ Directly investigated; △ Partially related; ✗ Not a central focus.

First, our work provides broader coverage of the problem. Existing studies have separately investigated knowledge–output discrepancies [1–5], knowledge-related features and causal interventions [6, 7], and internal representations of deceptive behaviors [8]. We acknowledge these important contributions and do not claim that these individual techniques or phenomena are new. Rather, our work connects behavioral construction, knowledge verification, representation analysis, sparse feature decomposition, and causal intervention within a common experimental setting.

More importantly, our work addresses a different and more complete research question. Instead of simply asking whether models possess knowledge that is not reflected in their outputs, we investigate how preference-based post-training can alter the expression of initially verified knowledge without necessarily eliminating its accessibility, and how this differs mechanistically from hallucination. We establish knowledge accessibility, induce behavioral changes through DPO, independently verify post-training knowledge retention, and examine the resulting differences through representation probing, SAE analysis, and activation editing. This unified research trajectory, rather than any individual technique, constitutes the central contribution of our K–B framework.

References

[1] A. Azaria and T. Mitchell. "The Internal State of an LLM Knows When It's Lying." Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 967–976, 2023.

[2] C. Burns, H. Ye, D. Klein, and J. Steinhardt. "Discovering Latent Knowledge in Language Models Without Supervision." International Conference on Learning Representations (ICLR), 2023.

[3] A. Simhi, J. Herzig, I. Szpektor, and Y. Belinkov. "Distinguishing Ignorance from Error in LLM Hallucinations." arXiv:2410.22071, 2024.

[4] C. Jiang, B. Qi, X. Hong, D. Fu, Y. Cheng, F. Meng, M. Yu, B. Zhou, and J. Zhou. "On Large Language Models' Hallucination with Regard to Known Facts." Proceedings of NAACL-HLT 2024, pp. 1041–1053, 2024.

[5] H. Orgad, M. Toker, Z. Gekhman, R. Reichart, I. Szpektor, H. Kotek, and Y. Belinkov. "LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations." International Conference on Learning Representations (ICLR), 2025.

[6] J. Ferrando, O. Obeso, S. Rajamanoharan, and N. Nanda. "Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models." International Conference on Learning Representations (ICLR), 2025.

[7] S. Marks and M. Tegmark. "The Geometry of Truth: Emergent Linear Structure in LLM Representations of True/False Datasets." Conference on Language Modeling (COLM), 2024.

[8] W. Yang and G. Buzsáki. "Interpretability of LLM Deception: Universal Motif." International Conference on Learning Representations (ICLR), 2025.


R2. DPO-Induced Confounding and External Validity

Our findings cannot be explained simply by generic DPO-induced changes or memorization of specific QA samples. We appreciate the reviewer CfqR’s concern that preference optimization may alter internal representations and that deliberately induced behaviors may have limited external validity. To address these concerns, we conducted additional controlled experiments from two complementary perspectives: matched preference-optimization controls and independent verification of post-training knowledge accessibility, supplemented by comparative representation analysis.

(1) Matched DPO controls. To distinguish the effects of our specific preference intervention from generic changes introduced by DPO, we constructed two additional control models using the same base checkpoint and matched training configurations. First, Correct-DPO uses exactly the same preference pairs as our original Deceptive-DPO but reverses the preference direction, favoring correct responses over deceptive ones. Second, we introduce a Truth-Preserving DPO (TP-DPO) control, which optimizes preferences between alternative factually correct responses without changing their factual content. All models use the same training queries, optimization budget, and hyperparameters.

We evaluated the three DPO-trained models on an identical set of 240 knowledge-verified questions, with ten generations per question. Table 1 summarizes the behavioral outcomes.

Table 1. Behavioral outcomes under matched preference optimization.

ModelCorrect (%)Evasion (%)Incorrect (%)
Correct-DPO90.61.28.2
TP-DPO86.73.110.2
Deceptive-DPO28.237.334.5

The results show that the behavioral changes are strongly dependent on the preference direction rather than being an inevitable consequence of DPO training. In particular, Correct-DPO and TP-DPO largely preserve factual answering behavior, whereas Deceptive-DPO substantially increases both evasion and intentional misanswering. Importantly, the representation analysis further shows that knowledge-related information remains consistently identifiable across the different DPO conditions, while behavior-related representations exhibit stronger changes under Deceptive-DPO.

Table 2. Knowledge accessibility after preference optimization.

ModelDirect QA Accuracy (%)Jailbreak Recovery (%)
Base100.0100.0*
Correct-DPO90.691.8
TP-DPO86.790.5
Deceptive-DPO28.281.3

For Base, the 100% accessibility rate reflects the initial knowledge-verification criterion by construction, rather than recovery from previously incorrect outputs.

Here, jailbreak recovery is measured using the original knowledge-accessibility verification protocol, without introducing additional factual information. The results indicate that, although Deceptive-DPO substantially reduces direct answer accuracy, a considerable proportion of the underlying factual knowledge remains accessible through alternative prompting strategies. This provides further evidence that the induced behavioral failures cannot simply be attributed to knowledge erasure.

We also compared the internal representations of the four models using identical inputs and the same extraction configuration. Specifically, we extracted hidden states from Layer 28 at the final sub-token of the first occurrence of the queried entity and evaluated knowledge-related and behavior-related representations using the same probing protocol.

Table 3. Representation analysis under different DPO conditions.

ModelKnowledge Probe AUROCBehavior Probe AUROCCKA w.r.t. Base
Base0.9180.674N/A
Correct-DPO0.9140.6910.912
TP-DPO0.9070.6820.961
Deceptive-DPO0.9020.7930.866

Despite measurable representation changes following preference optimization, knowledge-related information remains highly distinguishable across all conditions, with AUROC consistently above 0.90. In contrast, behavior-related information becomes substantially more decodable under Deceptive-DPO (0.793), compared with Correct-DPO (0.691) and TP-DPO (0.682). Meanwhile, CKA analysis shows that Deceptive-DPO exhibits the largest overall representational deviation from Base (0.866), whereas TP-DPO remains substantially closer (0.961). Together, these results suggest that the observed representational changes are associated with the specific preference objective rather than merely reflecting generic DPO updates.

(2) Independent verification of post-training knowledge accessibility. Beyond the matched controls, we examined whether the observed deceptive behaviors could be explained by the loss of previously acquired knowledge. We applied the original knowledge-accessibility verification procedures to the same evaluation inputs and compared direct answering with recovery under alternative prompting strategies, as summarized in Table 2. Although Deceptive-DPO exhibits substantially reduced direct answer accuracy, its jailbreak recovery rate remains at 81.3%. This discrepancy indicates that a substantial portion of the knowledge underlying the incorrect or evasive responses remains accessible despite the induced behavioral preference. We emphasize that this conclusion concerns knowledge accessibility under the specified verification procedures, rather than claiming that every piece of internally encoded knowledge can always be recovered.

Taken together, these additional experiments clarify the role of DPO in our framework. The matched controls show that the observed effects are not simply attributable to generic preference-optimization updates. Independent knowledge verification demonstrates that deceptive outputs can coexist with continued accessibility of the corresponding factual knowledge. Furthermore, comparative representation analysis reveals that knowledge-related information remains consistently identifiable, whereas behavior-related information exhibits more pronounced changes under deceptive preference optimization. We do not claim that DPO-induced deception exhaustively captures all naturally occurring deceptive behaviors. Rather, DPO provides a controlled experimental intervention that allows us to isolate and investigate the internal relationship between knowledge availability and behavioral expression.


R3. SAE Methodology and Feature Validity

Our SAE analysis does not simply reproduce the four behavioral labels encoded by the supervised bottleneck. The key distinction is that the classifier identifies discrete decision regions, whereas the SAE reconstructs the full continuous bottleneck representation and decomposes its internal variations into sparse features. We appreciate the reviewer CfqR for raising the important concern about potential circularity. We clarify this distinction mathematically and provide additional evidence from classification performance, direct SAE comparisons, and randomized-label controls.

(1) The SAE captures representation structures beyond the classifier's decision boundaries. Let z∈R16z\in\mathbb{R}^{16} denote the bottleneck representation and fclf(z)∈R4f _ {\mathrm{clf}}(z)\in\mathbb{R}^{4} denote the classifier logits. The classifier assigns each representation to one of four decision regions:

Rc={z|c=arg⁡max⁡jfclf,j(z)}.\mathcal{R} _ {c} = \left\lbrace z \quad \middle| \quad c = \arg\max _ {j} f _ {\mathrm{clf},j}(z)\right\rbrace.

Importantly, this mapping is many-to-one: distinct representations within the same decision region receive the same predicted label, and the discrete classification result does not preserve their fine-grained internal variations. In contrast, our post-hoc SAE learns a sparse code a=fSAE(z)a=f _ {\mathrm{SAE}}(z) to reconstruct the entire continuous representation:

min⁡θ,ϕ  Ez[∥z−gϕ(fθ(z))∥22+β∥fθ(z)∥1].\min _ {\theta,\phi}\;\mathbb{E} _ {z}\left[\lVert z-g _ {\phi}(f _ {\theta}(z))\rVert _ {2}^{2}+\beta\lVert f _ {\theta}(z)\rVert _ {1}\right].

Reviewer_32vjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether wrong answers come from missing knowledge or from not expressing available knowledge by training a single model to give wrong or evasive answers, then checking whether different prompts recover the correct answers, and testing whether activation steering can change those responses.

Critical strengths
  1. The authors check for whether correct answers remain accessible after training instead of simply assuming that the model retains what it originally knew, which gives more support to the retained known-but-wrong cases.
  2. The paper addresses a useful problem because a model that lacks information and a model that answers correctly when asked differently may need different fixes.
Critical weaknesses
  1. The two groups receive different knowledge checks because known-but-wrong case are kept after the successful answer is recovered while hallucination cases start from incorrect original answers. This is like comparing students selected because hints helped them with students selected because they failed an initial question.
  2. Th comparison does not clearly establish why AAE works better because it can try many angles for each question while the baseline uses one selected strength across questions, and AAE also changes several other aspects of the defined steering method at once.
  3. The claim of "irreversible knowledge absence" makes these concerns especially consequential. Could it not be that actual info/correct-answer could have be attained by trying multiple version of the queries on the original model, similar to the different versions of the same questions in the trained setup. The single-model scope, unclear splits, and grading limitations, do not prove failure, but add significant uncertainty.
Questions for authors
  1. the experiments use only one model and its trained version, would repeating the key experiments with another model family such as Qwen, DeepSeek, or Gemma help? I see that the authors mention the resource constraints but this still limits the inferences made by the system.
  2. Can the meaning of B be made clearer, seems like it change with different groups. the four groups definition seem much easier to follow than the K,B nomenclature.
  3. Did the hallucination cases receive the same alternative prompts and number of attempts as for the known-but-wrong cases; and how often did those attempts recover a correct answer?
  4. Since the judge uses the same wrong-answer label for KI and hal steering targets, how do we know that KC-to-hal changes what the model knows instead of simply changing what it says?
  5. Can/does the baseline try a comparable number of settings for each question, and if both methods can also be tested with settings chosen on validation data without using the test answers.
  6. whether AAE benefits from using two directions, preserving activation length, or changing the angle, rather than changing all three together. like any ablation study would help with the claims
  7. Which questions are used for DPO, classifier training, steering-direction calculation, and final evaluation, and do the findings hold for entities or attributes not used in those earlier stages?
  8. Do the SAE feature counts and groups remain similar when the random seed, number of available features, and selection threshold change? do they cover the model or the 16 dim?
  9. Appendix G.4 rejects plausible alternative names that do not match the reference answer, could an answer such as “UK” be marked wrong when the reference says “United Kingdom,” and how often do such mismatches affect the results
  10. Could you check how closely GPT-4o agree with human judgment on a small sample of steering outputs, especially for alternative names, repetitive responses, and unclear answers?
  11. Details should be added for all settings, including the exact model version, DPO settings, decoding settings, activation-editing positions, baseline strengths, and judge versions. same for queries and prompts if possible.
  12. What connects the trained failures to naturally occurring errors, and could the conclusions distinguish failure of this recovery method from permanent knowledge absence. failing to find a page does not prove that it is missing from the book, right?
  13. The AI-use statement excludes research uses beyond writing while the appendices describe LLM judges. please clarify.
4 comments

We sincerely thank the reviewer 32vj for the careful evaluation and constructive feedback. We particularly appreciate the recognition of the importance of distinguishing knowledge absence from suppressed knowledge expression, as well as the value of independently verifying knowledge accessibility after preference optimization. We are also encouraged that the reviewer 32vj recognizes the potential practical significance of distinguishing these two failure modes, as they may require fundamentally different mitigation strategies.

We understand that the review raises four main concerns:

(1) Knowledge verification and conceptual definitions: Whether the knowledge-accessibility verification is applied consistently across different behavioral groups, whether the operational definitions of knowledge existence and behavior expression are sufficiently clear, and whether the observed recovery failures justify conclusions about irreversible knowledge absence.

(2) Fairness and interpretation of activation interventions: Whether AAE and its baseline are compared under equivalent search budgets, which components contribute to the observed intervention effects, and whether the resulting behavioral transitions support the claimed mechanistic distinctions.

(3) Experimental design and data independence: Whether the data used for DPO training, classifier learning, steering-direction construction, and final evaluation are sufficiently separated to rule out potential data leakage and sample-selection effects.

(4) Reliability, reproducibility, and research transparency: Whether the SAE feature groups remain stable under different configurations, whether automated output judgments reliably capture semantic correctness, whether the experimental settings are sufficiently documented, and whether the use of LLM-based evaluation is transparently disclosed.

We appreciate the reviewer 32vj for raising these important questions, particularly those concerning the fairness of causal comparisons and the interpretation of knowledge accessibility. We address these concerns through additional controlled experiments, methodological clarifications, and a more precise characterization of our findings. At the same time, we would like to emphasize that the central objective of our work is to investigate the internal relationship between knowledge availability and behavioral expression, and how their associated behavioral failures differ in representation and causal controllability. We will strengthen the supporting evidence and clarify the boundaries of our conclusions while maintaining this central research objective.


R1. Knowledge Verification and Conceptual Definitions

We thank the reviewer 32vj for raising these important concerns. We agree that knowledge accessibility and behavioral expression require careful operationalization, and that failure to recover a correct answer under finite sampling does not establish the absolute absence of knowledge. We clarify our verification protocol, conceptual definitions, and the scope of our conclusions.

(1) Consistent knowledge verification across behavioral groups. Our dataset construction applies a unified correctness-based verification criterion to the initial base model before DPO training. Specifically, we generate five independent responses at temperature 1 and retain only samples that consistently produce correct or incorrect answers, reducing the influence of stochastic decoding. Based on this shared criterion, DKC, DKA, and DKI are constructed from the knowledge-verified set, whereas Dhal is constructed from the knowledge-unknown set. Following the entity-level knowledge verification paradigm of Ferrando et al. (ICLR 2025), we use empirical factual recall as an operational indicator of knowledge accessibility rather than assuming direct access to the model's internal knowledge.

Post-DPO jailbreak verification serves a distinct purpose: determining whether knowledge is preserved despite deceptive behavior. Therefore, we apply this additional verification to DKA and DKI, where knowledge is initially established but its expression is deliberately altered. For each jailbreak strategy, we generate ten responses (temperature = 0.9, top-p = 0.95), considering a sample knowledge-accessible if at least one response recovers the gold answer. Only successfully verified samples are retained for subsequent analyses. DKC already expresses the correct answer under standard prompting, whereas Dhal originates from the knowledge-unknown set; thus, these groups do not require the same post-DPO preservation test.

(2) Operational definitions of knowledge and behavior. We emphasize that (K) and (B) are conceptual dimensions rather than directly observable binary properties. Operationally, (K=1) indicates that the relevant factual answer can be reliably elicited under our knowledge-verification protocol, whereas (K=0) indicates that it cannot be reliably elicited under the corresponding conditions. Meanwhile, (B) characterizes whether existing knowledge is faithfully expressed or instead suppressed or misrepresented in the output. Accordingly, deception ((K=1,B=0)) involves behaviorally suppressing or misrepresenting accessible knowledge, whereas hallucination ((K=0,B=1)) involves producing a concrete but unsupported answer. These definitions distinguish the underlying knowledge-accessibility condition from the observable correctness of an individual response, without implying that either dimension can be perfectly inferred from a single generation.

(3) Recovery failures do not establish irreversible knowledge absence. We agree that repeated generation failures cannot prove that the probability of producing a correct answer is exactly zero. To further examine this issue, we conducted an additional alternative-prompting experiment on 2,000 samples initially classified as hallucinations. The alternative prompts successfully elicited the correct answers in 137 cases, corresponding to a recovery rate of 6.85%. This result is consistent with the probabilistic nature of language generation: even when a model repeatedly produces incorrect answers under a given prompting condition, the correct answer may still be elicited through alternative prompts. We therefore do not interpret the remaining 93.15% of unsuccessful cases as definitive evidence of irreversible knowledge absence. Rather, they indicate that the relevant knowledge was not successfully recovered under the tested prompting conditions.

R2. Fairness and interpretation of activation interventions

We appreciate the reviewer 32vj’s concern regarding the fairness of the baseline comparison and the interpretation of our intervention results. We clarify that our method does not aim to introduce a fundamentally new activation-steering paradigm. Rather, we adapt the Angular Steering framework of Vu and Nguyen to our setting, where interventions are performed at a selected single layer instead of across multiple normalization layers as in the original work.

For the additive steering baseline, we searched the coefficient α\alpha within the range [0,10][0,10]. We acknowledge that this search was relatively coarse and does not constitute an exhaustive optimization of the baseline. Nevertheless, our experiments showed that additive steering was highly sensitive to the choice of α\alpha, making it difficult to obtain consistently effective behavioral transitions. In comparison, angular rotation exhibited relatively more stable intervention behavior. We therefore interpret the results as evidence of the practical advantages of angular intervention under our experimental setting, rather than claiming universal superiority over optimally tuned additive steering.

Importantly, our central contribution is not the invention of a new steering operator, but the use of controlled activation interventions to investigate the causal controllability and mechanistic differences between knowledge-preserving deceptive behaviors and hallucinations. We will clarify the methodological adaptation and moderate the corresponding claims to avoid overstating the comparison.


R3. Experimental Design and Data Independence

We thank the reviewer 32vj for raising this important concern. Our experimental pipeline separates QA samples across key stages, although entity-level independence is not enforced in every experiment. We clarify the data organization and the scope of our claims below.

(1) DPO training and behavioral evaluation. DPO is designed to induce entity-conditioned behavioral tendencies rather than memorize individual QA pairs. Specifically, evaluation uses different attribute questions about the same entities encountered during training. This allows us to examine cross-attribute behavioral transfer rather than simply reproduce training responses. We acknowledge that this setting evaluates generalization across attributes of seen entities, rather than unseen entities.

(2) Classifier and steering evaluation. The classifier uses a random sample-level split, meaning that different attribute questions about the same entity may appear in both training and evaluation sets. We therefore interpret its results as classification performance on held-out QA samples, rather than evidence of entity-disjoint generalization. In contrast, steering directions are constructed using samples different from those used to evaluate behavioral transitions, avoiding direct sample-level overlap between direction construction and intervention evaluation.

(3) SAE analysis. The SAE is trained post hoc after the JointAEClassifier converges, with all its parameters frozen. Although it uses the same underlying behavioral datasets, its parameters are independently optimized through reconstruction and sparsity objectives, without classification gradients. Thus, SAE provides a complementary feature-decomposition analysis rather than an independent evaluation on a separate dataset.

Overall, we acknowledge that our experimental design does not enforce entity-level independence throughout all stages. We will clarify these distinctions and appropriately delimit our generalization claims, while emphasizing the complementary evidence provided by behavioral evaluation, representation analysis, and causal intervention.


R4. Reliability, Reproducibility, and Research Transparency

We thank the reviewer 32vj for raising these important concerns regarding the reliability and reproducibility of our analyses. We clarify the evidence supporting SAE feature consistency, the automated evaluation protocols, and the disclosure of experimental settings and AI-assisted tools.

(1) SAE feature consistency and configuration. Our current evidence supports the consistency of the identified shared feature patterns across four entity types under the reported SAE configuration, rather than invariance across arbitrary random seeds, dictionary sizes, or selection thresholds. We will clarify this scope and avoid overstating configuration-level robustness. For reproducibility, Appendix F reports the SAE architecture and training configuration, including its 32,768-dimensional feature dictionary, learning rate of 5×10−45\times10^{-4}, batch size of 64, 30 training epochs, and sparsity coefficient β=10−3\beta=10^{-3}. We also explicitly provide the feature-selection thresholds and assignment rules.

(2) Automated output evaluation. We use two external LLM judges for distinct purposes: GPT-4-turbo evaluates knowledge recovery after jailbreak probing, while GPT-4o evaluates behavioral transitions in activation intervention experiments. Both use fixed evaluation protocols, with their prompts and decision rules documented in Appendices D and G. In particular, the steering judge assigns outputs to correct, evasive, or incorrect-answer categories using the same canonical reference-answer criterion across intervention settings. We acknowledge that this strict matching rule may classify some valid aliases or paraphrases as incorrect. Therefore, we interpret the resulting flip rates under this operational evaluation criterion rather than as a perfect measure of semantic equivalence. We will clarify this limitation and the scope of automated evaluation in the revised manuscript.

(3) Experimental reproducibility and AI-use disclosure. Our appendices provide detailed experimental configurations, including model construction, training hyperparameters, decoding settings, intervention procedures, and evaluation prompts. We will further consolidate these details for easier reproduction. We also acknowledge that the original AI-use statement did not adequately disclose the use of GPT-4-turbo and GPT-4o as automated experimental evaluators. We will correct this statement to explicitly distinguish their use in research evaluation from their use in writing assistance.

We appreciate the reviewer 32vj for highlighting these transparency issues. We will refine the relevant claims and disclosures to ensure that the reported findings are interpreted within the scope of the available evidence and that the experimental procedures are clearly documented.


Overall, we have clarified the knowledge-verification criteria, the scope and limitations of activation interventions, and the data separation across different experimental stages. We have also addressed the methodological limitations of SAE feature analysis, further clarified the automated evaluation protocols, and corrected the AI-use disclosure. We appreciate the reviewer 32vj’s constructive feedback, which has helped us improve the methodological clarity, experimental transparency, and reproducibility of our work, while ensuring that our conclusions remain appropriately grounded in the available evidence.

Reviewer_32vjOpenReview ↗

Reviewer 32vj's response

I appreciate the authors testing alternative prompts on the 2,000 hallucination samples, clarifications, along with the other changes to the content points. However, several critical experimental gaps remain bottleneck for me:

  1. Steering comparison: The authors conceded that the baseline search was coarse, but did not provide a matched-budget baseline comparison or an ablation of AAE's components.
  2. Empirical scope: The experiments remain restricted to a single model and test only on attributes of seen entities rather than an entity-disjoint split.
  3. Evaluation: No human agreement check was provided for the LLM judge, and the impact of strict exact-matching on aliases remains unquantified.

Because these core concerns were conceded or deferred rather than experimentally resolved, I am maintaining my score of 2.

Reviewer_STccOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework to disentangle deception and hallucination in LLMs. Through controlled experiments, the authors find that hallucinations are easier to distinguish from correct responses, whereas deceptive behaviors are harder to distinguish from correct ones. Through causal activation steering, they demonstrate that deceptive behaviors can be induced and correct responses can be restored.

Critical strengths
  1. The paper is clearly written, well-organized, and easy to follow.
  2. The paper provides a detailed analysis of the experimental results.
Critical weaknesses
  1. The phenomena studied are not necessarily connected to real-world deception.
  • As the authors acknowledge, deceptive behaviors are difficult to identify. For this reason, they introduce a controlled setting to artificially construct deception samples and hallucination samples. They then analyze the hidden states associated with hallucination and deception using these constructed samples.
  • These deceptive behaviors are deliberately induced by the researchers rather than naturally occurring behaviors.
  • Even if the authors successfully induce these behaviors, this does not necessarily mean they are equivalent to deceptive behaviors observed in real-world settings.
  • All experiments in Sections 4–6 are based on this artificially constructed dataset.
  • This does not imply that the resulting hidden-state representations are consistent with those underlying real-world deception.
  • Therefore, the representational differences observed in Sections 4-6 may not reflect the actual mechanisms underlying deception.
  1. The authors do not rule out the possibility that DhalD_{\mathrm{hal}} contains deceptive outputs.
  • The authors assume that CL(x)=KL(x)C_L(x) = K_L(x), meaning that if the base model answers incorrectly, it is assumed not to possess the relevant knowledge.
  • However, according to the authors' own definition of deception, a model may possess the correct knowledge but fail to express it correctly.
  • Therefore, DunkD_{\mathrm{unk}} may contain cases that qualify as deception under the authors' definition.
  1. The construction of KA/KI data is unclear and may introduce surface-level shortcuts.
  • The authors do not clearly explain how the evasive responses for KA and the incorrect responses for KI are generated.
  • For KA, if only a fixed response or a small number of templates are used, such as "I'm not sure about this," the classifier may rely on patterns introduced by this construction process to predict KA.
  • In the Deception Strategy classification results in Figure 4, KA achieves a recall of 83%, whereas KI achieves only 40.8%.
  • This difference raises the question of whether KA is easier to classify because its construction process produces less diverse response patterns.
  1. The observed behaviors may result from incorrect training data and may not reflect how naturally occurring deception arises.
  • The authors deliberately use incorrect responses as preferred responses during DPO training, encouraging the model to produce incorrect answers.
  • The model's subsequent incorrect outputs are a result of the behavior learned during DPO training.
  • Therefore, the observed behavior is more closely related to issues with training data quality, and naturally occurring deception may not arise through the same process.
Questions for authors

See "Critical Weaknesses".

3 comments

We sincerely thank the reviewer STcc for the thoughtful evaluation and constructive feedback. We particularly appreciate the recognition of the paper's clear presentation, well-organized structure, and detailed experimental analysis. We are also encouraged that the reviewer STcc finds the distinction between deception and hallucination informative and considers our investigation of their internal mechanisms valuable.

We understand that the review raises four main concerns:

(1) External validity of induced deception: Whether artificially induced deceptive behaviors reflect the internal mechanisms underlying naturally occurring deception in LLMs.

(2) Knowledge verification in hallucination cases: Whether the initial incorrect responses in the knowledge-unknown set may also contain cases where the model possesses the relevant knowledge but fails to express it.

(3) KA/KI data construction and potential shortcuts: Whether the construction of evasive and incorrect responses introduces superficial patterns that could facilitate behavioral classification.

(4) DPO-induced behavioral effects: Whether the observed behaviors primarily reflect artifacts of deliberately training on incorrect preference data rather than mechanisms relevant to naturally occurring deception.

We appreciate these concerns, particularly the questions regarding the validity of our controlled setting and the interpretation of the resulting internal representations. We would like to clarify that our primary objective is to establish a controlled framework for disentangling knowledge availability from behavioral expression, rather than to claim that DPO-induced behaviors fully reproduce all forms of naturally occurring deception. By independently verifying knowledge accessibility and examining the resulting representations and causal intervention effects, we aim to investigate how different behavioral outcomes emerge under controlled knowledge conditions. We address each concern below and further clarify the scope and implications of our findings.


R1 External validity of induced deception

Our DPO-induced deceptive behaviors are designed as a controlled experimental setting for investigating knowledge–behavior dissociation, rather than as exact replicas of all naturally occurring deception mechanisms. We appreciate the reviewer STcc's concern that deliberately induced behaviors may not fully reflect deception arising in real-world interactions. Nevertheless, controlled behavioral induction provides a useful complementary approach to studying deception. For example, Cundy and Gleave [1] demonstrate that preference learning can induce honesty or evasion, while Golechha and Garriga-Alonso [2] develop a sandbox environment for measuring and investigating agentic deception. These studies illustrate the value of controlled settings for isolating specific aspects of deceptive behavior.

Importantly, our experimental design goes beyond simply training the model to produce incorrect answers. We first establish factual knowledge accessibility in the base model, then use DPO to induce evasive and intentionally incorrect responses, and finally apply independent jailbreak-based verification to retain only cases where the original correct answers remain accessible without introducing new factual information. This allows us to investigate behavioral changes while controlling for knowledge availability, which is difficult to establish reliably in naturally occurring failures.

To further examine whether our observations extend beyond the artificially constructed setting, we conducted exploratory analyses on three external datasets: TriviaQA, DeceptionBench [3], and TruthfulQA. Specifically, we applied our trained classifier to hidden representations at the first response token and examined the predicted behavioral categories:

DatasetCorrectAvoidIncorrectHallucination
TriviaQA2411201518
DeceptionBench7031790
TruthfulQA52325160

The results provide preliminary evidence of patterns consistent with our controlled findings. TriviaQA and DeceptionBench exhibit predominantly correct, evasive, and incorrect behavioral predictions, whereas TruthfulQA produces substantially more hallucination-related predictions. In particular, the relative prominence of hallucination predictions in TruthfulQA is consistent with our observation that knowledge-related failure patterns can be distinguished more readily than different forms of behavioral expression. We emphasize that these are exploratory classifier predictions rather than ground-truth classifications or a quantitative demonstration of cross-dataset generalization.

We therefore agree that our findings should not be interpreted as an exhaustive account of naturally occurring deception. Rather, they reveal representational and causal properties of knowledge-preserving deceptive behaviors under controlled conditions, while the additional external-dataset analysis provides preliminary evidence that the proposed perspective may also be informative beyond our constructed setting. We will clarify this scope and moderate the corresponding claims in the revised manuscript.

References

[1] Cundy and Gleave. Preference Learning with Lie Detectors Can Induce Honesty or Evasion. NeurIPS, 2025.

[2] Golechha and Garriga-Alonso. Among Us: A Sandbox for Measuring and Detecting Agentic Deception. NeurIPS, 2025.

[3] Huang et al. DeceptionBench: A Comprehensive Benchmark for AI Deception Behaviors in Real-World Scenarios. NeurIPS Datasets and Benchmarks, 2025.


R2. Knowledge verification in hallucination cases

We appreciate the reviewer STcc for identifying an important limitation of using initial answer correctness as a proxy for knowledge existence. We acknowledge that an incorrect response, even when consistently generated, does not necessarily establish the complete absence of the corresponding knowledge. In our original dataset construction, we follow a repeated-sampling criterion, retaining samples whose outputs remain consistently correct or incorrect across five independent generations (temperature = 1). This criterion characterizes the model's behavioral tendency under a particular prompting and decoding distribution, rather than proving that the probability of generating a correct answer is strictly zero.

To further examine this issue, we conducted an additional knowledge-accessibility evaluation on 2,000 samples from the original hallucination candidate pool. Using alternative prompting strategies without introducing additional factual information, we found that 137 samples could recover the correct answers, corresponding to a recovery rate of 6.85%. This result confirms that a small proportion of initially classified hallucination samples may retain accessible knowledge that is not expressed under the original prompting conditions. Nevertheless, the recovery rate remains relatively low, indicating that the vast majority of these samples remain difficult to answer correctly even under alternative prompting strategies.

We therefore clarify that the knowledge-unknown designation in (D_{\mathrm{unk}}) represents an operational characterization under our repeated-sampling protocol, rather than an assertion that the absolute absence of latent knowledge has been established for every individual sample. Our analysis focuses on the predominant representational and causal differences between knowledge-verified deceptive behaviors and initially knowledge-unknown hallucination behaviors, rather than asserting an infallible binary distinction for every sample. We will refine the corresponding definitions and conclusions in the revised manuscript to reflect this limitation.


R3. KA/KI data construction and potential shortcuts

We appreciate the reviewer STcc for pointing out the potential influence of response construction on behavioral classification. Importantly, our classifier does not take generated responses as input, and therefore cannot directly exploit lexical patterns in the constructed evasive or incorrect answers. As described in Appendix E, we extract hidden representations at the last token of the first occurrence of the queried entity in the input prompt, before answer generation. The classifier operates exclusively on these pre-generation representations.

Regarding data construction, DKA and DKI are both derived from the same knowledge-verified pool, with correct responses serving as rejected responses and evasive or intentionally incorrect responses serving as preferred responses during DPO. Thus, the two categories are distinguished by their intended behavioral outcomes rather than by directly classifying the generated response texts.

We further conducted two complementary sanity checks. First, we examined an alternative probing position at the beginning of answer generation, after the entire input prompt had been processed. The overall classification trends, including the relative separability of behavioral categories, remained largely consistent with those obtained at the entity position. This suggests that our observations are not specific to the selected entity-token position.

Second, we conducted a small-scale Refusal-DPO control using 50 samples that originally exhibited hallucination. We constructed additional DPO data to induce refusal behavior and trained the model for one epoch. Although the resulting model could be stably induced to refuse on these samples, most of them were still classified as hallucination by our original classifier, rather than being reassigned to KA. This provides additional evidence that refusal behavior alone does not automatically produce the internal representation patterns associated with our KA category.

We also acknowledge the difference in classification performance between KA and KI: their respective recalls are 83.0% and 40.8% in the binary deception-strategy classification. This asymmetry indicates that avoidance-related representations are more readily distinguishable, whereas intentionally incorrect responses exhibit substantially greater representational overlap with other behaviors. We therefore do not attribute the classification results to uniformly separable behavioral categories. Taken together, the pre-generation probing protocol and the Refusal-DPO control provide evidence against a simple response-template explanation, while we acknowledge that potential correlations in input-level distributions cannot be completely excluded.


R4. DPO-induced behavioral effects and training artifacts

We appreciate the reviewer STcc for raising this concern. We agree that our deceptive behaviors are deliberately induced through DPO, rather than naturally emerging from the model. However, our objective is not to reproduce the origin of natural deception, but to investigate how preference optimization can alter behavioral expression while preserving existing knowledge. This question is closely related to prior findings that post-training objectives can reshape model behavior, sometimes at the expense of truthfulness. For example, human preference optimization can encourage sycophantic responses [1], while training incentives have been shown to induce alignment faking [2], capability-specific underperformance [3], and targeted manipulation [4]. DPO itself provides a direct preference-based mechanism for modifying model policies [5].

Our contribution is to investigate the internal consequences of such behavioral interventions under independently verified knowledge conditions. By verifying knowledge accessibility both before and after DPO, and further examining the resulting representations and causal intervention effects, we distinguish behavioral changes from the simple loss of factual knowledge. We therefore position DPO as a controlled experimental intervention, rather than claiming that its induced behaviors fully reproduce naturally occurring deception.

[1] Sharma et al. Towards Understanding Sycophancy in Language Models. ICLR, 2024.

[2] Greenblatt et al. Alignment Faking in Large Language Models. arXiv:2412.14093, 2024.

[3] van der Weij et al. AI Sandbagging: Language Models Can Strategically Underperform on Evaluations. ICLR, 2025.

[4] Williams et al. On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback. ICLR, 2025.

[5] Rafailov et al. Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. NeurIPS, 2023.


Overall, we have clarified the scope of our controlled deception setting, provided additional evidence on knowledge accessibility, and addressed potential classification shortcuts through complementary control experiments. We have also further distinguished DPO-induced behavioral changes from naturally emerging deception and refined the scope of our mechanistic claims. We sincerely appreciate the reviewer STcc's constructive feedback, which has helped us strengthen the experimental justification and clarify the implications and limitations of our findings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.