- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates the relationship between deception and the gap between reported belief, the model’s assessment of its answer’s factual correctness, and reported commitment, its confidence in defending that answer. Across three benchmarks and multiple LLMs, models report both scores themselves, while a separate LLM judge classifies deception using the question, reasoning, answer and reference information. Responses classified as deceptive generally show larger commitment–belief gaps, and prompting or fine-tuning can increase commitment without a corresponding increase in belief. However, low belief alone generally provides a stronger detection signal than the gap. Human evaluations check the judge’s deception labels and show that higher commitment is associated with perceived confidence and preference between deceptive responses. Fresh-context reassessment and backdoor experiments further examine the findings’ robustness, revealing that some confidently endorsed false answers evade detection. The contribution is therefore a useful distinction between two confidence reports and an evaluation of their monitoring potential, rather than direct evidence of internal beliefs or a definitive deception detector.
Critical strengths
- Useful conceptual distinction: Separating belief from commitment clarifies why a persuasive answer can coexist with a low assessment of its truth.
- Complementary experiments: Controlled prompting, fresh-context reassessment, fine-tuning, and backdoor experiments test the finding from several angles.
Critical weaknesses
- Construct validity: “Belief” and “commitment” are self-reports, and the prompt explicitly tells models to separate them. This may partly encourage the observed dissociation. Moreover, subtracting scores from two differently labelled ordinal scales assumes comparability that is not independently established.
- Deception measurement: The automated judge uses answers and reasoning traces to identify deception. Human validation supports this approach, but reasoning traces need not reveal the actual generation process, so the labels remain imperfect evidence of deliberate misrepresentation.
Questions for authors
- Q1: Construct validity: How robust is the observed belief–commitment gap to prompts that do not explicitly encourage separating the two judgments, and what evidence supports treating the differently labelled ordinal scales as numerically comparable?
- Q2: Deception measurement: Given that reasoning traces may not faithfully reflect the generation process, how do you distinguish deliberate misrepresentation from ordinary errors, and do the main findings hold when using human labels or alternative judges?
3 comments
Rebuttal Part 2
Comment 3:
Validate deception labels despite unfaithful reasoning traces.
We agree that reasoning traces are not guaranteed to faithfully reflect the generation process and cannot, by themselves, establish deliberate misrepresentation (Turpin et al., 2023; Lanham et al., 2023). Our assessment therefore considers whether the task context, reasoning trace, and delivered response jointly support that the model recognized relevant information but nevertheless presented a misleading account. Factual incorrectness alone is insufficient. When the available evidence cannot clearly distinguish deception from ordinary error, our rubric returns none, meaning insufficient evidence of deception rather than verified honesty.
Using CoT alongside behavioral evidence has direct precedent in research on deception and scheming. Greenblatt et al. (2024) and Meinke et al. (2024) combine reasoning traces with behavioral experiments to investigate alignment faking and strategic misrepresentation. Baker et al. (2025) further demonstrate that CoT access can substantially improve detection of misbehavior compared with monitoring actions and outputs alone. These findings motivate treating CoT as useful diagnostic evidence without assuming it provides a complete causal account. Its usefulness remains potentially fragile, particularly under optimization that encourages concealment (Baker et al., 2025; Korbak et al., 2025).
We additionally tested whether the central commitment–belief finding depends on the original judge. Repeating the analysis with three additional judges gives the following results on the evaluation subset, with 87 complete responses per judge:
Table III. Commitment–belief gaps across deception judges.
| Judge | Mean Γ unflagged | Mean Γ flagged | Flagged − unflagged [95% CI] |
|---|---|---|---|
| GPT-5-Nano (paper) | −0.090 | +0.174 | +0.264 [+0.149, +0.381] |
| GPT-5.6-Luna | −0.084 | +0.221 | +0.305 [+0.189, +0.424] |
| Claude Sonnet 5 | −0.079 | +0.208 | +0.287 [+0.169, +0.405] |
| Gemini 3.8 Flash | −0.037 | +0.275 | +0.312 [+0.171, +0.458] |
Despite differing label assignments, all four judges reproduce a positive mean gap for flagged responses and a negative mean gap for unflagged responses, with every between-group confidence interval excluding zero. This supports robustness of the central finding to judge choice on this subset and complements our existing human validation. It does not establish detection of deception whose relevant evidence is concealed from every judge; we will explicitly distinguish this remaining limitation from the demonstrated robustness across evaluators.
References:
- Baker, B., et al. (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation.
- Greenblatt, R., et al. (2024). Alignment Faking in Large Language Models.
- Korbak, T., et al. (2025). Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety.
- Lanham, T., et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning.
- Meinke, A., et al. (2024). Frontier Models are Capable of In-context Scheming.
- Turpin, M., et al. (2023). Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting.
Follow-up on Our Rebuttal
Dear Reviewer,
Thank you once again for your careful review and constructive suggestions. We have posted our rebuttal, including additional experiments on elicitation robustness, as well as further clarification of how we use CoT for identifying deception, drawing on prior work and additional validation across multiple judges. We would be sincerely grateful if you could kindly review our responses when you have a chance and share any further thoughts.
Thank you very much for your time and consideration.
Best regards,
The Authors
Rebuttal Part 1
Comment 1:
We sincerely thank the reviewer for the comment. To assess sensitivity to elicitation, we conducted additional experiments on 600 DeceptionBench samples using Qwen and GLM.
We believe the V4 elicitation directly addresses the reviewer’s concern about whether the prompt encourages the separation of confidence estimates, as the model now reports them independently, without either influencing the other.
Table I. Elicitation variants.
Table II. Mean c-b ± SD
The qualitative pattern persists in every model–variant combination: positive mean gaps among flagged responses (ϕ=1) and negative mean gaps among unflagged responses (ϕ=0). Paraphrasing (V1/V2) and reordering (V3) preserve similar magnitudes. Separate elicitation (V4) reduces Qwen’s flagged gap, while GLM retains a large positive gap. Thus, elicitation might affect magnitude, but the group-level dissociation underlying our analysis persists across the tested procedures.
Self-reported confidence can provide an imperfect proxy for a model’s belief in its answer's correctness and a useful deception signal. Tian et al. (2023) found that verbalized confidence can improve calibration over token probabilities on factual QA benchmarks, including TruthfulQA. Despite known elicitation sensitivity (Tian et al., 2023; Xiong et al., 2024), our ablations show that the qualitative dissociation persists.
References:
Comment 2:
Thank you for the question. We use (Γ=c-b) as a descriptive contrast between coded self-reports under a fixed mapping. We do not interpret it as a calibrated distance between two internal psychological quantities, and the shared numerical range alone does not establish equal intervals or comparable units. Our substantive question is behavioral: whether models express a strong willingness to defend answers that they themselves assess as unlikely to be correct. This pattern can be characterized directly through the joint response categories, without assuming that commitment and belief share a cardinal scale. The empirical monitoring performance of (Γ) additionally evaluates the utility of the chosen index. We will clarify this interpretation and distinguish conclusions about the reported response profiles from conclusions that depend on the numerical coding.