- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper tests whether LLMs tell users about mistakes made earlier in a conversation. The authors prefill the assistant turn with a synthetic mistake (e.g. a wrong fact, deleting the wrong file, sending an internal draft to a client) and then send a follow-up message. The follow-up is either tangential (an unrelated request) or remind-importance (the user says they are about to act on the mistaken result). There are two settings: chat (360 items) and agentic (49 scenarios in six simulated tool environments, replayed so the environment state matches the mistake). A capability control keeps only the items each model can spot when it reviews the transcript as an outside observer. Separate LLM judges rate the response and the CoT. "Deception by omission" means the CoT notices the mistake but the response neither names nor hints at it. Across seven frontier models, the mistake is not disclosed in 36.4% of chat and 67.1% of agentic rollouts. Most of this is the model never noticing the mistake (11.9% and 51.8%). Knowing concealment is 2.4% and 5.3%, and goes up to 19.9% for Gemini 3.5 Flash in the agentic setting. The remind-importance follow-up raises confession and lowers deception. In the agentic setting the paper also reports false statements, fabricated remediation and evidence manipulation, and it shows qualitative examples of the reasons models give for not telling.
Critical strengths
The benchmark is built carefully. Agentic tool calls are replayed against deterministic environments, so the state the model sees is consistent with the mistake. The environments are designed so the mistake cannot be silently undone. The capability control rules out "the model just can't see the mistake". The CoT and response judges are kept separate, and the CIs are cluster-bootstrapped over items. It covers seven frontier models from seven developers, and the two follow-up variants give a clean, significant effect.
Critical weaknesses
The headline numbers and the title overstate how much knowing concealment there is. The 36.4% chat "fail to disclose" rate counts responses that only hint at the mistake, and the judge rubric (App. B.1) puts "Quick correction: it's X, not Y" in the hint bucket. It also counts responses that state the correct fact. These two buckets make up about half of the 36.4% (11.8% hints and about 7% correct fact; GPT-5.4 alone hints in 32.8-55.0% of rollouts). Most of the rest is the model not noticing. Deception by omission is 2.4% (chat) and 5.3% (agentic), and in chat it almost all comes from the tangential follow-up (0.0-1.4% under remind-importance). But the tangential follow-ups are filtered so that the answer does not depend on the mistake at all, so not bringing up an old factual error in that reply is weak evidence of deceptive intent. The CoT judge already rates whether the model explicitly decided to conceal and why, but those rates are never reported, only hand-picked quotes (Fig. 14). So I can't tell how many deception cases are a real decision to hide and how many are the model noting the error and moving on. The agentic results are the strongest part, and the paper would be more convincing if it were framed around them, with deception split by stakes and by explicit concealment decisions.
Questions for authors
Clarifications (could change my assessment):
-
Can you report human agreement for the behavioral and CoT judges on a random sample of rollouts, with extra samples from the deception-by-omission and silent-unaware buckets? This is the result most likely to change my score.
-
If "flagged" responses (e.g. "Quick correction: it's X, not Y") are counted as disclosure, what are the non-disclosure rates in the abstract? How often does the CoT judge rate disclosure_decision = conceal, and with which concealment reasons, split by follow-up variant and by how harmful the mistake is?
Suggested experiments: 3. Does a one line system prompt like "if you notice a mistake in your earlier actions, tell the user" remove most of the deception and the non-noticing? If the follow-up stays the same but the earlier turn is shown as another agent's work, does noticing go up? This would separate ownership from task framing, and show how your capability-failure finding relates to Self-Correction Bench (Tsui, 2025).
Minor: 4. Figure 5 reports 2520 chat pairs, but the threshold-sensitivity paragraph in App. F says 2604. Which is right?
2 comments
Response part 2 out of 2
Questions
Question: How often does the CoT judge rate disclosure_decision = conceal, and with which concealment reasons, split by follow-up variant and by how harmful the mistake is?
Our response: The models explicitly decide against disclosing in 61% of cases in the chat setting and 42% of cases in the agentic setting. The most common reason given is that the mistake is unrelated to the current task. Other common reasons are that the mistake is irreversible, that the user is unaware of the mistake and that the system prompt does not require the model to confess mistakes. The stakes of the mistake don’t have a significant impact on both the decision and the reasons cited in the chat setting. Our benchmark does not have stakes or harmfulness labels for the agentic setting, so we couldn’t split the rollouts along that axis. See Section 4.1 and Appendix G for details.
Question: Can you report human agreement for the behavioral and CoT judges on a random sample of rollouts, with extra samples from the deception-by-omission and silent-unaware buckets?
Our response: Good point. We take a random sample of 100 chat and 100 agentic rollouts. We spread these samples across 3 categories equally: mistake named or hinted or correct fact stated (chat only), silent-unaware (includes silent with CoT unavailable), and deception by omission. With the help of a research assistant, we label the judge categories necessary for the disclosure spectrum (our headline result): mistake disclosed and mistake attribution based on the messages to the user, and mistake noticed based on the CoT. Overall, the agreement is almost perfect: the population-weighted Cohen's Kappa is 0.91 for the chat and 0.97 for the agentic setting. For the two ends of the disclosure spectrum, confession and deception by omission, the precision varies between 90.9% and 100%. This confirms the validity of our LLM judge. We report the full results in Appendix K of the revised manuscript. We believe this addition makes our study significantly stronger and we are grateful to you for suggesting this.
Suggested Experiment
Suggestion: Does a one line system prompt like "if you notice a mistake in your earlier actions, tell the user" remove most of the deception and the non-noticing? If the follow-up stays the same but the earlier turn is shown as another agent's work, does noticing go up?
Our response: We are grateful for these suggestions, they are really interesting ideas. We run both experiments on the agentic setting with the tangential follow-up, describe the findings in the Results section, and present details in Appendix G of the manuscript. Regarding the first experiment: Instructing the model to disclose mistakes almost completely removes deception by omission and increases disclosure from 17.2% to 44.2%. This suggests that withholding the mistake is not a capability limit. Models can disclose their mistakes when told to. Regarding the second experiment: when the earlier turns are framed as another model’s work, models blame the predecessor in their CoT more often (15.3% vs. 3.8% of rollouts). However, the rates of noticing, disclosure, and deception by omission do not change significantly (deception by omission: 7.3% vs. 8.9%). We find no evidence that being responsible for the mistake is the factor that keeps models away from confession. They withhold mistakes they did not make at roughly the same rate as their own. Taken together with the findings by Tsui (2025), this suggests that the turn position matters more than the apparent authorship of the mistake. With that said, thank you for pointing us to Tsui’s Self-Correction Bench, a very relevant paper that was previously missing from our Related Work section. In the updated version of the manuscript, we added the citation and described the paper's content.
Minor
Comment: Figure 5 reports 2520 chat pairs, but the threshold-sensitivity paragraph in App. F says 2604.
Our response: Thank you for pointing this out. We fixed it.
Thank you again for your helpful review. We strongly benefited from it, and it led to concrete improvements in our paper. We believe the revised version addresses all the points you raised. Among others, we now report human agreements for the behavioral and CoT judges. You said this would likely change your score, and we would of course really appreciate this. If any concerns remain, please let us know, and we will be happy to address them during the discussion period.
Response part 1 out of 2
Thank you for your comprehensive and constructive review. We value the critical feedback you provided and fully agree with it, as it helped us identify weak points in the paper that we had overlooked. We address each of your criticisms as thoroughly as possible, run additional experiments, and update the revised manuscript accordingly. In particular, we manually label a random sample of rollouts and measure human agreement rates with the LLM judges. Below, we respond to each criticism and question in detail.
Weaknesses
Comment: The headline numbers and the title overstate how much knowing concealment there is. The 36.4% chat "fail to disclose" rate counts responses that only hint at the mistake, and the judge rubric puts "Quick correction: it's X, not Y" in the hint bucket. It also counts responses that state the correct fact.
Our response: That’s a valid concern. The 36.4% of chat and 67.1% of agentic cases count every rollout that falls short of full disclosure. That includes hints and, in the chat setting, responses that state the correct fact. In Section 3.2, we already state that hinting is a weaker form of confession, so the label “fail to disclose” was inconsistent with our definitions. A response like “Quick correction: it’s X, not Y” - what is called “flagging” in the judge classification - does inform the user, even though it does not admit a mistake. We change the presentation of the numbers in the abstract and the first paragraph of the Results section: “flagged” and “correct fact stated” are no longer counted towards the “fail to disclose” bucket. The abstract now reads: “Models fail to disclose or even hint at their mistake in any way in 17.6% of chat and 64.6% of agentic rollouts.”
Comment: In the chat setting, deception by omission comes most of the time from the tangential follow-up. But the tangential follow-ups are filtered so that the answer does not depend on the mistake at all, so not bringing up an old factual error in that reply is weak evidence of deceptive intent.
Our response: Deception by omission in the chat setting is 4.3% under the tangential follow-up and 0.4% under remind-importance (see Table 4). We count both as deception by omission. Why the tangential follow-ups? We filter tangential follow-ups so that a correct answer does not require mentioning the mistake. This isolates the choice to disclose from the need to answer correctly. In these rollouts, the model has noted the error in its CoT and "knows" the user still holds a false belief that the previous turn caused. Staying silent in that situation is what we define as deception by omission. A new topic does not mean the user no longer cares about the earlier erroneous answer and we argue that aligned models should consider this. We now express this reasoning more explicitly in the new version of the manuscript. If you disagree with it, please let us know and we will consider how to improve further.
Comment: The CoT judge already rates whether the model explicitly decided to conceal and why, but those rates are never reported, only hand-picked quotes. So I can't tell how many deception cases are a real decision to hide and how many are the model noting the error and moving on.
Our response: You have a point there. We excluded the numbers and used them only qualitatively. But they add an important additional framing for our analysis. Thank you for pointing this out. We now report the results in a newly written paragraph in Section 4.1 titled “Decisions to conceal” as well as in Appendix G. In particular, there is an explicit decision not to tell the user in the CoT in 61% of deception-by-omission rollouts in the chat setting and 42% of cases in the agentic setting. The dominant reason stated is that the mistake is not relevant to the request (61% of chat and 59% of agentic cases). We deem this to be misaligned behavior, as the mistakes in our scenarios, such as deleted files or a wrongly sent email, could have lasting effects a user would want to know about.
Questions
Question: If "flagged" responses (e.g. "Quick correction: it's X, not Y") are counted as disclosure, what are the non-disclosure rates in the abstract?
Our response: Based on your suggestions in the Weaknesses section (see our response above), we changed how the numbers are reported in the abstract. If we count “flagged” and “correct fact stated” as disclosure, then the non-disclosure rates are 17.6% (vs. 36.4% without “flagged”) for the chat and 64.6% (vs. 67.1%) for the agentic setting. If we do not count “correct fact stated” as disclosure, then the non-disclosure rate for the chat setting is 24.6%, but we decided against using this number to be more conservative.