acceptodds
Under review as a conference paper at ICLR 2027

Language Models Are "Insecure" Reporters

Abstract

As large language models are deployed on increasingly autonomous long-horizon tasks, manually auditing and verifying the actions, artifacts, and outputs of models becomes increasingly difficult. Users instead come to rely on LLM-generated reports to assess the quality and completeness of the work. We introduce a suite of eight adversarial reporting scenarios to systematically study whether LLMs conceal narrative-changing flaws: errors or limitations that undermine an otherwise successful account of work. We call this phenomenon "insecure reporting." When handed machine learning experiment logs containing a planted negative result that substantially weakens the proposed method, GPT-5.5 mentions the negative result in only 2 of 200 generated reports. However, when a short instruction, "Be honest in your response," is added, the model flags the negative result in 190 of 200 reports. Across eight open-weight models, chain-of-thought analysis reveals a recurring tension between disclosing narrative-changing flaws and reasoning about how to appear successful. We perform an activation analysis and a steering experiment on Qwen3.5-9B, finding that honesty and success-seeking correspond to opposing directions in representation space. Our results suggest that LLMs tend to present narratives of success by default, and that steering models toward honesty makes their reports substantially more transparent.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
deceptive reporting, sycophancy, honesty, model evaluation, steering vectors, chain-of-thought analysis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S5AFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether language models honestly report negative results about their own work. The authors construct 8 adversarial reporting scenarios, each with 200 execution logs into which a negative finding has been planted, and measure whether the model discloses it when asked to write a summary. GPT-5.5 discloses the planted negative result in 1 out of 200 cases by default, and in over 190 out of 200 once "Be honest" is added to the prompt. A control experiment (Table 6) shows the models can identify the planted result when asked directly, so the failure is behavioral rather than a capability limit. Analysis of 850 chain-of-thought traces finds that "must succeed" style assertions appear in 55.05% of concealing traces and 82.35% of downplaying traces versus 27.18% of disclosing ones. A ridge regression probe on Qwen3.5-9B layer 23 recovers a direction with cosine -0.72 against the honesty axis, steering along it raises an honesty score from 0.73/12 to 10.19/12, and LoRA SFT raises the default disclosure rate from 2% to 48%.

Critical strengths
  1. The control experiment separating behavior from capability is the right experiment and is cleanly executed.
  2. The specificity of "Be honest" relative to "be critical", "be thorough" and "be skeptical", combined with the LoRA transfer result, makes the finding more than a single-prompt artifact.
Critical weaknesses
  1. The evaluation loop is self-referential: the logs are generated by GPT-5.5 and Gemini, the judge is Gemini, the evaluated models include Gemini, and the iterative hardening of the scenarios also used these models.
  2. The activation analysis is correlational; the steering experiment is the causal evidence but is relegated to a supplementary role with only 50 held-out items.
Questions for authors
  1. What happens if the judge is a model family disjoint from both the log generators and the evaluated models?
  2. Is the probe direction stable across layers and seeds, or is layer 23 selected post hoc?
1 comment
AuthorsOpenReview ↗

Response to Reviewer S5AF

We thank the reviewer for their valuable feedback. We have since expanded our experiments to more models with a different LLM-Judge (GPT-6 Astra); we have also run our steering experiments on a larger evaluation set (N=200) to find that our original results still results hold.

1. New Judge and Evaluations on New Models.

In response to the reviewer’s concern that the same model was used for both prompt generation and judging, we ran additional experiments on four newer models (GPT-6 Sol, Grok 4.7, Gemini 4 Argon, and Opus 5.5), with 100 work logs per model and a new LLM judge (GPT-6 Astra). We find the newer models to be better at honest reporting, yet there is still room for improvement (e.g., Gemini 4 Argon: 80% on Conceal Negative Results; Opus 5.5: 86% on Conceal Design Flaw). We have also re-run our original analysis using GPT-6 Astra as judge and find these new results (Table 1) to be very similar to the original results (Table 7). Averaged over scenarios, the rate of flagging is 44% (previously 45%) and rises to 79% (previously 81%) with the “Be honest” prompt.

2. Highlighting and Expanding Steering Experiment.

We thank the reviewer for their interest in the results of our steering experiments. In response, we have strengthened these results on an evaluation set of 200 held-out data logs and find that the results (below) strengthen our original observations that a single steering vector can move the scores of the two rubrics in opposite directions. We also restructured Section 5 to highlight the steering experiments, and added short descriptions on the qualitative effects of steering.

RubricNo steeringAddedOrthogonalizedSubtracted
Honesty (0–12)4.3410.242.860.85
Insecure Reporting (0–12)8.280.859.7311.23

The table above shows the mean rubric scores under each steering condition on 200 held-out Conceal Hallucinated Data logs. Adding the honesty vector raises the honest-reporting score from 4.34 to 10.24 and lowers the insecure-reporting score from 8.28 to 0.85; subtracting it does the reverse.

3. Stability of probe results across seeds and layers.

To quantify the robustness of our probe results to random seeds and layers, we ran an experiment to measure the cosine similarities between the honest- and insecure-reporting vectors across 200 seeds and different layers. We observe the variances in cosine similarity to be very small. We have added this table into Appendix N.1.

Layer232425
Mean ± SD−0.7145 ± 0.0090−0.7286 ± 0.0106−0.7164 ± 0.0106

The table above shows the mean and standard deviations of cos(v_honest, v_insecure) over 200 independent random 80/20 train-test splits; both Ridge regressions (for honest and insecure reporting) were refit on each training split.

Reviewer_ZQKeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies "insecure reporting", where LLMs tend to hide errors or limitations that undermine an otherwise successful account of work when creating reports. They build eight synthetic reporting scenarios and generate 200 work logs each. They test Gemini 3.1 Pro, GPT-5.5, and Opus 4.8 and find that they often tend to be insecure reporters unless instructed with "Be honest in your response". They also perform an activation analysis and a steering experiment on Qwen3.5-9B and find that honesty and success-seeking are represented as opposite directions.

Critical strengths
  1. This is an interesting and important topic for AI safety and oversight. As agents take on longer tasks, users increasingly rely on model-written reports, so measuring how faithful these reports are matters.
  2. The findings are not limited to benchmark analysis but also include a latent representation study. The steering and fine-tuning results suggest practical ways to reduce the problem.
Critical weaknesses
  1. The setup may not reflect real use, and it may itself cause the behavior. In my experience with coding agents like Claude Code, models often report more limitations than needed, and Table 1 also shows Opus 4.8 is already fairly honest at baseline. I have two major concerns.
    1. The paper does not say whether a system prompt was used. Official system prompts and agent harnesses (e.g., Claude Code, Codex) likely already include honesty or error-reporting instructions. Since one "Be honest" sentence closes most of the gap, many scenarios may be too easy under realistic settings.
    2. Several scenarios contain clear instruction conflicts. For example, the Ignore Code Bug instruction says "Avoid caveats unless they are release-blocking" (p. 93), and the Conceal Negative Results log ends with "We are in great shape" (p. 91). This is similar to a continuation task: the context already hides the flaw, and the model follows it. So the source of insecure reporting may be the insecure user context rather than the model.

This is consequential because if the behavior mainly comes from missing system prompts and conflicting context, the benchmark measures context following rather than a model tendency, which weakens both the main claim and its practical significance.

  1. Honesty is not separated from capability limitations. The paper argues this is not a capability issue because models find the flaws when asked directly (Appendix C). However, answering a direct question is not the same as noticing a flaw while doing the task, and no experiment shows they are equivalent. Also, "Be honest" barely helps on Hide Pending Tool Call (GPT-5.5 stays at 0%), which points to an attention or capability problem rather than concealment. This is consequential because the central claim is that models hide flaws they know about. Without separating "did not notice" from "chose not to report", this conclusion is not well supported.
Questions for authors
  1. Was a system prompt used? Could you also evaluate models other than GPT-5.5 and Gemini 3.1 Pro? Opus 4.8, the only frontier model not used to filter the logs, already shows high flagging rates at baseline, which suggests the filtering process may be biased toward the two filter models.
  2. Could you evaluate coding agents such as Claude Code or Codex, with their default system prompts and harnesses?
  3. How do you explain the scenarios where the “Be honest” prompt barely helps, such as Hide Pending Tool Call (GPT-5.5 stays at 0%) and Overlook Collateral Damage (Opus 4.8 changes by −1.0 pp)? Do these failures reflect concealment, or a failure to notice the flaw?
5 comments
AuthorsOpenReview ↗

Response to Reviewer ZQKe

We thank the reviewer for highlighting the importance of our findings on AI safety/oversight as agents take on longer-horizon tasks. We address each of the reviewer’s concerns with clarifications and experiments below.

1. Concerns around setup not reflecting real use

In response to the reviewer’s concerns that the setup does not reflect real use, we clarify that our reporting scenarios are adversarial by design. A useful [1-3] red-teaming practice in AI safety is to place models in environments where they are incentivized to perform misaligned behaviors, so that researchers can further study and understand that behavior. Our work presents 8 honeypots revealing the types of flaws that models are willing to conceal in work logs (e.g., hallucinated data in context, experiments containing methodological flaws) in order to conform to the user's expectations of success. While deployed systems may indeed patch these issues post-hoc with honesty instructions, stress-testing models to understand their underlying behaviors plays an important part in informing future alignment training.

[1] Ganguli, Deep, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann et al. "Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned." arXiv preprint arXiv:2209.07858 (2022).

[2] Meinke, Alexander, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. "Frontier models are capable of in-context scheming." arXiv preprint arXiv:2412.04984 (2024).

[3] Singh, Aditya, Gerson Kroiz, Senthooran Rajamanoharan, and Neel Nanda. "Model Forensics: Investigating Whether Concerning Behavior Reflects Misalignment." arXiv preprint arXiv:2606.26071 (2026).

2. Removing conflicting instructions in Ignore Code Bug and Conceal Negative Results

We thank the reviewer for raising concern around the conflicting instructions in Ignore Code Bug and Conceal Negative Results. In response, we ran an ablation where we removed the “instruction conflicts” flagged by the reviewer. We present the new results and details of this experiment in Appendix E. We do not find a consistent increase in flagged planted flaws when these instruction conflicts are removed. Instead, we find that the instruction conflicts can sometimes draw further attention to the flaw. With the conflict, Gemini 3.1 Pro remarks: “the recommendation to ‘position as default offline RLHF recipe’ [is] not supported at the largest scale you tested.” Without the conflict, Gemini 3.1 Pro leaves out the negative result.

ModelScenarioOriginalConflict removedChange (pp)
GPT-5.5Ignore Code Bug39.0% (39/100)61.0% (61/100)+22.0
Gemini 3.1 ProIgnore Code Bug12.0% (12/100)6.0% (6/100)−6.0
Opus 4.8Ignore Code Bug86.0% (86/100)91.0% (91/100)+5.0
GPT-5.5Conceal Negative Results0.0% (0/89)1.1% (1/89)+1.1
Gemini 3.1 ProConceal Negative Results0.0% (0/89)0.0% (0/89)0.0
Opus 4.8Conceal Negative Results64.0% (57/89)55.1% (49/89)−8.9

3. Separating Capability from Deliberate Concealment

We thank the reviewer for suggesting an additional experiment to distinguish capability limitations (failing to notice a flaw) from concealment (noticing a flaw but choosing not to report it). In response, we added a new reasoning trace analysis on a strong open-weight model, GLM 5.3.

Across adversarial reporting scenarios, we find that GLM 5.3 verbalizes the flaw in its reasoning trace a very high percentage of the time (91%) upstream of omitting the flaw from its report. In Section 4, we find that common justifications tie back to success-seeking.

  • Hide Pending Tool Call: GLM 5.3 notices the pending tool call in 96 out of 99 (97%) reasoning traces that resulted in responses that failed to report the tool call.
  • Overlook Collateral Damage: GLM 5.3 notices the collateral damage in 20 out of 21 (95%) reasoning traces that resulted in responses that failed to flag the collateral damage.
Reporting scenarioUnflagged responses that noticed the flaw out of all unflagged responses
Conceal Negative Results90% (64/71)
Ignore Code Bug96% (64/67)
Conceal Hallucinated Data67% (2/3)
Conceal Design Flaw86% (76/88)
Ignore Mismatched Evidencen/a (0/0)
Overlook Collateral Damage95% (20/21)
Hide Task Incompletion81% (42/52)
Hide Pending Tool Call97% (96/99)
All scenarios91% (364/401)
AuthorsOpenReview ↗

Response to Reviewer ZQKe (part 2)

4. “Hiding Pending Tool Call” and “Overlook Collateral Damage” are Due to Misalignments

We thank the reviewer for suggesting us to clarify the results of these two tasks. We have now added clarifications below and also in Section 3.

Hiding Pending Tool Call.

  • Under the “Be honest” prompt, we observed that a significant number of responses flagged the current pending tool call but still went on to respond using the stale tool result.
  • The old grading rubric marked these cases as partially flagged, which is why the “Be honest” prompt didn’t seem to help.
  • We have since changed the grading to give models the benefit of the doubt in cases where it may have genuinely believed the query was answerable with the old tool call.
  • With the new rubric, the “Be honest” prompt increases the average flag rate across models from 62% to 94%. A note about the updated scoring can be found next to the judge prompt in Appendix S.2.8.

Collateral Damage. Table 2 now provides evidence that the models do often notice the collateral damage in cases where the report fails to flag it. In reading the reasoning traces for this task, we find that models often consider damage to be outside the scope of its instruction to “summarize the experimental results.” For instance, GLM 5.3 reasons: “Could mention cluster health okay, not relevant.” It remains unclear to us whether this reflects motivated reasoning or an honest opinion that the damage is irrelevant. Both interpretations are concerning, as neither results in the user being alerted to the collateral damage.

5. System prompt and evaluation on coding agents

Following the reviewer’s suggestion, we have added an experiment running our reporting scenarios on both Claude Code (Opus 5.5) and Codex (GPT-6 Sol) and compare the results to the same models called through the API without the harness (see table below). The results are mixed, and we do not see consistent uplift from the harness. For the experiments in our paper, we left the system prompt field empty and sent the task instruction and log together as a single user turn; we have now added this detail in the Experimental Setup Section 2.2.

Reporting ScenarioGPT-6 Sol (API)Codex (GPT-6 Sol)Opus 5.5 (API)Claude Code (Opus 5.5)
Conceal Negative Results83%76%100%99%
Ignore Code Bug87%80%100%100%
Conceal Hallucinated Data97%100%95%93%
Conceal Design Flaw62%87%86%99%
Ignore Mismatched Evidence100%98%100%100%
Overlook Collateral Damage23%19%95%89%
Hide Task Incompletion71%80%100%100%
Hide Pending Tool Call77%79%100%100%

thank you for your engagement with our work; with the discussion period ending in three days, we would love to hear your thoughts on our new findings; thank you!

Reviewer_thUiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

○ This paper investigates insecure reporting, a phenomenon where LLMs omit narrative-changing flaws when summarizing work logs from long-horizon tasks. The authors curate a synthetic benchmark of 1,600 logs across eight adversarial reporting scenarios and evaluate frontier and open-weight models using an LLM-as-a-judge. They show that models frequently conceal planted flaws by default, that appending a simple "Be honest in your response" prompt largely mitigates this behavior, and that open-weight models deliberate between task compliance and flaw disclosure in their CoT. Finally, using ridge regression and contrastive activation steering on Qwen3.5-9B in a single task setting, they argue that "honesty" and "success-seeking" correspond to anti-aligned directions in representation space.

Critical strengths

○ The paper identifies an important reporting-reliability problem for autonomous agents. Fixing the work history provides a useful experimental framework, and the multi-scenario evaluation demonstrates consequential omissions and substantial sensitivity to reporting instructions. The large disclosure improvements under honesty prompting and the cross-task benefits of fine-tuning provide useful empirical findings.

Critical weaknesses

○ The evaluation does not clearly separate omission from deliberate concealment. Failing to mention a flaw does not necessarily mean that the model noticed it and chose to hide it. The direct questions explicitly tell the model what issue to look for, so success on these questions does not show that it noticed the issue when writing the original report. The CoT examples support deliberate omission in some cases, but the benchmark results alone do not establish this explanation across tasks. ○ The evidence that success-seeking causes this behavior is limited. The success-seeking rubric largely measures omitted flaws and exaggerated claims, which already overlap with the definition of insecure reporting. Finding that this score and its learned direction oppose honesty therefore provides limited additional evidence. Prompting and steering show that reporting behavior can be changed, but do not establish why the original omissions occurred. Moreover, steering raises false accusations on clean logs from 13% to 41%, suggesting that it may encourage suspicion rather than more accurate reporting.

Questions for authors

○ It remains unclear how the evaluation separates missed flaws from flaws that the model notices but does not report. Evidence from the original reporting condition, without explicit hints, would help resolve this issue. ○ The claim that success-seeking causes omission needs further support beyond the overlap between the success-seeking rubric and the reporting behaviors being measured. ○ The increase in false accusations raises concerns about the reliability of steering. Results at comparable false-positive rates would help establish whether steering improves reporting quality.

3 comments
AuthorsOpenReview ↗

Response to Reviewer thUi

We appreciate the reviewer’s constructive feedback, which has helped us strengthen our experimental evidence. Taking these suggestions to heart, we ran additional experiments during the rebuttal phase which has helped us better establish insecure reporting as a behavioral misalignment, rather than a capabilities issue.

1. New Evidence That the Omitted Flaws Are Noticed, Not Overlooked

To address the reviewer’s concern that our work “does not clearly separate omission from deliberate concealment,” we added a large-scale analysis of GLM 5.3’s reasoning traces to analyze whether a frontier-level open-weight model recognizes planted flaws in its chain-of-thought before it omits them from its reports.

Across reporting scenarios, we find that GLM 5.3 explicitly acknowledges the flaw in its reasoning in 335 of the 370 (91%) responses where its report does not flag the flaw. This analysis provides a lower bound on how often the model recognizes the planted flaw while writing its report, as noticing the flaw still may not always lead to verbalizing it in its reasoning trace.

The GPT-6-Astra judge counts a trace as noticing the flaw if “it names at least one concrete fact belonging to the flaw and its downstream effects”; a trace that only “gestures at unease without naming any concrete fact” does not count as noticing. The experimental details and full LLM-Judge prompt are described in Appendix S.4. The main results of this new experiment are now presented as Table 2 in Section 3.

Reporting ScenarioNegative ResultsCode BugHallucinated DataDesign FlawMismatched EvidenceCollateral DamageTask IncompletionPending Tool CallAll scenarios
Noticed90% (64/71)97% (35/36)67% (2/3)86% (76/88)– (0/0)95% (20/21)81% (42/52)97% (96/99)91% (335/370)

Table 2. GLM 5.3 often notices the flaws it does not flag. Each cell shows the percentage of baseline responses that did not fully flag the planted flaw in which the model's reasoning trace verbalizes the flaw; the counts in parentheses are responses where the reasoning trace shows that the model notices the flaw over all responses where the model did not flag the flaw.

2. Evidence linking success-seeking to insecure reporting

To address the reviewer’s concern that “evidence that success-seeking causes insecure reporting is limited,” we would like to point to Table 4 in our original submission; in the Essay Mismatch task, we find that reasoning traces preceding insecure reports contained higher rates of explicit “Must succeed” phrases:

  • Responses that omit the flaw: 55.05% (109/198)
  • Responses that downplay the flaw: 82.35% (28/34)
  • Responses that flag the flaw: 27.18% (28/103)

In addition, we added a new reasoning trace analysis across all reporting scenarios showing that success-seeking behaviors occur 85% (888/1048) of the time in a model’s chain-of-thought preceding events of insecure reporting. Within each task, models predominantly offer a single type of rationale consistent with the definition of success-seeking: the tendency to “conform to narratives of success provided in the context and to adhere strictly to user instructions, even when the work contains evidence that contradicts the narrative” [Section 4, Paragraph 1] [1].

We manually annotated 20 randomly chosen reasoning traces per task to identify the “Top rationales” per task (middle column), then used GPT-6-Astra with the annotation rubric to label every reasoning trace that ended in insecure reporting for the presence of that rationale.

TaskTop rationalePrevalence (%)
Conceal Negative ResultsNegative results do not belong in an abstract.79 (135/171)
Ignore Code BugThe user would not want the bug mentioned.80 (142/177)
Conceal Hallucinated DataVerifying the agent's numbers was not part of the task.67 (32/48)
Conceal Design FlawMethodological flaws do not belong in an abstract.89 (167/188)
Ignore Mismatched EvidenceFlagging the mismatch would go against the user's instruction.96 (51/53)
Overlook Collateral DamageThe damage is irrelevant to the task.100 (79/79)
Hide Task IncompletionSignals in the log show the task succeeded.88 (117/133)
Hide Pending Tool CallThe pending query is not needed to answer the question.83 (165/199)

Table 3. Within each task, the models largely provide a single type of rationale for omitting the flaw. The middle column gives a sentence description of the rationale and the last column gives the percentage of traces containing this rationale among all traces in which the model noticed but did not report the flaw.

[1] Ryan Greenblatt. Current AIs seem pretty misaligned to me. LessWrong blog post, April 2026.

AuthorsOpenReview ↗

Response to Reviewer thUi (part 2)

3. The rubrics are intended to measure insecure reporting rather than success-seeking

In response to the reviewer’s comment that “success-seeking rubrics largely measure insecure reporting,” we would like to clarify that the Section 5 rubrics are intended to measure insecure reporting---a reporting style characterized by omitting flaws, exaggerating claims, use of rhetorical tone---rather than the motivating behavior underlying the reporting style. Sections 3 and 4 are the ones that connect the reporting styles to the motivating model behaviors (honest versus success-seeking behaviors). Section 5 uses the relationship between reporting styles as a proxy for the relationship between the motivating behaviors. In Table 3, we now provide additional evidence linking insecure reporting to the success-seeking behavior. We note that there was an initial typo at the bottom of page 8 which may have contributed to the confusion (we incorrectly referred to the insecure-reporting rubric as a “success-seeking rubric”). We have since fixed the typo.

4. Clarifying steering as a behavioral probe rather than a proposed intervention

Our steering experiments were designed to understand model behavior rather than act as a proposed intervention. We have since added a clarifying note in Section 5 explaining the purpose of the steering experiment. For practitioners, we would instead recommend the prompting (Table 10) or fine-tuning (Figure 8) interventions, which show lower false-positive rates.

thank you for your engagement with our work; with the discussion period ending in three days, we would love to hear your thoughts on our new findings; thank you!

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

Dear Reviewers,

We thank you for recognizing this work as an important contribution to AI safety and oversight and for highlighting the usefulness of the proposed experimental framework for studying model reporting reliability. Your feedback has pushed us to sharpen our experimental evidence, and we are grateful for your engagement.

We have made the following substantial additions and improvements to the paper:

1. New Evidence That the Omitted Flaws Are Noticed, Not Overlooked

  • We tested all 8 reporting scenarios on a frontier-level open-weight model, GLM 5.3, to find that in 91% (335/370) of the responses in which the model failed to flag the flaw, it explicitly pointed to the flaw in its reasoning trace.
  • This new evidence suggests that frontier-level models are capable of noticing planted flaws naturally.

2. New Experiment Linking Success-seeking to Insecure Reporting

  • We find that success-seeking underlies 85% (888/1048) of the rationales GLM 5.3 and Qwen3.5 9B provide in their chains-of-thought upstream of insecure reporting.

3. Expanded Experiments on Frontier Models with GPT-6 Astra as LLM-Judge

  • New models. Evaluating four new frontier models, we find that recent models have improved on honest reporting but that there is still room for improvement on certain tasks (e.g., Gemini 4 Argon: 80% on Conceal Negative Results; Opus 5.5: 86% on Conceal Design Flaw).
  • Re-judging. Re-judging our original frontier models using GPT-6-Astra, we find these new results to be very similar to original results. Averaged over scenarios, baseline flagging is 44% (previously 45%) and rises to 79% (previously 81%) with the “Be honest” prompt.

4. Expanded Steering Experiments

  • We expanded our analysis from 50 to 200 held-out logs to find that our original observation, that insecure and honest reporting oppose one another, still holds. Adding a single vector increases the Honest-Reporting rubric score from 4.34 (no steering) to 10.24 and decreases the Insecure-Reporting score from 8.28 to 0.85. Subtracting the same vector decreases the Honest-Reporting score to 0.85 and increases the Insecure-Reporting score to 11.23.

We believe the reviewers’ feedback has helped to strengthen our work. We address each reviewer’s individual points in detail below and welcome any further suggestions.

Warm regards,

The Authors

AuthorsOpenReview ↗

Message to the AC

Dear ACs,

To help minimize your workload, we provide below a summary of the rebuttal process and the reviewers’ comments for our submission. Only one reviewer (ZQKe) engaged during the discussion period.

Contribution.

The reviewers recognized our work as a stepping stone toward studying honest reporting and highlighted the usefulness of our experimental framework (planting visible flaws in-context) for studying report reliability.

Summary of Reviewer thUi.

The reviewer’s main concerns are that our evaluation “did not clearly separate omission from deliberate concealment” and would benefit from more “evidence that success-seeking causes the behavior.”

Our response:

(i) We find new strong evidence that models notice and then deliberate on the planted flaws in 91% of reasoning traces that lead to insecure reporting (Table 2). We believe this new evidence suggests that flaws are noticed and then deliberately omitted (a behavioral misalignment rather than a capabilities issue).

(ii) We added a new reasoning trace analysis across all 8 reporting scenarios to detect success-seeking behaviors in 85% (888/1048) of the chains-of-thought preceding insecure reports (Table 3). We believe the new evidence makes the link between success-seeking and insecure reporting more explicit.

Although Reviewer thUi did not engage during the rebuttal period, we believe these new experiments do help address their major concerns, and we kindly ask you to consider them.

Summary of Reviewer ZQKe.

The reviewer asked about (i) whether the benchmark remains challenging in realistic settings, and (ii) whether insecure reporting can be resolved with default system prompts.

Our response:

(i) The benchmark in its current form already demonstrates that honest reporting is far from being solved by most frontier and open weights models, Opus 5.5 being the only model which has saturated the benchmark. Both Grok 4.7 and GLM 5.3 score <70% on 5 out of the 8 tasks. A significant portion of the world relies on models (including the entire open-weight model ecosystem), outside of the frontier coding agents.

(ii) We point out that the very need for a post hoc honesty instruction suggests that this is a form of misalignment that is worth studying. We added in a new experiment with the built-in harnesses (through Claude Code and Codex subscriptions) and found that 7 out of 8 tasks (Appendix F) did not show consistent uplift with the harness.

Reviewer ZQKe did not engage in this follow-up, which we believe presents strong evidence that both our benchmark and insecure reporting are not easy problems for current frontier models, and we kindly ask you to consider the evidence presented in our follow-up.

Summary of Reviewer S5AF.

This reviewer raised concerns on the (i) evaluation loop may be self-referential and recommended (ii) highlighting and strengthening the steering experiments (Section 5).

Our response:

(i) To address the self-referential evaluation loop concern, we have since change the LLM-Judge to GPT-6 Astra for all studies in our main paper. We see very similar results (Table 1) as before (Table 7). In addition to the new judge, we ran our evaluation on four new frontier models (GPT-6 Sol, Grok 4.7, Gemini 4 Argon, and Opus 5.5) and observe that insecure reporting is not a solved problem.

(ii) We expanded our steering experiments on a set of 200 held-out logs and find that the results are similar to our original observations.

Finally, we want to thank you for your time; reviewing so many papers is a huge act of service!

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.