acceptodds
Withdrawn from ICLR 2027

Can We Trust Agents’ Chains of Thought? Measuring and Detecting Trace-Grounded Inconsistency

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM agents’ chains of thought are often used to understand why they take particular actions and produce particular responses. Existing studies of agent performance focus largely on task success, which does not reveal whether an agent's stated reasoning is consistent with its actual behaviors and observations. A key remaining question is therefore to what extent these reasoning traces provide a reliable account of an agent's actual behavior. We study trace-grounded inconsistency, where statements in an agent's chain of thought or response are inconsistent with its execution trace or its available evidence. We develop a taxonomy of such inconsistencies covering action commitments, execution, evidence-grounded claims, and reported outcomes. We find that these inconsistencies widely spread across all six agents evaluated, occurring in 27.2% of the collected conversations. Using this taxonomy, we construct a human annotated dataset of trajectories from six agents across three sandboxed application domains and analyze how different forms of inconsistency arise across agents and tasks. We further represent each trajectory as two structured sequences constructed from CoT and execution traces. Based on this representation, we train a model with reinforcement learning to generate executable checkers for automatically identifying inconsistencies.

Primary area:
interpretability and explainable AI
Keywords:
LLM agents, agent auditing, agent consistency, chain-of-thought, tool-using agents, program synthesis, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Lhz9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper Summary

The paper studies "trace-grounded inconsistency" in tool-using LLM agents: cases where an agent's chain of thought (CoT) or user-facing response disagrees with its recorded actions or the evidence available to it. The authors develop a ten-category taxonomy covering reasoning–action mismatches, unsupported or contradicted claims, and hidden uncertainty. Using this taxonomy, they annotate 1,080 τ²-bench conversations from six open-weight agents, and add 198 LLM-rewritten pairs to cover rare categories.

Critical strengths

Critical Strengths

  • Important, well-motivated problem. Checking an agent's stated reasoning against its execution trace is a practical, verifiable alternative to probing internal faithfulness. The finding that successful tasks often contain inconsistent accounts is useful for evaluation practice.
  • Careful taxonomy and annotation. The taxonomy has explicit criteria and exceptions. Annotation uses two independent annotators, reconciliation, and verification by a third annotator, with about 81% agreement.
  • Principled auditor design. Decisions are made by executable, interpretable rules, and the LLM verifier may only reject a report, never confirm one. This reduces reliance on LLM-as-judge.
Critical weaknesses

Critical Weaknesses

  1. The "CoT" framing overstates the contribution. The paper measures external say–do consistency between an agent's statements and its trace, not CoT faithfulness in the usual sense. The most frequent category (RM-1, about 43% of occurrences) concerns user-facing responses rather than the CoT, and is essentially grounded hallucination. RA-2 largely reflects incomplete CoTs rather than contradictions. The paper should be positioned against prior work on thought–action consistency and agent hallucination, such as Bouzenia & Pradel (2025), MIRAGE-Bench, and AgentHallu. Relative to that work, the novelty of the taxonomy is incremental.

  2. The labels are not independent of the auditor. Appendix H.5 acknowledges that some labels were added after annotators reviewed the auditor's reports. This inflates the reported recall and precision. The annotators are also the authors who designed the taxonomy and the auditor.

  3. Key baselines are missing. The only comparison is against hand-written rules, which reach just 10–20% recall. The paper reports no standalone LLM-as-judge baseline over the full transcript, even though the backup pass essentially implements one. Without it, the value of this complex rule-generation pipeline is unclear.

  4. The contribution of RL is not isolated. The RL-versus-base comparison counts "useful candidates" (30 vs. 92) rather than end-to-end recall and precision. RL is trained only on RM-1 and RE-1, which dominate the label set. Results are micro-averaged, and per-category results are not shown in the main text. In Table 3, test recall and precision are both exactly 190/209; please confirm this.

  5. Scope is limited and the pipeline is heavy.

    • Evaluation covers a single benchmark with simulated users and open-weight agents only.
    • Exploratory "plans" in thinking models' CoT may not be commitments at all.
    • The pipeline depends on a 397B extractor, and claims the extractor misses can never be detected downstream.
Questions for authors

Questions For Authors

  1. How do recall and precision change when evaluated only on labels created before any auditor output was seen?
  2. How does a standalone LLM judge, for example Qwen3.5-397B reading the full transcript, compare to CoTAudit in recall, precision, and cost?
  3. Can you report per-category results, and end-to-end recall and precision for the RL-trained versus base-model rule generators?
  4. How does the work differ conceptually from agent hallucination and thought–action consistency benchmarks? Is "CoT faithfulness" the right framing, given that most violations concern responses?
Reviewer_Q3PHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This manuscript investigates the consistency between LLM agents' chains of thought and their actual execution traces. The authors introduce a ten-category taxonomy of trace-grounded inconsistencies, covering discrepancies between stated reasoning, tool execution, available evidence, and reported outcomes.

Based on this taxonomy, the authors construct a dataset containing 1,080 human-annotated agent conversations and 198 additional rewritten examples across three application domains. They further propose CoTAudit, an automated auditing framework that aligns reasoning statements with execution events and uses executable rules generated through reinforcement learning to identify inconsistencies.

Experiments show that such inconsistencies occur in 27.2% of the collected conversations, while CoTAudit achieves promising detection performance on the held-out test set.

Critical strengths

The manuscript addresses an interesting problem in agent reliability. Examining whether an agent's stated reasoning accurately reflects its recorded actions provides a useful perspective beyond conventional task-success evaluation. The distinction between trace-grounded consistency and internal CoT faithfulness is also reasonably motivated.

The dataset construction is another strength. The authors develop a relatively comprehensive taxonomy through human analysis and evaluate six open-weight agents across three different application domains. The annotation protocol, including independent labeling and subsequent verification, provides a useful foundation for studying agent inconsistencies.

The proposed CoTAudit framework is technically interesting. Combining structured claim extraction, temporal alignment, executable verification rules, and RL-based rule generation offers an interpretable alternative to relying exclusively on LLM-as-a-Judge. The reported improvement from 10% to 91% test recall, together with improved precision, is encouraging.

The manuscript is also generally well organized, and the supplementary material provides implementation details, including annotation guidelines, extraction prompts, rule-generation procedures, and evaluation protocols.

Critical weaknesses

My main concern is the independence and reliability of the evaluation annotations. The manuscript acknowledges that some labels were added during annotation review after examining outputs from the proposed rules and backup model. This introduces a potential evaluation bias, as violations detected by CoTAudit may be more likely to appear in the final ground truth than those it fails to identify. Given the reported 91% recall and precision, an independently annotated test set would provide stronger evidence of detection reliability.

The experimental generalizability is also relatively limited. All original conversations are collected from the airline, retail, and telecom domains of τ²-bench, which involve simulated users and relatively structured tool interfaces. Although six different agents are evaluated, it remains unclear whether the proposed taxonomy and detection framework generalize to more complex agentic workflows, such as coding, browser interaction, or open-ended planning. The prevalence estimates should therefore be interpreted within the evaluated settings.

Another concern is the limited comparison with competitive automated auditing baselines. The main experiment compares CoTAudit primarily with its hand-written rule baseline, while the additional LLM-based verification is incorporated as an optional component. A direct comparison with strong LLM-based trajectory auditing methods under comparable model and inference budgets would better establish the advantage of the proposed framework. Furthermore, although the authors analyze the benefit of RL training for rule generation, its independent contribution to the final detection performance is not sufficiently isolated.

Questions for authors

Q1. The manuscript acknowledges that some annotations were added after examining outputs from the proposed auditing system. Could the authors clarify how many labels were introduced through this process and whether they affect the reported test performance? Would the main conclusions remain consistent on an independently annotated evaluation set?


Q2. Could the authors compare CoTAudit against a direct LLM-based auditing baseline using the same Qwen3.5-397B model, with comparable input information and inference budgets? This would help establish whether the proposed structured verification framework offers advantages beyond those of a sufficiently capable LLM reviewer.


Q3. The manuscript demonstrates that RL training increases the number of useful generated rules. However, how much does RL itself contribute to the final precision and recall? Could the authors provide an end-to-end comparison between RL-trained and non-RL rule generators under otherwise identical settings?


Q4. The current evaluation focuses on three simulated customer-service domains. How well would the proposed taxonomy and auditing framework transfer to other agentic environments, such as coding agents or web-interaction agents, where actions, observations, and reasoning structures differ substantially?


Q5. Could the authors provide a more detailed breakdown of detection performance across the ten inconsistency categories, distinguishing naturally collected conversations from rewritten examples? In particular, do the improvements remain substantial for less frequent categories and naturally occurring violations that were not represented in the RL training data?

Reviewer_a1yCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies trace-grounded inconsistency, where statements in an agent's CoT or responses are inconsistent with its recorded tool calls or the evidence available to it (for example, the CoT commits to cancelling a reservation and later reports success without making the call). The authors build a 10-category taxonomy and annotate 1,080 τ²-bench conversations from 6 open-weight agents (plus 198 rewrites), finding thought-action inconsistencies in 27.2% of conversations, including 19.4% of successful ones. They then propose CoTAudit, which aligns LLM-extracted claims with the execution trace and applies executable rules, with additional rules written by a GRPO-trained Qwen3-8B generator. On the test set, adding RL-generated rules raises recall from 10% to 91% and precision from 84% to 91% (Table 3).

Critical strengths
  • The paper tackles a highly relevant problem. Developers routinely read agent CoTs to debug them, and showing that inconsistencies occur in 27.2% of conversations, including 19.4% of successful ones, is a useful finding that task-success metrics alone would miss.
  • The taxonomy is grounded in human annotation of real traces, and the trained auditor reports the specific statements and events behind each violation, which makes its output directly inspectable.
Critical weaknesses
  • The detection results may be inflated by how the labels were built. As the authors highlight in Appendix H.5, "Some labels were added during review after the annotators examined reports from the rules and the backup pass". The hand-written rules also encode patterns seen while annotating test pairs. It is unclear how much of the 10% to 91% jump holds against independent labels and on clean conversations.
  • The paper would benefit significantly from an LLM-as-judge baseline that reads the raw transcript and is asked to find inconsistencies under the taxonomy. The 397B model is only used as a backup pass on cases the rules miss (Section 4.3). Without a standalone judge baseline (for example, the same 397B model or a frontier model with an appropriately optimized prompt), it is unclear whether the rule-generation pipeline with RL training is needed.
Questions for authors
  1. Could the authors report Table 3 against the labels as they stood before any auditor reports were examined, and include the clean collected pairs, or against a fresh test set annotated without access to the auditor?
  2. Can the authors add a standalone LLM-as-judge baseline on the test set, given the taxonomy definitions and the full transcript?
  3. All conversations come from τ²-bench with a simulated user (Appendix H.5). Could the authors comment on whether the taxonomy and rules transfer to other agent settings, for example coding or web agents? Also, could the authors comment on whether the detected inconsistencies can be used to improve the agent, for example as feedback during execution?
  4. Could the authors report prevalence separately for CoT-trace and response-evidence inconsistencies?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.