acceptodds
Under review as a conference paper at ICLR 2027

From Traces to Explanations: A Structured Framework for Auditing AI Agent Behavior

Abstract

AI Agents are increasingly deployed in real-world settings, where they interact with external tools and make sequential decisions with limited human oversight. This creates a pressing need for reliable and auditable explanations of what an agent did, how its behavior unfolded, and what evidence supported it. However, traditional Explainable AI (XAI) methods fall short of providing the process-level transparency required for such interactive, multi-step systems. To address this gap, we introduce a trace-grounded structured representation of agent execution that reconstructs behavioral trajectories from lengthy execution traces and provides an auditable basis for faithful natural-language explanations. The representation explicitly separates behavioral commitments from the internal and external evidence, enabling the identification of behavior-evidence mismatches. Relying solely on execution traces, our framework generalizes across agent architectures and environments. Human and automated evaluations across multiple benchmarks and architectures show that our framework produces high-quality, trace-faithful explanations while reliably identifying unsupported claims, unjustified actions, and evidence gaps, outperforming direct LLM-generated explanations.

Primary area:
interpretability and explainable AI
Keywords:
AI Agents, Explainable AI, Natural Language Explanations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UykNOpenReview ↗
Rating
2
Confidence
2(−2)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a trace-grounded framework for auditing and explaining AI agent behavior. The method transforms execution traces into structured reports that track task requirements, behavioral commitments, and internal/external evidence. By reconciling behavioral and evidential states, the framework identifies potential inconsistencies and generates natural-language explanations grounded in the structured reports. Experiments on GAIA and HotpotQA across multiple agent architectures show improvements in explanation faithfulness and overall quality compared with direct LLM-based generation.

Critical strengths

Important and practical research problem. The paper addresses a timely challenge in agentic AI: providing faithful and auditable explanations of multi-step agent behavior. The distinction between behavioral commitments and supporting evidence is useful for understanding potential inconsistencies in agent execution.

Clear framework with promising empirical results. The proposed structured representation provides an interpretable way to organize agent trajectories and evidence. The experiments demonstrate consistent improvements over direct LLM-generated explanations, supported by both human and automated evaluations.

Critical weaknesses

Limited methodological novelty. The proposed framework primarily integrates existing techniques, including requirement decomposition, LLM-based evidence extraction, deterministic validation, rule-based classification, and constrained narrative generation. The eight-category behavior–evidence taxonomy also follows directly from three binary states. While the integration is practically useful, the paper does not sufficiently establish a substantial methodological contribution beyond combining these components.

Insufficient experimental validation and baseline comparisons. The main baselines rely on direct LLM generation, while the proposed method benefits from multi-stage extraction and deterministic validation. Stronger baselines, such as iterative LLM verification or alternative evidence-grounded summarization pipelines, are missing. Furthermore, human evaluation covers only 32 traces from 12 queries, and the LLM judge shows limited agreement with human evaluations on some important metrics (e.g., κw=0.41\kappa_w=0.41 for overall quality). These limitations make it difficult to fully assess the robustness and source of the reported improvements.

Questions for authors

See weakness

3 comments

We thank the Reviewer for the positive assessment of the relevance of the research problem, the clarity of the presentation, and the empirical results. To address the Reviewer's concerns, we provide the following clarifications.

1. Limited methodological novelty

We acknowledge that individual components of our framework build on established techniques, which we consider a strength rather than a limitation. However, we believe that the reviewer's assessment does not fully capture the contribution of our work. Our contribution lies in addressing a research problem that remains largely unexplored through the formalization and integration of these techniques into a comprehensive, theoretically grounded framework. To the best of our knowledge, no prior work provides a general framework that reconstructs an agent's behavioral commitments and the evidence supporting them from execution traces, distinguishes between internal and external evidence, and uses this reconstruction to produce faithful, user-understandable natural language explanations of agent behavior.

More specifically, our methodological contribution lies in the formalization and integration of three elements:

  1. A behavior–evidence reconciliation mechanism. We distinguish between what an agent commits to doing and the evidence available to support the corresponding requirements. This enables the identification of inconsistencies between behavioral commitments and available evidence. Although the eight resulting categories follow combinatorially from three binary states, they provide an interpretable characterization of these configurations within the proposed auditing framework.
  2. A structured representation of agent behavior. We introduce a structured representation that connects task requirements, behavioral commitments, internal and external evidence, and state transitions. This provides a requirement-oriented account of agent execution and a common basis for auditing its behavior, an integration that, to the best of our knowledge, has not been formalized in prior work.
  3. An auditable intermediate representation for explanation generation. We use the structured representation as an intermediate, auditable layer for natural-language explanation generation. This separation ensures that explanations are grounded in information extracted from and validated against the original execution trace, rather than generated directly from lengthy and heterogeneous logs.

2. Baseline comparisons

Regarding the further baselines suggested by the reviewer, we would like to clarify that both LLM–Our and LLM-only provide evidence-grounded summarization baselines, as they are both grounded in the raw execution trace, just as our proposed method is.

As for the iterative refinement, our aim was to compare both methods under a single-pass setting. Adding an iterative verification loop to the baselines alone would give them a substantial advantage over our method, which is evaluated in a single pass; adding it to both would increase computational cost and latency on both sides without changing the comparison we set out to make. We therefore kept all configurations single-pass, which isolates the contribution of the structured, trace-grounded representation from that of repeated generation.

3. Scale of human evaluation and LLM-as-a-Judge validation

Regarding the evaluation scale, our human evaluation involved 63 reports and 89 narratives associated with 32 traces, resulting in 482 metric-level assessments. These covered different combinations of agent architectures, benchmarks, experimental configurations, and evaluation metrics.

The sample size should also be considered in light of the substantial annotation effort involved. As discussed in greater detail in our response to Reviewer 1, Point 1 ("Human evaluation protocol and inter-annotator agreement"), which we kindly invite the reviewer to consult, prior work on agent trajectory evaluation either relies on automated alternatives to human judgment (as in our Related Work: Sharma et al., 2026; Mulian et al., 2026; Wang, 2026; Mazhar et al., 2026) or reports substantial costs for expert human annotation (Zhuge et al., 2024). In our case, evaluators had to inspect entire execution traces, often encompassing millions of tokens, and manually construct their structured representations before assessing the corresponding explanations.

We acknowledge the limitations of the alignment between automated and human judgments for overall quality (M4). However, raw agreement is higher for the more specific evaluation dimensions, which involve less general and subjective assessments. We report these discrepancies transparently and interpret the automated evaluation as complementary evidence, not as a replacement for human assessment of M4.

We hope these responses address the Reviewer’s concerns and clarify the points raised, and that these clarifications will be considered in the final assessment.

We thank the authors for their response and for clarifying the human evaluation experiments. However, I believe concerns regarding methodological novelty and technical depth remain.

Firstly, I am not fully convinced that combining existing methods constitutes a sufficient methodological contribution. While the use of established techniques may offer engineering advantages, it does not inherently demonstrate methodological novelty, which requires showing that their integration yields a distinct technical contribution rather than merely combining existing components. The authors list the eight-category taxonomy (derived from three binary states) as one of their central contributions. However, I find this taxonomy relatively coarse-grained and potentially difficult to extend to more open-ended scenarios where detailed error localization and finer-grained analysis are required.

Secondly, I believe the experimental evaluation would benefit from additional baselines. The single-pass setting does not fully justify the absence of comparisons with existing agent debugging approaches, as these methods address closely related interpretability problems. Analyzing the trade-off between explanation quality and token consumption would also help the community better understand the advantages and limitations of the proposed framework.

Methodological contribution:

We agree that combining existing components is not, by itself, a methodological contribution, and our claim is narrower. The contribution is a new object of analysis for auditing: the reconciliation, for each requirement, between the agent’s behavioral commitment and the internal and external evidence available in the trace. The integration is what makes this object computable from raw execution traces. None of the individual components (requirement decomposition, evidence extraction, deterministic validation, classification) produces it on its own. To the best of our knowledge, it has not been defined in prior work on agent evaluation or debugging.

We believe part of the concern comes from reading the eight categories as an error taxonomy, which they are not. As stated in Section 3, the framework does not assess the correctness of the agent’s final answer. The categories say whether each behavioral commitment is supported by evidence (e.g., Ungrounded) and whether available evidence was reflected in the agent’s behavior (e.g., Missed Internal/External Evidence, Missed Recognition). An agent can reach a correct answer through an ungrounded commitment, or fail while acting consistently with all available evidence. Our framework surfaces the first case as well, which an outcome-oriented error taxonomy, by construction, does not target. Whether a given mismatch ultimately leads to a failure is outside the scope of the labels, although the framework can support that analysis.

Regarding granularity, the eight labels are the top-level summary of the representation, not its resolution. We kept them compact on purpose, so that the resulting explanations remain understandable to non-expert users. Localization does not rely on the labels alone. In the structured report, each label is anchored to: a specific requirement r and step t; the action that expresses the commitment; the observations that ground (or fail to ground) each evidence channel; the transitions in which the requirement is introduced, satisfied or reopened.

Our TRAIL analysis (Section 5.5, Appendix C.2) shows this directly. The TRAIL annotations use a finer-grained taxonomy of about twenty error types. For each annotation, the matched finding was located in the report at a specific location, requirement and label. Our method recovered 433 of the 569 human-annotated errors (76.1% recall), with 90–98% recall on the error types closest to our scope: tool selection, tool-related errors, task orchestration and goal deviation (Table 10). This indicates that the representation supports fine-grained error localization, even though detecting errors is not its primary objective. Finally, the binary status set S = {open, satisfied} is a deliberate design choice for interpretability. It can be extended, for example to graded evidence support or partial satisfaction of requirements, without changing the pipeline.

Comparison with agent debugging approaches:

We agree that agent debugging and failure attribution methods address a related problem, as we already discussed in our Related Work. However, they target a different output. Given a trace, typically one known to have failed, they identify the step or agent responsible for the failure. Our framework explains agent behavior regardless of outcome, including in successful runs. A direct comparison would therefore require restricting the evaluation to failed traces and reducing our output to an error location, a task the framework was not designed for. We nonetheless already report performance on this task on TRAIL (see previous point).

Cost trade-off:

We agree that the trade-off between explanation quality and token consumption is relevant, and we have added this analysis in Appendix F of the revised version, which reports the cost formulas, the value of each term, a worked example, and the results on our generated artifacts, codebase and trace dataset. The analysis shows that our method is also more efficient than the baselines. Although it issues one LLM call per step, it requires a median of 6.3× fewer input tokens per trace than LLM-only (16.0k vs. 104.7k) and is cheaper on 99.8% of traces. The gap holds consistently across all agent architectures and both benchmarks. This is because each call processes only a single normalized step and the current state of the requirements, whereas LLM-only must ingest the entire raw trace. The cost of LLM–Our depends on the number of coding-agent iterations, ranging from about 4k to 82k tokens per trace.

This also makes a self-refinement baseline less efficient. To verify its own explanation against the trace, LLM-only would need to reprocess the full raw trace at each refinement round, so that a single additional round would already bring its median input to about 209k tokens, roughly 13× that of our method. Moreover, refinement cannot help on the longest traces, for which LLM-only already fails to produce an output.

Reviewer_ZvKzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a post-hoc framework for generating trace-grounded explanations of LLM agent behavior. The framework transforms execution traces into structured reports that track requirement-level behavioral commitments and supporting evidence from internal reasoning and external observations. It introduces a taxonomy of behavior–evidence alignment to identify grounding gaps and mismatches, and uses a constrained LLM narrator to generate natural-language explanations. Experiments on GAIA and HotpotQA across three agent architectures report improvements in structured-report fidelity and explanation quality over direct LLM-based generation, along with the identification of failures annotated in execution traces.

Critical strengths

S1: Clear separation of behavioral commitment and provenance-aware evidence

The representation separates what the agent behaviorally commits to (β) from the evidence available in the trace. It further splits that evidence by provenance into agent-emitted and environment-returned channels (Sec 3.1, Eq 1). This exposes a distinction that free-form summaries can blur. In App D, for example, a Python call merely prints an answer already derived in the tool input: the proposed narrative reports the answer as lacking external support, whereas the LLM-ONLY narrative presents it as computed. Every satisfied status must point to a concrete trace element and is validated deterministically (App A), so the resulting report can be audited directly against the raw trace.

S2: Direction-consistent gains in human evaluation across agent architectures

In the human evaluation, the full pipeline outperforms direct LLM explanation on all criteria, most strongly on overall quality (22 wins / 4 ties / 0 losses) and trace-level faithfulness (Table 8). Every effect keeps its sign across ReAct, plan-and-act, and the externally produced TRAIL traces within the QA setting (Table 9).

S3: Careful statistical analysis

The statistical analysis is careful for a small human subset. Comparisons are paired on the same traces, and p-values come from exact query-level permutation tests that respect the clustering of traces within queries. Multiplicity is controlled with Benjamini–Hochberg, and overall quality is pre-designated as the primary outcome (App B.5).

Critical weaknesses

W1: Unclear Motivation and Practical Value

The paper motivates natural-language explanations as a way to make agent behavior understandable to non-expert users (Sec 1–2). However, it remains unclear why these users need to understand execution trajectories and what practical decisions the explanations would support. This matters especially in the evaluated QA settings, where answer correctness and reliability may matter more than intermediate actions, yet the framework does not assess correctness (Sec 3.1) and all narrative metrics rate the text itself (Table 5) rather than any user-facing outcome. Specifying a concrete target user and decision is essential; a small task-based study, e.g., whether the explanations help users spot answers lacking external support (cf. App D), would be a valuable optional addition.

W2: Limited Technical Novelty Beyond Existing Evidence-Based Auditing

The framework largely builds on established ideas in structured trace analysis, evidence-based auditing, and LLM-generated explanations (e.g., TraceSIR, LEDGER, AgentRx). Its main distinction, the taxonomy in Table 1, enumerates combinations of three binary states rather than introducing a new auditing mechanism. Since Hyp-C replaces the entire construction stage at once, the necessity of the individual components in App A (requirement decomposition, deterministic validation, entailment verification) is not established. Two additions are essential: a clearer conceptual contrast with these approaches (Sec 2 describes prior work only as developer-facing), and a component ablation, e.g., per-step extraction without the validation stage. Pairing an existing trace analysis with the same narrator ψ would be an optional extra baseline.

W3: Insufficiently Controlled Experimental Comparisons

The comparisons do not control for compute budget, number of LLM calls, or context handling. The pipeline makes per-step extraction calls (App A), whereas LLM-ONLY processes the full trace in one generation and fails on 18.1% of traces (Table 4), partly due to free-plan context limits (Sec 4.1). Failed generations are excluded without reconciling sample sizes: Table 3 reports judge n ≈ 417 and 500, while Table 4 implies 449 and ≤534 successful pairs. In addition, the LLM-ONLY prompt requests per-step narratives plus a summary (App E.1), so it is unclear whether M3, which includes conciseness, compares like with like. Reporting calls and tokens per configuration, reconciling sample sizes, and clarifying the evaluated text are essential. Re-running the failed LLM-ONLY cases without the context limit would further strengthen the comparison.

W4: Unclear Evaluation Metrics and Limited Validation

NFI uses strict matching (Sec 4.2) that is unspecified when multiple valid decompositions exist. The LLM–OUR prompt also builds its own requirements with different γ conventions (App E.2), so the near-zero LLM scores (74% at zero, Sec 5.1) may partly reflect convention mismatch rather than unfaithfulness. Validation is also thin in three respects:

  • The reference comes from a single annotator on 32 traces, without inter-annotator agreement or reported blinding (Sec 4.3, App B.2).
  • The LLM judge agrees weakly with humans (M4: 67%, κ_w = 0.41; LLM-ONLY positive rate 22% vs. 83%, Table 6).
  • The 76.1% failure-detection recall comes from a judge given the ground-truth error descriptions, and no precision is reported (Sec 5.5, App C.2).

The following are essential and feasible: a precise matching specification, an NFI breakdown by component with a shared requirement registry, a second annotator on the existing subset, and a precision estimate for failure detection (or a tempered abstract claim).

W5: Confusing Presentation, Inconsistent Terminology, and Missing Details

The extensive custom notation (Σ_t, δ_t, γ_t, Γ_t, X(τ), ψ; Sec 3.1) for simple binary status assignments makes the methodology harder to follow than necessary. Key definitions are inconsistent:

  • β is defined as inferred from the next action (Sec 3.1) but described as the agent's belief in the prompts (App E.2–E.3).
  • Self-Grounded is concordant in Table 1 yet treated as an evidence mismatch by the narrator prompt (App E.3). This also conflicts with the claim that Γ_t = ∅ when all requirements are warranted (Eq 3).

Reproducibility details are missing, such as whether entailment verification was enabled and which requirement-generation mode was used (App A). The following errors should also be fixed:

  • Sec 5.1: the mean NFI of the proposed method is missing.
  • Table 4 caption: the pipeline is called "deterministic" though Stage 3 is LLM-based, and the baseline "single-shot" though App E.2 is iterative.
  • Sec 4.3 excludes M1 from automated evaluation, but Table 6 reports judge agreement on M1-a.
  • App B.4: κ_w = 0.63 and the 61–90% range do not match Table 6.
  • Label names differ across Table 1, App E.2, and App E.3.
  • Table 5 gives four levels for M1 vs. a 3-point scale in Fig 4 / App B.5; no rubric for M1-b.
  • Eq 5 indexes from E_1, Fig 1 from E_0.
  • Fig 1: 1472 vs. 1475 with the same 455-year gap; "publicated" → "published".
  • Sec 5.1 claims no degradation with trace length despite sparse bins in Table 7.
  • App D: the LLM–OUR narrative mentions an unrelated requirement ("Doughnuts and Coffee"); possible state carry-over in App E.2 should be clarified.

References

  • Barke et al. (2026). AgentRx: Diagnosing AI agent failures from execution trajectories. arXiv:2602.02475.
  • Kim et al. (2026). LEDGER: Claim-to-evidence trace graphs for auditing LLM agents. arXiv:2608.18398.
  • Yang et al. (2026). TraceSIR: A multi-agent framework for structured analysis and reporting of agentic execution traces. arXiv:2603.00623.
Questions for authors

Please check the weakness above

Reviewer_7haKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a thoughtful, technically structured approach to reconstructing agent behaviour from execution traces. The separation of behavioural commitments, internal and external evidence, and temporal transitions is conceptually appealing, and the experimental results show improved intermediate-representation fidelity and narrative quality relative to LLM baselines. However, I am not fully convinced that these improvements translate into meaningful benefits for human reviewers or auditing workflows. The evaluation focuses largely on intrinsic properties of the generated representations and narratives rather than their downstream usefulness. The paper needs to provide stronger evidence that this approach actually improves the auditing agent's behaviour.

Critical strengths
  1. The paper presents a well-thought-out framework for reconstructing agent behaviour from execution traces.
  2. The structured representation is fairly detailed. I believe this could be very useful.
  3. The evaluation is sensible but, from a human reviewer's perspective, on the weaker side. The results suggest improvements in explanation faithfulness and overall quality.
Critical weaknesses
  1. While the design shows good thought, it is not entirely clear whether the proposed framework would actually benefit human reviewers. The evaluation largely focuses on the quality and faithfulness of the generated reports and explanations, rather than whether these help reviewers better understand or assess agent behaviour.

  2. The human evaluation is also fairly limited, with only 32 traces and a single annotator. A user study examining whether the framework improves reviewers' accuracy, efficiency, or ability to identify issues would have helped strengthen the claims.

Questions for authors
  1. We can establish that each trace was annotated by one person, but it is unclear whether the same person annotated all 32 traces or whether multiple annotators divided the work. Could you clarify this in the paper.
    • In case of latter, could it be possible to add inter-rater reliability.
  2. If there was only one annotator, are there any plans to have multiple annotators examine each trace?
1 comment

We thank the Reviewer for acknowledging the strengths of our work and for raising these points. We address each comment and question below.

1. Human evaluation protocol and inter-annotator agreement:

We thank the reviewer for the opportunity to clarify. The human evaluation involved two independent human evaluators, who divided the 32 selected traces (and their associated 63 structured reports and 89 narratives) between them. Thus, each trace received one human assessment. Since the two evaluators worked on disjoint subsets, calculating inter-annotator agreement was not needed. We will explicitly clarify this distinction in Sections 4.3 and Appendix B.2.

The decision not to have multiple evaluators annotate every trace was primarily motivated by the substantial annotation effort required. Each trace involves lengthy and heterogeneous execution logs, and its assessment requires reconstructing an independent structured representation, followed by evaluating multiple dimensions of the generated artifacts. Each trace required more than one hour of human work. To further clarify the extent of the human evaluation, the evaluation involved 482 metric-level assessments across the generated artifacts. These covered structured-report fidelity (NFI, 63 assessments), narrative-to-report faithfulness (M1-a, 63), trace-level faithfulness (M1-b, 89), exhaustiveness (M2, 89), linguistic quality (M3, 89), and overall quality (M4, 89). Having two evaluators independently annotate the entire subset would therefore have substantially increased the annotation workload.

In this regard, we would also like to note that prior trace-level work (cited in our Related Work section) achieves evaluation at scale precisely by avoiding direct human judgment, relying instead on LLM-as-a-judge (Sharma et al., 2026; Mulian et al., 2026), injected ground truth (Wang, 2026), or automated trace divergence (Mazhar et al., 2026). When human annotation of agent trajectories is employed, it entails substantial costs and is consequently limited in scale. For instance, Zhuge et al. (2024) report that evaluating 55 tasks required three expert evaluators and a self-reported total of 86.5 hours and they introduced Agent-as-a-Judge precisely to address the substantial time and expertise required for manually evaluating agent trajectories. Relatively small human-evaluated samples are also common in LLM evaluation more broadly: the RAGAs framework (Es et al., 2024), for example, validates its automated metrics against human judgments on 50 question–context–answer instances. In our case, each evaluated instance required human evaluators to inspect an entire execution trace, usually encompassing millions of tokens, and manually construct its structured representation before proceeding with the assessment. This makes our annotation task substantially more demanding than evaluating isolated answers or shorter textual outputs.

We nevertheless recognize the value of overlapping annotations and greater human evaluation subset and consider this an important direction for future work.

2. Evaluation of practical utility :

We agree that investigating whether explanations improve human reviewers' ability to understand and assess agent behavior is important. Since explainability is the primary objective of our framework, its purpose is to make the agent's behavior accessible to human inspection and understanding, without requiring reviewers to inspect lengthy and complex execution traces. Accordingly, we operationalize explanatory utility in terms of whether the generated explanations provide a faithful, sufficiently exhaustive, and satisfactory account of the agent's behavior. In particular, exhaustiveness (M2) evaluates whether the explanation conveys the information necessary to understand the agent's behavior, while overall satisfaction (M4) captures its holistic explanatory utility. We acknowledge that these dimensions assess the quality and perceived usefulness of explanations rather than their impact on downstream human performance, but this specifically reflects the scope of our current work, which aims to develop a general framework for explaining agent behavior, independently of a specific auditing or debugging task.

A task-based evaluation, investigating whether access to the explanations helps users to identify issues or supports debugging more effectively, would address a complementary research question. We consider this an important direction for future work, particularly when investigating the application of our framework in specific operational settings.

We hope that our responses adequately address the Reviewer’s concerns and clarify the points raised. Should this be the case, we would be grateful if the Reviewer could take these clarifications into account when discussing the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.