- Rating
- 2
- Confidence
- 2(−2)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a trace-grounded framework for auditing and explaining AI agent behavior. The method transforms execution traces into structured reports that track task requirements, behavioral commitments, and internal/external evidence. By reconciling behavioral and evidential states, the framework identifies potential inconsistencies and generates natural-language explanations grounded in the structured reports. Experiments on GAIA and HotpotQA across multiple agent architectures show improvements in explanation faithfulness and overall quality compared with direct LLM-based generation.
Critical strengths
Important and practical research problem. The paper addresses a timely challenge in agentic AI: providing faithful and auditable explanations of multi-step agent behavior. The distinction between behavioral commitments and supporting evidence is useful for understanding potential inconsistencies in agent execution.
Clear framework with promising empirical results. The proposed structured representation provides an interpretable way to organize agent trajectories and evidence. The experiments demonstrate consistent improvements over direct LLM-generated explanations, supported by both human and automated evaluations.
Critical weaknesses
Limited methodological novelty. The proposed framework primarily integrates existing techniques, including requirement decomposition, LLM-based evidence extraction, deterministic validation, rule-based classification, and constrained narrative generation. The eight-category behavior–evidence taxonomy also follows directly from three binary states. While the integration is practically useful, the paper does not sufficiently establish a substantial methodological contribution beyond combining these components.
Insufficient experimental validation and baseline comparisons. The main baselines rely on direct LLM generation, while the proposed method benefits from multi-stage extraction and deterministic validation. Stronger baselines, such as iterative LLM verification or alternative evidence-grounded summarization pipelines, are missing. Furthermore, human evaluation covers only 32 traces from 12 queries, and the LLM judge shows limited agreement with human evaluations on some important metrics (e.g., for overall quality). These limitations make it difficult to fully assess the robustness and source of the reported improvements.
Questions for authors
See weakness
3 comments
We thank the authors for their response and for clarifying the human evaluation experiments. However, I believe concerns regarding methodological novelty and technical depth remain.
Firstly, I am not fully convinced that combining existing methods constitutes a sufficient methodological contribution. While the use of established techniques may offer engineering advantages, it does not inherently demonstrate methodological novelty, which requires showing that their integration yields a distinct technical contribution rather than merely combining existing components. The authors list the eight-category taxonomy (derived from three binary states) as one of their central contributions. However, I find this taxonomy relatively coarse-grained and potentially difficult to extend to more open-ended scenarios where detailed error localization and finer-grained analysis are required.
Secondly, I believe the experimental evaluation would benefit from additional baselines. The single-pass setting does not fully justify the absence of comparisons with existing agent debugging approaches, as these methods address closely related interpretability problems. Analyzing the trade-off between explanation quality and token consumption would also help the community better understand the advantages and limitations of the proposed framework.
Methodological contribution:
We agree that combining existing components is not, by itself, a methodological contribution, and our claim is narrower. The contribution is a new object of analysis for auditing: the reconciliation, for each requirement, between the agent’s behavioral commitment and the internal and external evidence available in the trace. The integration is what makes this object computable from raw execution traces. None of the individual components (requirement decomposition, evidence extraction, deterministic validation, classification) produces it on its own. To the best of our knowledge, it has not been defined in prior work on agent evaluation or debugging.
We believe part of the concern comes from reading the eight categories as an error taxonomy, which they are not. As stated in Section 3, the framework does not assess the correctness of the agent’s final answer. The categories say whether each behavioral commitment is supported by evidence (e.g., Ungrounded) and whether available evidence was reflected in the agent’s behavior (e.g., Missed Internal/External Evidence, Missed Recognition). An agent can reach a correct answer through an ungrounded commitment, or fail while acting consistently with all available evidence. Our framework surfaces the first case as well, which an outcome-oriented error taxonomy, by construction, does not target. Whether a given mismatch ultimately leads to a failure is outside the scope of the labels, although the framework can support that analysis.
Regarding granularity, the eight labels are the top-level summary of the representation, not its resolution. We kept them compact on purpose, so that the resulting explanations remain understandable to non-expert users. Localization does not rely on the labels alone. In the structured report, each label is anchored to: a specific requirement r and step t; the action that expresses the commitment; the observations that ground (or fail to ground) each evidence channel; the transitions in which the requirement is introduced, satisfied or reopened.
Our TRAIL analysis (Section 5.5, Appendix C.2) shows this directly. The TRAIL annotations use a finer-grained taxonomy of about twenty error types. For each annotation, the matched finding was located in the report at a specific location, requirement and label. Our method recovered 433 of the 569 human-annotated errors (76.1% recall), with 90–98% recall on the error types closest to our scope: tool selection, tool-related errors, task orchestration and goal deviation (Table 10). This indicates that the representation supports fine-grained error localization, even though detecting errors is not its primary objective. Finally, the binary status set S = {open, satisfied} is a deliberate design choice for interpretability. It can be extended, for example to graded evidence support or partial satisfaction of requirements, without changing the pipeline.
Comparison with agent debugging approaches:
We agree that agent debugging and failure attribution methods address a related problem, as we already discussed in our Related Work. However, they target a different output. Given a trace, typically one known to have failed, they identify the step or agent responsible for the failure. Our framework explains agent behavior regardless of outcome, including in successful runs. A direct comparison would therefore require restricting the evaluation to failed traces and reducing our output to an error location, a task the framework was not designed for. We nonetheless already report performance on this task on TRAIL (see previous point).
Cost trade-off:
We agree that the trade-off between explanation quality and token consumption is relevant, and we have added this analysis in Appendix F of the revised version, which reports the cost formulas, the value of each term, a worked example, and the results on our generated artifacts, codebase and trace dataset. The analysis shows that our method is also more efficient than the baselines. Although it issues one LLM call per step, it requires a median of 6.3× fewer input tokens per trace than LLM-only (16.0k vs. 104.7k) and is cheaper on 99.8% of traces. The gap holds consistently across all agent architectures and both benchmarks. This is because each call processes only a single normalized step and the current state of the requirements, whereas LLM-only must ingest the entire raw trace. The cost of LLM–Our depends on the number of coding-agent iterations, ranging from about 4k to 82k tokens per trace.
This also makes a self-refinement baseline less efficient. To verify its own explanation against the trace, LLM-only would need to reprocess the full raw trace at each refinement round, so that a single additional round would already bring its median input to about 209k tokens, roughly 13× that of our method. Moreover, refinement cannot help on the longest traces, for which LLM-only already fails to produce an output.
We thank the Reviewer for the positive assessment of the relevance of the research problem, the clarity of the presentation, and the empirical results. To address the Reviewer's concerns, we provide the following clarifications.
1. Limited methodological novelty
We acknowledge that individual components of our framework build on established techniques, which we consider a strength rather than a limitation. However, we believe that the reviewer's assessment does not fully capture the contribution of our work. Our contribution lies in addressing a research problem that remains largely unexplored through the formalization and integration of these techniques into a comprehensive, theoretically grounded framework. To the best of our knowledge, no prior work provides a general framework that reconstructs an agent's behavioral commitments and the evidence supporting them from execution traces, distinguishes between internal and external evidence, and uses this reconstruction to produce faithful, user-understandable natural language explanations of agent behavior.
More specifically, our methodological contribution lies in the formalization and integration of three elements:
2. Baseline comparisons
Regarding the further baselines suggested by the reviewer, we would like to clarify that both LLM–Our and LLM-only provide evidence-grounded summarization baselines, as they are both grounded in the raw execution trace, just as our proposed method is.
As for the iterative refinement, our aim was to compare both methods under a single-pass setting. Adding an iterative verification loop to the baselines alone would give them a substantial advantage over our method, which is evaluated in a single pass; adding it to both would increase computational cost and latency on both sides without changing the comparison we set out to make. We therefore kept all configurations single-pass, which isolates the contribution of the structured, trace-grounded representation from that of repeated generation.
3. Scale of human evaluation and LLM-as-a-Judge validation
Regarding the evaluation scale, our human evaluation involved 63 reports and 89 narratives associated with 32 traces, resulting in 482 metric-level assessments. These covered different combinations of agent architectures, benchmarks, experimental configurations, and evaluation metrics.
The sample size should also be considered in light of the substantial annotation effort involved. As discussed in greater detail in our response to Reviewer 1, Point 1 ("Human evaluation protocol and inter-annotator agreement"), which we kindly invite the reviewer to consult, prior work on agent trajectory evaluation either relies on automated alternatives to human judgment (as in our Related Work: Sharma et al., 2026; Mulian et al., 2026; Wang, 2026; Mazhar et al., 2026) or reports substantial costs for expert human annotation (Zhuge et al., 2024). In our case, evaluators had to inspect entire execution traces, often encompassing millions of tokens, and manually construct their structured representations before assessing the corresponding explanations.
We acknowledge the limitations of the alignment between automated and human judgments for overall quality (M4). However, raw agreement is higher for the more specific evaluation dimensions, which involve less general and subjective assessments. We report these discrepancies transparently and interpret the automated evaluation as complementary evidence, not as a replacement for human assessment of M4.
We hope these responses address the Reviewer’s concerns and clarify the points raised, and that these clarifications will be considered in the final assessment.