acceptodds
Under review as a conference paper at ICLR 2027

TraceSIR: A Multi-Agent Critique Framework for Structured Analysis and Reporting of Agentic Execution Traces

Abstract

Agentic systems augment large language models with external tools and iterative decision making, enabling complex tasks such as deep research, function calling, and coding. However, their long and intricate execution traces make failure diagnosis and root cause analysis extremely challenging. Focusing solely on final task outcomes further discards critical behavioral information required for accurate issue localization. To address these limitations, we introduce **Trace Critique** as the task of producing structured analyses and reports over agentic execution traces, and propose **TraceSIR**, a multi-agent framework that operationalizes this task. TraceSIR coordinates three specialized agents: (1) StructureAgent, which introduces a novel abstraction format, *TraceFormat*, to compress execution traces while preserving essential behavioral information; (2) InsightAgent, which performs fine-grained diagnosis including issue localization, root cause analysis, and optimization suggestions; (3) ReportAgent, which aggregates insights across task instances and generates comprehensive analysis reports. To evaluate TraceSIR, we construct **TraceBench**, covering three real-world agentic scenarios, and introduce *ReportEval*, an evaluation protocol for assessing the quality of analysis reports aligned with industry needs. Experiments show that TraceSIR consistently produces coherent, informative, and actionable reports, significantly outperforming existing approaches across all evaluation dimensions. We further show that TraceSIR provides correct and actionable diagnoses, matching human-annotated root causes, achieving higher scores in a format-independent factual-grounding audit, and enabling substantial recovery of originally failed tasks. Our code is publicly available.

Primary area:
datasets and benchmarks
Keywords:
trace critique, agentic execution trace, multi-agent framework, report generation, issue localization, root cause analysis, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kPNdOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Trace Critique, the task of producing structured, evidence-grounded analyses of agent execution traces, and proposes TraceSIR, a multi-agent framework that compresses and structures lengthy traces, performs instance-level error localization and root-cause analysis, and aggregates diagnostic findings into comprehensive reports. The authors also construct TraceBench, a benchmark comprising 150 failed traces from deep research, function-calling, and agentic coding tasks, and introduce ReportEval to assess report quality and practical utility. Experiments show that TraceSIR outperforms general-purpose agent systems in report quality, factual grounding, diagnostic accuracy, and the recovery of failed tasks through report-guided re-solving.

Critical strengths
  1. The experimental evaluation is comprehensive and covers three representative agentic scenarios: deep research, function calling, and agentic coding. It combines human evaluation, LLM-based evaluation, diagnostic-correctness analysis, factual-grounding audits, and report-guided task recovery, providing a strong and multifaceted evidence chain.
  2. The paper introduces TraceBench, a dedicated protocol for evaluating the quality and practical utility of trace-analysis reports. These resources could support future research on agent diagnosis and evaluation.
  3. The overall framework is logically designed and well aligned with the target problem. StructureAgent addresses lengthy and redundant traces, InsightAgent performs fine-grained instance-level diagnosis, and ReportAgent aggregates recurring patterns across cases. This division of responsibilities makes the pipeline coherent and interpretable.
  4. TraceSIR achieves substantial and generally consistent improvements over general-purpose agent baselines, particularly in error analysis, root-cause analysis, and factual grounding. The report-guided re-solving results further demonstrate that its diagnoses are actionable rather than merely descriptive.
Critical weaknesses
  1. The methodological novelty appears limited and is primarily derived from system-level integration. As acknowledged in the related-work section, the individual components—including trace structuring, length-aware summarization, per-instance diagnosis, and cross-instance aggregation—are closely related to existing techniques. While their integration into an end-to-end Trace Critique pipeline is practically meaningful, the paper does not clearly establish which technical component constitutes a fundamentally new method rather than a well-engineered composition of existing LLM-based analysis procedures.
  2. The baseline selection does not fully isolate the benefit of the proposed framework. Consequently, the experiments mainly show that a specialized pipeline outperforms generic agents on its target task. Comparisons with stronger task-specific alternatives, such as hierarchical summarization, retrieval-based trace diagnosis, map-reduce aggregation, and carefully matched single-agent pipelines, would be necessary to substantiate the methodological advantage of TraceSIR.
  3. The paper lacks sufficient component-level ablations and compute-matched comparisons, making it unclear whether the gains arise from TraceFormat, length-aware abstraction, multi-agent decomposition, or simply a larger inference budget.
  4. The paper reports absolute token usage, latency, and monetary cost, but does not compare them against the baselines or evaluate quality–cost trade-offs. Therefore, the claimed efficiency and scalability advantages remain insufficiently supported.
Questions for authors

Please refer to the Weaknesses

Reviewer_3RzTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces TraceSIR, a multi-agent framework that structures execution traces, performs instance-level failure diagnosis, and aggregates findings into cross-case analysis reports. It also introduces TraceBench, comprising 150 failed trajectories across three agentic environments, and ReportEval, a report-quality evaluation protocol supplemented by expert annotations and report-guided task recovery experiments.

Critical strengths
  1. The paper addresses a practically relevant problem: extracting actionable diagnostic insights from long and complex agent execution traces.
  2. The evaluation spans three distinct application settings and includes both expert-referenced diagnostic assessment and downstream task recovery.
Critical weaknesses
  1. The benchmark relies heavily on subjective evaluation, limiting reproducibility and comparability. The main evaluation uses expert ratings and LLM judges to assess dimensions such as report structure, root-cause analysis, and overall usefulness. Although Table 3 reports agreement with expert annotations and Table 4 measures task recovery, diagnostic matching still depends on an LLM judge, and the benchmark lacks a sufficiently specified, reproducible scoring procedure for its central diagnostic objectives. Different expert panels or judge configurations may produce different results, making reliable comparisons across future methods difficult. More operationalized targets such as annotated error steps, violated constraints, or omitted actions with explicit matching rules would strengthen the benchmark.
  2. The downstream improvements are limited and uneven across environments. In Table 4, TraceSIR improves recovery from 24% to 26% on Deep Research and from 14% to 16% on Agentic Coding, corresponding to only one additional successful task in each 50-task setting. The larger improvement is concentrated in Function Calling, from 84% to 96%. Without repeated executions, uncertainty estimates, or paired significance tests, the two smaller gains cannot be reliably distinguished from stochastic variation. Furthermore, comparison only against a generic retry does not establish whether TraceSIR provides more useful guidance than an alternative diagnostic method. These results therefore provide limited support for broad claims of consistently actionable improvements across environments.
Questions for authors
  1. Can the authors provide a fixed, executable scoring protocol for diagnostic correctness, including explicit matching criteria and fully specified judge configurations? Which evaluation targets could be scored deterministically, and how stable are method rankings across different expert panels and judge models?
  2. Can the authors report repeated, paired re-solving experiments with confidence intervals and significance tests, particularly for the one-task gains in Deep Research and Agentic Coding? Could they also compare TraceSIR-guided recovery against recovery guided by ClaudeCode-generated diagnoses, rather than only a generic retry hint?
Reviewer_MD2pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Tracesir, which is a three agent pipeline. (Structureagent, insightagent, report agent).

The pipeline copresses agent execution traces, diagnoses each failed trace, and writes a cross-case analysis report.

The author evaluates this framework on tracebench.

Critical strengths
  1. the problem is practical.
  2. the author reports cost and releases anonymous code and demo
  3. the evaluation goes beyond report style the author add many things like gold root-cause annotations, a factual-grouding audit, etc; I think this is therigt direction for testing wheteher reports are correct and not just well formateed.
Critical weaknesses
  1. My main concern is that the main comparison has a very smal sample. Each method produces one final report per scenario, so tables 1 and 2 compare 3 reports per method and backbone, with no generatiion variance and no significance tests. And with claude, the llm judge gains are only 1.3, 3.3, and 0.7.
  2. the diagnostic correctness results have no baseline. the author only compare with 50% random baseline which does not mean much for this task. running claudecode, codex etc through the same gold-based check is needed to support the claim that tracesir diagnoses better.
  3. I think there. might be informaiton leakage in tas-k-recovery. So tracesir uses gold judge, which can contain evaluator feedback. In the case study, the report hands the agent the exact refund amount that the evaluator checks for.
  4. there is no component ablation. beyond the token reduction, the paper does not show what insightagent or report agent contribute. Table 6 is a self assessment rather than evidence by the way.
  5. novelty and posiitoning are a bit weak. tracebench is 150 failed traces from a single model with no successful or partially successful runs. The realted work cites several radiology report generation papers but missess most of the agent failure attribution literature.
Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.