- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates failure diagnosis in long-horizon agent execution, aiming to identify the root causes of task failures. The authors propose TIDE, which constructs a dependency graph to capture information dependencies between execution steps and employs a diagnostic agent for post-hoc analysis. The method is evaluated on two benchmarks, Who&When and AgentRX. Experimental results show that TIDE can improve failure root-cause localization.
Critical strengths
- The research problem is meaningful to the community, as diagnosing agent failures is an important step toward further improving agent systems.
- Using graph structures to analyze failure steps is intuitively well motivated, and the experimental results using the same backbone provide solid evidence of its effectiveness.
Critical weaknesses
- I find the contribution somewhat incremental. CHIEF and GraphTracer already use graph structures to model execution dependencies, yet the paper does not sufficiently clarify its methodological differences from these approaches. In particular, it remains unclear what additional capabilities the interactive trajectory environment provides over CHIEF’s hierarchical diagnosis procedure. Although the explanations and ablations support the effectiveness of some components, they do not adequately establish where TIDE’s gains over the closest prior methods come from.
- Although the authors conduct same-backbone comparisons against standard LLM-based judging baselines, comparisons with the closest methods, such as CHIEF and GraphTracer, use different backbone models. This makes it difficult to distinguish improvements due to the proposed method from those due to model capabilities. I recommend adding comparisons using the same backbone and, where feasible, comparable inference budgets to establish the independent contribution of the proposed mechanism.
Questions for authors
see the weaknesses above
Thank you for the detailed rebuttal and for running the additional experiments. The new backbone-controlled and rubric-controlled results against ECHO effectively address my concern that TIDE's gains over prior methods on Who&When were solely driven by stronger backbones or prompt engineering. However, several core concerns remain unresolved:
Degradation on Very Long trajectories. While the abstract highlights up to 55% reduction on long-horizon trajectories, Figure 3 shows this is the overall average. On "Very Long" traces without ground truth, TIDE is even 9% worse than the baseline, and Figure 4(c) shows 0 pp dependency gain on "Long" traces.
Small sample size and uncontrolled benchmark. Who&When Hand-Crafted has only 58 traces, yet no larger trajectory benchmark was added. AgentRX comparisons remain uncontrolled against GPT-5 baselines.
Unvalidated dependency graph as metric. Dependency edge precision and recall remain unmeasured on any annotated subset. The dependency-distance tolerance in Figure 4 is computed solely over TIDE's self-generated graph without evaluating baselines under the same graph.
While the new controlled baselines strengthen the submission, these remaining gaps keep the paper slightly below the ICLR acceptance bar, so I maintain my score of 2.