acceptodds
Under review as a conference paper at ICLR 2027

Interactive Trajectory Environments for Long-Horizon Agent Diagnosis

Abstract

Diagnosing agentic systems requires identifying where an execution went wrong, which becomes increasingly challenging for long-horizon executions as the diagnostic space grows and relevant evidence becomes dispersed across distant but dependent steps. Existing approaches primarily focus on designing the diagnosis agent or procedure, while treating the execution trajectory as passive context. We instead formulate diagnosis as agent–environment interaction, jointly designing the diagnosis agent and the environment through which it investigates an execution. We introduce TIDE (Trajectory-to-Interactive Diagnosis Environment), which transforms a failed execution trajectory into an interactive environment for diagnosis. Specifically, we construct a dependency graph over execution steps and hierarchically organize it into execution processes; together, these structures define an environment in which the agent can navigate across levels of abstraction and inspect dependency-related evidence. An Agent-as-a-Judge adaptively interacts with this environment to explore the diagnostic space, acquire relevant evidence, and localize the erroneous step. Experiments on two benchmarks show that TIDE consistently improves error localization across agentic systems and judge models, reducing mean localization distance by up to 55% on long-horizon trajectories. Further analyses show that dependency-aware exploration is particularly beneficial when diagnostically related steps are distant in execution order.

Primary area:
interpretability and explainable AI
Keywords:
LLM Agent, Agentic Diagnosis, Evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ueoiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates failure diagnosis in long-horizon agent execution, aiming to identify the root causes of task failures. The authors propose TIDE, which constructs a dependency graph to capture information dependencies between execution steps and employs a diagnostic agent for post-hoc analysis. The method is evaluated on two benchmarks, Who&When and AgentRX. Experimental results show that TIDE can improve failure root-cause localization.

Critical strengths
  • The research problem is meaningful to the community, as diagnosing agent failures is an important step toward further improving agent systems.
  • Using graph structures to analyze failure steps is intuitively well motivated, and the experimental results using the same backbone provide solid evidence of its effectiveness.
Critical weaknesses
  • I find the contribution somewhat incremental. CHIEF and GraphTracer already use graph structures to model execution dependencies, yet the paper does not sufficiently clarify its methodological differences from these approaches. In particular, it remains unclear what additional capabilities the interactive trajectory environment provides over CHIEF’s hierarchical diagnosis procedure. Although the explanations and ablations support the effectiveness of some components, they do not adequately establish where TIDE’s gains over the closest prior methods come from.
  • Although the authors conduct same-backbone comparisons against standard LLM-based judging baselines, comparisons with the closest methods, such as CHIEF and GraphTracer, use different backbone models. This makes it difficult to distinguish improvements due to the proposed method from those due to model capabilities. I recommend adding comparisons using the same backbone and, where feasible, comparable inference budgets to establish the independent contribution of the proposed mechanism.
Questions for authors

see the weaknesses above

Reviewer_AJruOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper treats failure attribution in agentic systems as agent–environment interaction. It introduces TIDE, which turns a failed trajectory into an environment with two parts: an LLM-built dependency graph over OTAR step nodes and a hierarchy tree of self-contained causal processes. An Agent-as-a-Judge navigates this environment with 3 actions, goto, check, and finish, to find the decisive error step. On Who&When (58 trajectories), TIDE with GPT-5.6-Luna reaches 37.93 step accuracy, compared with 29.31 for CHIEF. On AgentRX it lowers average step distance substantially, though exact accuracy is mixed. Ablations remove the dependency graph and the agentic interaction.

Critical strengths
  1. Shifting trajectory diagnosis from passive ingestion to an interactive environment solves a fundamental architectural challenge in agent evaluation. Long-horizon traces easily saturate the evaluator's context window with non-critical execution data. By coupling a hierarchical execution tree with a dependency DAG, TIDE provides a principled mechanism to restrict evidence exposure strictly to causal chains of interest.

  2. Even when exact step identification fails, TIDE consistently narrows down the search space close to the actual root cause. It beats same-backbone LLM-as-a-Judge baselines on Who&When, removing the graph causes a large drop, and AgentRX localization distance falls sharply.

Critical weaknesses
  1. Lack of backbone parity on key baselines: In Table 1, TIDE is run on cutting-edge models (GPT-5.6-Luna and DeepSeek-V4-Flash), but CHIEF, ECHO, and the AgentRX baselines are copied from prior papers that used older or different backbones. It is impossible to disentangle whether TIDE's gains stem from its interactive formulation or superior base-model reasoning.

  2. Prompt engineering confounder: Only TIDE’s agent receives the engineered rubric in prompt, which mirrors the benchmark’s labeling definition. The simple LLM-as-a-judge baselines used default prompts without this rubric, and the paper does not clarify whether the "w/o Agent" ablations were equipped with it.

  3. Unvalidated graph quality: The dependency graph is constructed entirely by an LLM without any precision/recall evaluation against ground-truth causal links.

  4. Sample size and marginal gains: Who&When Hand-Crafted contains only 58 trajectories. In the ablation study, with ground truth provided, TIDE achieves 34.5% accuracy while the non-agentic static graph baseline achieves 32.8%. Agentic interaction as a core contribution amounts to a difference of exactly one trajectory. Also, there is no error bars, multiple seeds, or statistical significance tests reported.

  5. Long-horizon claim contradicted: The headline 55% is the overall figure and the gain shrinks to −22% on very long trajectories with GT and reverses to +9% without GT. The empirical advantage degrades precisely where it is claimed to be strongest.

  6. Absence of computational cost: Building the environment requires pairwise LLM evaluations across preceding steps to construct the DAG, followed by recursive LLM summarization for every abstract process node, and finally multi-turn search. The paper provides zero metrics on token consumption, API calls, latency, or monetary cost, leaving it unclear whether marginal accuracy gains justify orders-of-magnitude higher computational costs.

Questions for authors
  1. Controlled Baseline Parity: Can you provide controlled experimental results evaluating CHIEF and ECHO on Who&When using the exact same model backbones (GPT-5.6-Luna and DeepSeek-V4-Flash) and providing them with the identical 7-rule decisive-error rubric?

  2. Ablation Prompt Parity: Did the "w/o Agent (Graph)" and "w/o Agent (Tree)" ablation baselines in Figure 3 receive the identical prompt rubric, intervention test instructions, and negative constraint rules as the interactive agent?

  3. Dependency Graph Precision and Recall: Have you evaluated the accuracy of LLM-generated dependency edges against a human-verified ground-truth causal graph? How sensitive is the diagnosis policy to false-positive or false-negative edges?

  4. Variance and Significance: Given that Who&When has only 58 trajectories, what are the mean, standard deviation, and paired significance values (p-values) across at least three independent runs of environment construction and diagnosis?

  5. Long-Horizon Discrepancy: How do you reconcile the claim of superior long-horizon localization with the results in Figure 3 (where TIDE is 9% worse than Step-by-Step on "Very Long" traces without ground truth) and Figure 4(c) (showing 0 pp gain on "Long" traces)?

  6. Computational and Token Cost: What is the average token usage (input/output), number of API calls, and wall-clock latency per trajectory for TIDE across its three stages (graph construction, tree summarization, interactive navigation) compared to CHIEF, Step-by-Step, and All-at-Once?

1 comment

Thank you for the detailed rebuttal and for running the additional experiments. The new backbone-controlled and rubric-controlled results against ECHO effectively address my concern that TIDE's gains over prior methods on Who&When were solely driven by stronger backbones or prompt engineering. However, several core concerns remain unresolved:

  1. Degradation on Very Long trajectories. While the abstract highlights up to 55% reduction on long-horizon trajectories, Figure 3 shows this is the overall average. On "Very Long" traces without ground truth, TIDE is even 9% worse than the baseline, and Figure 4(c) shows 0 pp dependency gain on "Long" traces.

  2. Small sample size and uncontrolled benchmark. Who&When Hand-Crafted has only 58 traces, yet no larger trajectory benchmark was added. AgentRX comparisons remain uncontrolled against GPT-5 baselines.

  3. Unvalidated dependency graph as metric. Dependency edge precision and recall remain unmeasured on any annotated subset. The dependency-distance tolerance in Figure 4 is computed solely over TIDE's self-generated graph without evaluating baselines under the same graph.

While the new controlled baselines strengthen the submission, these remaining gaps keep the paper slightly below the ICLR acceptance bar, so I maintain my score of 2.

Reviewer_2gUmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TIDE, which turns execution trajectories into interactive environments for agent failure diagnosis. It uses a dependency graph and a hierarchical representation to help a diagnosis agent navigate a trajectory and locate the source of failure. The evaluation uses Who&When and AgentRX with GPT-5.6-Luna and DeepSeek-V4-Flash, reporting diagnosis accuracy and localization distance. The paper also includes component ablations and an analysis across trajectory-length groups.

Critical strengths

The motivation is reasonable and practically relevant. Looking only at the final outcome provides limited information about where an agent went wrong, particularly when relevant evidence is spread across an execution trajectory. Treating diagnosis as an interactive process is a sensible approach. The dependency graph and hierarchical representation provide a concrete way to explore the trajectory rather than treating it only as a flat sequence of text.

Critical weaknesses

The scope of the long-horizon evaluation is not sufficiently clear to me. Figure 3 provides a useful analysis across trajectory-length groups, but I could not determine the actual step-count ranges and sample counts underlying these groups. The length distribution of each benchmark or subset would help establish how long the evaluated trajectories are and how much evidence supports the conclusions at the longer end of the distribution.

The model coverage is limited to GPT-5.6-Luna and DeepSeek-V4-Flash. Results from these two models are useful, but evaluating additional models would provide stronger evidence that the benefits generalize beyond the particular models used here.

These concerns currently lead me to lean toward weak rejection, although I would be open to revisiting my recommendation if the authors provide sufficient clarification and supporting evidence.

Questions for authors

Could the authors provide the trajectory-length distribution for Who&When and each evaluated AgentRX subset, including the definition of a step? Please specify the thresholds and sample counts for the Short, Medium, Long, and Very Long groups in Figure 3. If these statistics are already reported, please point me to the relevant section.

Are results with additional models available? Such evidence would help assess whether the observed benefits extend beyond the two evaluated models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.