acceptodds
Under review as a conference paper at ICLR 2027

"Thinking with Trajectories" for Critical Failure Localization in Long-Horizon Agents

Abstract

Large language model (LLM) agents tackle complex tasks through sequences of interdependent steps, where an early critical failure can derail subsequent decisions and ultimately lead to task failure. As agents advance toward increasingly long-horizon tasks, detecting such failures becomes more challenging: global analysis requires reasoning over lengthy trajectories with scattered evidence, while step-by-step inspection can be overwhelmed by local errors and incurs rapidly growing computational overhead. Effective auditing therefore requires allocating reasoning budget to informative trajectory segments while preserving a global view. We address this challenge by modeling critical failure localization as an agentic search problem. Specifically, we propose Dean, a framework that enables a “*Thinking with Trajectories*” paradigm: it first scans a compressed trajectory overview to identify promising regions, and then progressively zooms into selected segments for fine-grained analysis. This global-to-local reasoning loop iteratively narrows the search space until the critical failure is grounded in localized trajectory evidence. Experiments across diverse benchmarks demonstrate that Dean achieves state-of-the-art performance, with its advantage widening as trajectories grow longer while exhibiting substantially more favorable token scaling. We further show that localized critical failure evidence can be readily integrated into existing agent pipelines to support self-improvement, highlighting its potential as a source of effective feedback for long-horizon agents. These results demonstrate the effectiveness and scalability of the “*Thinking with Trajectories*” paradigm for long-horizon critical failure localization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-Horizon Agents, Critical Failure Localization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tAdLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DEAN for critical failure localization in long-horizon agent trajectories: identifying the earliest unrepaired step responsible for a failed outcome. DEAN first applies model-free, length-dependent, and role-aware trajectory compression, then builds an LLM-generated hierarchical Global Trajectory Prior. A driver selects spans for inspection through a bounded zoom-in loop, while separate analyst calls classify candidate causes and symptoms. Evaluation covers seven suites, three LLM backbones, specialized localization baselines, token costs, and downstream uses of localization feedback.

Critical strengths
  1. The global-to-local search addresses a practical debugging problem, and evaluation across seven suites and three backbones provides evidence that the approach is not confined to one domain or model.
  2. Model-free compression, supplied prompts, pseudocode, and per-suite token counts make the approach easier to understand and reproduce. Connecting localization to downstream improvement also helps assess whether the output has practical value beyond an attribution score.
Critical weaknesses
  1. Baseline fidelity and resource comparability remain unclear. Under GLM-5.2, AgentDebug trails Base in accuracy (19.79 vs. 34.52); its lower input usage on SWE-Bench Pro (17.86k vs. 50.70k) also warrants clarification of context handling, termination, and accounting. DEAN uses substantially more input tokens than Base without compute-matched reasoning or repeated-sampling controls, weakening the effectiveness comparison.

  2. Mechanism attribution and scaling evidence are insufficient. Without a separate Global Trajectory Prior ablation, gains from compression and search remain confounded with extra LLM analysis. The cost-exponent fit mixes benchmarks with different domains and tokens per step; a positive limiting retention ratio and bounded search turns do not support asymptotic sublinearity. Missing uncertainty estimates for three-run results further limit confidence in small aggregate gains and the claimed accuracy/cost tradeoff.

Questions for authors
  1. Were the specialized baselines run from original implementations or reimplemented? Please specify context limits, truncation, termination and parse-error handling, and explain the AgentDebug token counts relative to Base in Table 5.
  2. How were compression hyperparameters and driver/analyst prompts selected? Was there a development split separate from the reported evaluation suites, and did baseline prompts receive comparable tuning? 3.What exactly changes in the compression ablation, and are separate Global Trajectory Prior ablations, compute-matched controls, or within-benchmark absolute cost analyses already available? Please summarize existing evidence if so.
Reviewer_4dgvOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the task of error localization in long trajectories by designing an agent framework, DEAN, which implements a global-to-local error localization strategy through simple agent mechanisms. Experiments on multiple benchmarks demonstrate that the proposed method can effectively improve error localization accuracy.

Critical strengths
  1. The idea of the paper is very clear. It uses a top-down agent strategy to perform error localization, making the method easy to understand and the paper easy to read.
  2. The task addressed by the paper is important. If errors can be identified in time, better improvement strategies can be developed, which would be helpful for mainstream approaches such as tree search and RL.
  3. The experiments are relatively comprehensive, covering three benchmarks, and the results are also good.
Critical weaknesses
  1. Insufficient contribution. The task is relatively simple and text-only, and using an agent to solve it is a fairly obvious approach without particularly surprising ideas. This submission does not involve theoretical derivation, and the design of the agent modules is also insufficient. The main contribution is entirely concentrated on the agent workflow. However, in recent years, the main research focus in the agent field has been on general-purpose agents. Therefore, agent-based work targeting a specific task (especially a simple, verifiable, single-modality task) naturally faces the problem of insufficient contribution. I also noticed that several major related works cited in the paper are all arXiv papers, which may indicate that this type of simple prompt-engineering work is difficult to get accepted at major conferences. In addition, error localization is often used as an internal module in some agents, which further suggests that studying only this single module makes it difficult to demonstrate a sufficient contribution.
  2. Some methodological details lack rigor. For example, Eq. 3 uses a logarithmic transformation to address the problem of low information density in long sentences. However, long sentences do not necessarily have low information density; their length may instead result from the difficulty of the corresponding step.
  3. Insufficient baselines. The paper lacks comparisons with directly using general-purpose agent frameworks to identify error locations in trajectories. It also does not analyze the advantages of DEAN over such approaches.
Questions for authors
  1. What advantages does DEAN have compared with general-purpose agents?
Reviewer_zyyfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies critical failure localization in long-horizon LLM agent trajectories, i.e., identifying the earliest unrepaired error step that causally leads to task failure. The authors propose DEAN, which reframes the task as a global-to-local agentic search: a model-free trajectory compression module (adaptive retention ratio, log-weighted step-wise budget allocation, and role-aware truncation) first builds a compact global overview, and an LLM then iteratively selects suspicious segments to zoom into, with each segment analyzed in an isolated segment-centric reasoning process whose distilled evidence is fed back into the main reasoning chain.

Critical strengths
  1. The problem is well motivated and practically important; the observation that failure-relevant evidence is unevenly distributed and that uniform step-level analysis is distracted by local noise is sound, making the global-to-local search reformulation natural.
  2. The evaluation is broad: seven benchmark settings, three backbones, five baselines, two-level ablations on components and compression strategies, hyperparameter sensitivity analysis, and Pareto/scaling analyses.
  3. The long-horizon advantage is substantial and convincing: baselines nearly collapse on the longest trajectories while DEAN remains stable, with the slowest empirical token-cost growth.
Critical weaknesses
  1. Limited methodological novelty. The core mechanism, a compressed overview plus zoom tool calls plus per-window analysis and aggregation, closely mirrors existing hierarchical summarization and agentic long-context search practices, and Who&When already explored search-based localization; the three compression sub-mechanisms are heuristic rules, so the contribution is mainly system integration and empirical evidence rather than new methods.
  2. Absolute accuracy is low (average exact Acc of only 25–38 in the best setting), and DEAN is not first in several columns (e.g., ALFWorld and GAIA under Qwen3-32B, where AgentDebug wins), so the “consistently outperforming” claim is stronger than the tables support.
  3. Fairness concerns: DEAN’s prompts contain extensive, carefully engineered search-discipline instructions, and it is unclear whether baselines received comparable prompt tuning; the effect of the fixed zoom budget K=8 is not analyzed.
  4. Ground-truth reliability of the task itself is not discussed, i.e., whether the “earliest unrepaired causal step” is annotated consistently across benchmarks; code is not released, so reproducibility rests on the appendix prompts
Questions for authors
  1. Were the baseline prompts tuned to a comparable degree as DEAN’s? Can you provide a comparison where baselines operate under the same token budget?
  2. How sensitive are results to the zoom budget K? How often does the search terminate early or exhaust its budget and fall back, and what is the impact on accuracy?
  3. Are the critical-step annotation criteria consistent across benchmarks? To what extent does label noise (particularly the large gap between Tol.@2 and exact Acc) bound the achievable accuracy on this task?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.