- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes DEAN for critical failure localization in long-horizon agent trajectories: identifying the earliest unrepaired step responsible for a failed outcome. DEAN first applies model-free, length-dependent, and role-aware trajectory compression, then builds an LLM-generated hierarchical Global Trajectory Prior. A driver selects spans for inspection through a bounded zoom-in loop, while separate analyst calls classify candidate causes and symptoms. Evaluation covers seven suites, three LLM backbones, specialized localization baselines, token costs, and downstream uses of localization feedback.
Critical strengths
- The global-to-local search addresses a practical debugging problem, and evaluation across seven suites and three backbones provides evidence that the approach is not confined to one domain or model.
- Model-free compression, supplied prompts, pseudocode, and per-suite token counts make the approach easier to understand and reproduce. Connecting localization to downstream improvement also helps assess whether the output has practical value beyond an attribution score.
Critical weaknesses
-
Baseline fidelity and resource comparability remain unclear. Under GLM-5.2, AgentDebug trails Base in accuracy (19.79 vs. 34.52); its lower input usage on SWE-Bench Pro (17.86k vs. 50.70k) also warrants clarification of context handling, termination, and accounting. DEAN uses substantially more input tokens than Base without compute-matched reasoning or repeated-sampling controls, weakening the effectiveness comparison.
-
Mechanism attribution and scaling evidence are insufficient. Without a separate Global Trajectory Prior ablation, gains from compression and search remain confounded with extra LLM analysis. The cost-exponent fit mixes benchmarks with different domains and tokens per step; a positive limiting retention ratio and bounded search turns do not support asymptotic sublinearity. Missing uncertainty estimates for three-run results further limit confidence in small aggregate gains and the claimed accuracy/cost tradeoff.
Questions for authors
- Were the specialized baselines run from original implementations or reimplemented? Please specify context limits, truncation, termination and parse-error handling, and explain the AgentDebug token counts relative to Base in Table 5.
- How were compression hyperparameters and driver/analyst prompts selected? Was there a development split separate from the reported evaluation suites, and did baseline prompts receive comparable tuning? 3.What exactly changes in the compression ablation, and are separate Global Trajectory Prior ablations, compute-matched controls, or within-benchmark absolute cost analyses already available? Please summarize existing evidence if so.