acceptodds
Withdrawn from ICLR 2027

Cirdan: Structured Deep Investigation for Long-Horizon Code Diagnosis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Diagnosing distributed system failures requires long-horizon investigation: LLM agents must trace causality across components, connect observed symptoms to responsible code, and distinguish root causes from downstream symptoms. Through an empirical study, we observe that agents often settle on plausible explanations but leave key causal steps unexplored. We introduce Cirdan, a framework for structured investigation of such failures. Cirdan performs broad repository survey, triages competing hypotheses, and explores promising explanations in a tree that supports refinement and backtracking. Its context management retains detailed evidence along each branch and summaries of alternatives, while preserving shared prefixes for prompt-cache reuse. Tool-call context injection brings in structural and architectural information to help the agent look beyond the currently inspected subsystem. We evaluate Cirdan against four agent frameworks across five models on 34 real-world bugs from 11 distributed systems. Cirdan improves diagnosis accuracy with four of the five models. It raises block coverage by 6.3–22.7 percentage points with GPT-5.2-Codex and by 7.4 and 3.6 points with Opus 5 and Gemini-3.8-Flash. With GPT-6-Astra, it maintains coverage comparable to mini-SWE-agent's while reporting half as many code blocks, raising precision from 42.2% to 61.4%. Our results show that keeping competing explanations and their evidence available, while leaving investigation decisions to the model, improves code diagnosis with most models, and that the harness's budgets and guidance must match how each model reasons.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, long-horizon reasoning, multi-hypothesis search, context management, software debugging, root cause analysis, distributed systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xyaROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Cirdan, a framework for diagnosing failures in distributed systems. It organizes competing explanations and their supporting evidence in a tree structure and performs diagnosis through three stages. Experiments show promising results across multiple models and in comparisons with several agent frameworks.

Critical strengths
  1. This paper is well motivated: investigating upstream root causes helps prevent models from progressing further down incorrect investigative paths.
  2. This paper provides extensive experimental evaluation and reports both the method’s applicability limits and the settings in which it underperforms.
Critical weaknesses
  1. One claimed innovation of this paper is deep-branch backtracking. However, all ten structured trajectories in Appendix H already contain the main failure mechanism during the initial survey, with no subsequent deep branch switches, revisits, or second-level expansions. This does not demonstrate the mechanism’s effectiveness.

  2. In my opinion, one innovation of this paper appears to be tree-structured exploration of competing explanations and backtracking. However, similar mechanisms already exist in methods such as SWE-Search, leaving the incremental novelty unclear.

Questions for authors
  1. Can the authors provide direct evidence that deep branching and backtracking improve diagnosis?
  2. What specific advances distinguish Cirdan from SWE-Search, rather than merely a difference in application domains, because SWE-Search is mainly used to output solutions for software problem repair scenarios, which in my view covers the fault localization scenario in this paper.
  3. Which design choices specifically target distributed-system diagnosis, and can Cirdan generalize to other software diagnosis settings?
Reviewer_VBpFOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Cirdan for diagnosing distributed-system failures using LLM agents. Cirdan combines phased hypothesis exploration, branch context management, and context injection for deep investigation. The authors assess Cirdan on 34 bugs from 11 distributed systems using five models. The results show improvements in localization coverage or precision with four models.

Critical strengths
  • The paper addresses an important problem.
  • The experiments include several models and agent frameworks
Critical weaknesses
  • The evaluation dataset is too small to support the paper’s claims. With only 34 bugs, the evaluation is simply too limited. Given that the configurations were developed based on benchmark casest, the result is not sufficient to show generalization.

  • Higher block coverage does not necessarily meanbetter diagnosis. A reported block can overlap with an annotated root-cause block without correctly explaining its role in the failure. It is unclear whether the reported localization gains can really reflect accurate causal explanations.

  • The paper uses some weird terms such as “broad repository survey” and “triage”. These terms are widely used elsewhere for different meaning. For example, this is not really a survey, and triage in software engineering often refer to bug triaging. Using terms that are more common in the research literature would make the paper more readable.

  • Some issue reports already provide localization hints such as stack traces. Many prior SE research has shown that using stack traces along can achieve good localization results. Although the paper separates results by stack-trace availability, a simple baseline that follows stack-trace locations and inspects related code would help establish how much the proposed investigation adds beyond these hints.

  • Most of the coverage gains are on bugs with a single root-cause block. The approach provides little improvement on multi-block or cross-file bugs, although these complex failures are a main motivation for the paper.

Questions for authors
  1. Why were only 34 of the 120 benchmark bugs included? How were they selected? Why didn't the authors evaluate on larger benchmarks?
  2. How does the approach perform on bugs that were not used to develop or tune its configurations?
Reviewer_7fwSOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CIRDAN, a structured diagnosis framework for identifying bugs in distributed systems using LLM agents. The method organizes debugging reasoning as a hierarchical hypothesis tree: agents iteratively propose, score, expand, and select hypotheses to investigate, following a multi-phase workflow including architecture survey, triage, and deep investigation. The framework incorporates explicit scoring heuristics for leaf node hypotheses and structured prompting guidance to enforce rigorous causal debugging and self-critique. It also specifies full reproducibility requirements for model configurations, inference settings, and evaluation protocols to support consistent comparison of results.

Critical strengths
  1. Problem relevance and practical framing. The work addresses a high-impact problem, and the proposed approach adapts general LLM reasoning frameworks to the specific constraints of distributed system diagnosis, introducing domain-specific adjustments that fill a gap in existing LLM agent research for software engineering. This practical, domain-tailored design makes the work useful for both researchers and practitioners working on AI-powered software debugging.

  2. High reproducibility and clear presentation of implementation details. The paper provides unusually complete documentation of implementation details, enabling other researchers to directly replicate, modify, and evaluate the approach. It is a significant benefit for the community.

Critical weaknesses
  1. Limited novelty in core methodology. The core hierarchical reasoning structure of the approach is adapted directly from existing tree-of-thought frameworks for LLM reasoning. While the paper adds domain-specific heuristics for distributed system debugging, the fundamental innovation is incremental. The technical contribution is therefore more incremental than foundational.

  2. Incomplete empirical validation. The paper only references an anonymous benchmark (DistDebug-Bench) for evaluation, and no quantitative empirical results comparing CIRDAN to baseline bug diagnosis methods. It lacks clear performance metrics (e.g., diagnosis accuracy, mean time to identify root cause) demonstrating that CIRDAN outperforms existing approaches.

  3. Limited dataset size. The evaluation workload only covers a limited set of public bugs from distributed systems, which results in a relatively small dataset for testing the proposed method.

Questions for authors

The paper uses 34 shared bugs from 11 distributed systems for main evaluation, could you elaborate on three points: (1) what is the specific source of these 34 bugs (e.g., which types of production distributed systems are they collected from), (2) what explicit criteria did you use to select these bugs into the evaluation set, and (3) how do you address the concern that the results on this small, curated dataset do not generalize to uncurated, real-world distributed system debugging scenarios encountered by developers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.