acceptodds
Under review as a conference paper at ICLR 2027

Making Evidence Count: Grounded Diagnosis for Software Agents

Abstract

Diagnosing a software failure requires more than locating relevant code. An agent must connect the failure mechanism to the observed symptoms, identify where and how a repair should occur, and ground these claims in acquired evidence. We make three contributions toward evaluating and supporting this reasoning. First, we introduce FailureSnapshotHub (FSH), an interactive benchmark of 60 real continuous-integration failures from 49 repositories. Agents investigate only failed-side logs, code, tests, configuration, and execution context, while expert-adjudicated references and corroborating developer repair diffs remain hidden. Evaluation separately assesses mechanism, causal direction, repair owner/location, repair direction, completeness, and evidence grounding. Second, we formulate evidence-centric context compilation and present EviCompile, a deterministic method that transforms tool observations into source-linked evidence memory through recoverable text coding, typed evidence construction, cross-observation consolidation, and cost-aware global ordering. EviCompile itself neither generates new observations nor invokes another language model; causal inference remains with the diagnostic agent. Third, we empirically characterize the effects of evidence representation in a sequential Tool Agent, an OpenRCA-style controller–executor agent, and fixed-evidence diagnosis. Case analyses show that agents can locate the relevant code or recognize an immediate mismatch yet infer the wrong mechanism or repair boundary. Compared with original observation histories, EviCompile improves mean semantic scores by 9.72 and 10.42 points in the two interactive settings and by 5.55 points when acquired evidence is held fixed. Local text coding underperforms EviCompile in all three settings; with evidence membership fixed, full ordering also outperforms chronological and relevance-only presentation. Blinded expert reassessment on 40 cases preserves positive mean gains. Together, these results show that evidence organization can improve causal and repair judgments without additional observations, while FSH exposes diagnostic errors that location-based evaluation cannot resolve.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agentic Diagnosis, Root Cause Analysis, Evidence Representation, Continuous Integration, Auditable AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4pVZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FailureSnapshotHub, a 60-case interactive software-failure diagnosis benchmark spanning 49 repositories, with expert-adjudicated references and six-dimensional semantic evaluation. It also proposes EviCompile, a deterministic pipeline that compresses recoverable text, constructs typed source-linked evidence, consolidates compatible observations, and orders the complete evidence memory using relevance, coverage, representativeness, and character cost. Experiments report improved diagnosis scores in two interactive agent settings and a fixed-evidence setting, with positive mean gains retained in a selected blinded human reassessment.

Critical strengths
  1. The benchmark distinguishes failure mechanism, causal direction, repair responsibility, and evidence grounding rather than treating localization as sufficient diagnosis. Expert reference construction and failed-side access restrictions are useful aspects of the evaluation design.

  2. The fixed-evidence comparison helps separate representation effects from differences in evidence acquisition. The reported diagnosis-score improvements and blinded human reassessment provide evidence that the complete pipeline is useful within the evaluated setting.

Critical weaknesses
  1. The paper does not sufficiently distill the practical need for evidence organization into a distinctive technical problem or methodological contribution. EviCompile largely combines recoverable compression, typed records, deduplication, provenance retention, and coverage-aware ordering. Exceptions, source locations, test outcomes, and path inventories adapt this general approach to software data, but the paper does not clearly establish what technical obstacle these domain characteristics introduce that existing evidence-organization methods cannot address, or what new mechanism resolves it. Applying a general idea to software diagnosis can be valuable, but the current formulation and evidence do not sufficiently establish that independent technical contribution.

  2. The comparisons and evaluation scope do not adequately resolve this novelty concern. Raw histories, local text coding, and internal ordering variants show the utility of the combined pipeline, but do not establish an advantage over strong conventional structured, provenance-preserving, deduplicated evidence memories with comparable ordering. Typing, sharing, and reconstruction are not fully separated, and the ordering ablation provides a relatively small part of the total gain without reported uncertainty in that table. The study also uses one diagnostic model on 60 selected cases. Human reassessment supports diagnosis quality on its selected subset, but the broader transfer and practical consequences of the improvement remain uncertain. Actual repair success is not evaluated; this limits the demonstrated downstream utility rather than constituting a requirement that every diagnosis paper perform repair.

Questions for authors
  1. What precisely is the technical obstacle, beyond heterogeneous formatting and redundant observations, that distinguishes this problem from general evidence organization? Which component introduces a new solution to that obstacle rather than a software-specific data-schema adaptation?

  2. How does EviCompile compare with a strong structured and deduplicated evidence-memory baseline using the same source attribution and comparable relevance/diversity ordering? Component-level comparisons with paired uncertainty would help establish what independently contributes beyond conventional organization.

2 comments

Dear Reviewer,

Response 1 of 2. Methodological contribution, strong baselines, and experimental attribution

Thank you for your detailed review and for recognizing the value of the fixed-evidence experiment. Your questions prompted us to examine what EviCompile adds to conventional evidence organization and which gains can be attributed specifically to ordering.

1. What technical problem does EviCompile address?

Evidence that determines where a fix belongs may have little textual similarity to the visible failure symptom. An exception and stack trace can show where a failure surfaced while leaving open whether to change the test, the caller, the callee, or the configuration. The evidence that resolves this question may come from the implementation on the other side of an interface or from an upstream producer.

EviCompile incorporates diagnostic roles and candidate repair responsibilities into its evidence coverage objective. Its ordering rule considers relevance, complementary coverage, representativeness, and character cost. An item can receive priority because it helps distinguish between candidate repair locations, even when its wording differs from the failure description. Evidence retention is independent of presentation order, merged records preserve source attribution, and the diagnostic agent makes the final causal judgment.

2. How does EviCompile compare with conventional evidence organization?

To address your request for a structured evidence memory with deduplication and provenance, we implemented a Conventional Evidence Memory baseline. In the fixed-evidence setting, it stores each tool response as a record containing the tool name, source path, line numbers, content, and source list. It merges duplicates using TF-IDF cosine similarity and retains all sources. It scores relevance to the initial failure description using BM25, then uses MMR to balance relevance and textual diversity, with an optional length cost. It uses none of EviCompile’s evidence types, role hints, or specialized priors.

The table lists the configurations for which the paired comparisons showed a statistically supported gain. Differences are EviCompile minus baseline. Wins, ties, and losses are counted from EviCompile’s perspective. θ is the TF-IDF deduplication threshold, and α controls the balance between relevance and textual diversity in MMR.

Baseline configurationMean difference95% CIWins/ties/lossesWilcoxon p
θ=0.8, α=0.3, length cost on+1.99[0.33,4.45]21/30/90.0467
θ=0.8, α=0.7, length cost on+2.68[0.85,5.43]23/24/130.0228
θ=0.8, α=0.5, length cost off+2.82[0.85,5.85]21/26/130.0367

We also compared EviCompile with MMR, evidence truncation, and LLMLingua-2 using the same paired comparison.

BaselineMean difference95% CIWins/ties/lossesWilcoxon p
MMR α=0.3+2.41[0.77,3.94]22/30/80.0326
MMR α=0.5+5.86[1.20,10.95]25/21/140.0434
Truncation k=10+5.93[1.05,10.17]30/20/100.0081
Truncation k=20+2.47[0.34,6.78]28/25/70.0413
Truncation k=30+3.04[0.71,7.20]20/30/100.0302
LLMLingua-2+5.65[1.06,11.16]25/25/100.0232

3. Which gains can be attributed to Ordering?

The baseline comparisons evaluate the method as a whole. To isolate Ordering, we held the typed and shared records fixed, so every strategy presented the same evidence. Full Ordering improved the mean score by +2.78 over chronological ordering (95% CI [1.20,4.36], Holm-adjusted p=0.0024) and by +2.08 over simple relevance ordering (95% CI [0.65,3.55], Holm-adjusted p=0.012).

In the six-dimensional analysis, scores for identifying where the fix belongs and determining the direction of the repair action improved by +0.1333 and +0.1667, respectively. Both gains remained significant after Holm correction across 18 comparisons. Typing cannot be removed in isolation because both Sharing and Coverage depend on typed records.

Thank you again for asking us to examine attribution more closely. Your suggestion led us to test a baseline that preserves provenance and to quantify what changes when only the presentation order varies.

Dear Reviewer,

Response 2 of 2. Benchmark contribution, multi-model results, and scope of the evaluation

Thank you for recognizing FSH’s six-dimensional evaluation and the work involved in constructing expert reference diagnoses. We appreciate your questions about how broadly the findings apply and what they establish about repair.

1. What does FSH evaluate, and how were its cases selected?

FSH contains 60 real CI failures from 49 repositories. Through expert adjudication and access restricted to failure-side evidence, FSH evaluates six dimensions: failure mechanism, causal direction, repair responsibility, repair action, completeness, and evidential support. These dimensions distinguish diagnostic errors that traditional localization metrics can miss. For example, a diagnosis may identify where a failure surfaced yet give the wrong repair direction.

Experts selected cases for which the failure-side evidence was sufficient to infer the mechanism and determine where the fix belonged without inspecting the fix diff. They excluded cases with multiple independently viable repair locations. This selection makes it possible to assess a diagnosis using the evidence available to the agent. FSH was not designed as a representative sample of all CI failures.

2. How consistent are the gains across models and workflows?

To address the limitation of evaluating a single diagnostic model, we tested three models in three settings. Each entry is EviCompile’s mean improvement over Raw.

ModelTool AgentOpenRCA-styleFixed-evidence
Qwen 3.5 Flash+2.78+4.03+1.81
DeepSeek V4 Flash+9.72+10.42+5.55
Claude Opus 4.8+14.24+18.75+10.00

All nine combinations showed positive gains. In table order, the models’ mean Raw scores across settings were 53.66, 60.19, and 79.17, with corresponding mean gains of +2.87, +8.56, and +14.33. The models with stronger Raw diagnostic performance also gained more from EviCompile, suggesting that evidence organization and model reasoning ability may be complementary.

For all three models, the gains followed the same order: OpenRCA-style > Tool Agent > Fixed-evidence. This pattern suggests that the benefit of evidence organization may also depend on the agent workflow.

3. Do the gains extend to an independent RCA benchmark?

We evaluated EviCompile on the original OpenRCA Benchmark using DeepSeek V4. Strict accuracy rose from 23.53% to 31.57%, partial accuracy rose from 36.63% to 52.71%, and total token consumption fell by 35.3%. Accuracy changes in the table are expressed in percentage points (pp).

MetricBase+EviCompileChange
Strict accuracy23.53%31.57%+8.04 pp
Partial accuracy36.63%52.71%+16.08 pp
Mean steps11.009.04-17.8%
Input tokens/query769,812473,711-38.5%
Output tokens/query85,97279,940-7.0%
Total tokens/query855,784553,651-35.3%
Model calls/query38.231.9-16.5%
Mean runtime907.1s847.6s-6.6%
Diagnoses exceeding 10 minutes4636-10
Sandbox aborts65-1

These results extend the findings beyond FSH’s semantic scores. On an independent benchmark, EviCompile improved both strict and partial diagnostic accuracy while reducing token use, model calls, and mean runtime.

4. What do these results establish about repair success?

Our evaluation concerns diagnosis before patch execution, specifically whether the diagnosis identifies where the fix belongs and what change would address the failure. End-to-end repair success also depends on the ability to generate a working patch and requires a separate evaluation.

Thank you again for your thoughtful review. Your comments helped us clarify why FSH’s cases were selected, how the findings extend across models and benchmarks, and where the evidence about diagnosis ends.

Reviewer_bvQKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors tackle the problem of root cause analysis (RCA) in software systems. Specifically, large amounts of code and telemetry evidence can overwhelm the context of a language model. The main contributions are threefold: an interactive benchmark (FailureSnapshotHub) of 60 real continuous-integration failures from 49 repositories, a domain-specific context re-structuring algorithm called EviCompile, and an empirical study of EviCompile's impact on RCA. The authors demonstrate that EviCompile improves mean semantic scores by 9.72 and 10.42 points in the Tool Agent and OpenRCA-style settings and by 5.55 points in the fixed-evidence setting.

Critical strengths
  1. Applying cost-aware ordering to model context re-structuring for root cause analysis is novel.
  2. The authors considered multiple settings: Tool Agent, OpenRCA-style, Fixed-evidence
  3. I appreciated the inclusion of correlation analysis between LLM judge and human expert scores.
Critical weaknesses
  1. It's unclear how some of the terms in the cost model are obtained. For example, the relevance terms in Equation 3 rely on b_e, a_role, a_type, but the authors did not describe how these values were obtained. Or maybe I just missed it.
  2. The authors use a semantic score as the headline metric. It's unclear how this metric relates to downstream metrics, like accuracy on OpenRCA.
Questions for authors
  1. How were the values in Table 7 chosen? Does this procedure need to be repeated for every new dataset?
  2. Given that the authors consider an OpenRCA-style setting, do the authors have results evaluating the OpenRCA agent paired with EviCompile on the OpenRCA benchmark? This would further strengthen the argument that EviCompile improves RCA accuracy.
1 comment

Dear Reviewer,

Thank you for recognizing the value of cost-aware evidence ordering for RCA and for your positive assessment of our evaluation across diagnostic settings and our analysis of correlations with human judgments. In response to your questions, we have clarified how we chose the ordering parameters, examined their reuse across datasets, and evaluated EviCompile on the original OpenRCA Benchmark to assess downstream RCA accuracy.

1. How were the cost model parameters chosen? Must they be retuned for each dataset?

We agree that the original manuscript gave too little detail about the rationale for the parameters in Equation (3). We set beb_e, arolea_{\mathrm{role}}, and atypea_{\mathrm{type}} manually before formal evaluation, based on the expected diagnostic value of different evidence types and roles. We used neither benchmark results nor reference diagnoses to tune these parameters.

Evidence importance and type priors. We assign base importance on six equally spaced levels, with be=0.6+0.2rb_e = 0.6 + 0.2r for integer rr from 0 to 5. Exceptions and failing tests receive higher priority because they directly reveal failure symptoms or violations of expected behavior. Source code snippets help explain the failure mechanism. The diagnostic value of environment information and ordinary logs usually depends on the surrounding evidence. Base importance also contributes to the weight assigned to each item in the evidence representation, while the type prior adjusts its initial reading priority.

Role priors. We assign priority according to the role each item plays in diagnosis. Clues about fault creation and production receive higher priority, followed by clues about propagation and candidate repair locations. Symptoms provide a starting point for investigation. General context receives lower priority, and suspected noise is downweighted but retained.

Reuse across models, agents, and benchmarks. We used the same fixed parameters in every experiment. The table reports the mean gain of EviCompile over Raw on FSH for each model and diagnostic setting.

ModelTool AgentOpenRCA-styleFixed-evidence
Qwen 3.5 Flash+2.78+4.03+1.81
DeepSeek V4 Flash+9.72+10.42+5.55
Claude Opus 4.8+14.24+18.75+10.00

All nine combinations showed gains, and every 95% paired bootstrap confidence interval lay entirely above zero. The same parameters also yielded accuracy gains on the original OpenRCA Benchmark with DeepSeek V4. Across the models, agent architectures, and benchmarks evaluated, these gains required no new search for ordering weights. For datasets with different observation formats, the adaptation concerns how evidence is extracted and mapped to types. The ordering parameters can be reused.

2. How does the Semantic Score relate to downstream RCA Accuracy?

The Semantic Score assesses six dimensions of a diagnosis: failure mechanism, causal direction, repair target location, repair action direction, completeness, and evidential support. RCA Accuracy measures whether a diagnosis meets the benchmark’s criteria for root cause identification. OpenRCA, for example, evaluates the identified component, time, and cause.

Repair target location is closest to conventional localization metrics. It asks which component actually needs modification or within which responsibility boundary the fix belongs. Semantically equivalent fixes within that boundary are accepted. The other dimensions assess how well the diagnosis explains the mechanism, traces causality, identifies a repair direction, and supports its conclusions with evidence. The two metrics are complementary: the Semantic Score describes the quality of the diagnosis in detail, while RCA Accuracy measures correctness under the benchmark’s task definition.

To assess downstream accuracy directly, we evaluated the OpenRCA Agent with and without EviCompile on the original OpenRCA Benchmark, using DeepSeek V4 Flash and the same ordering parameters. Accuracy changes are reported in percentage points (pp).

MetricBase+EviCompileChange
Strict accuracy23.53%31.57%+8.04 pp
Partial accuracy36.63%52.71%+16.08 pp
Mean steps11.009.04-17.8%
Input tokens/query769,812473,711-38.5%
Output tokens/query85,97279,940-7.0%
Total tokens/query855,784553,651-35.3%
Model calls/query38.231.9-16.5%
Mean runtime907.1s847.6s-6.6%
Diagnoses exceeding 10 minutes4636-10
Sandbox aborts65-1

These results extend the FSH findings to an independent RCA benchmark. With the same ordering parameters, EviCompile improved both strict and partial accuracy while reducing token use, model calls, and runtime.

Thank you for your comments. We hope the parameter rationale and the OpenRCA results address your questions about reuse and practical diagnostic value.

Reviewer_GmHpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Localization says where to look; diagnosis must explain the mechanism, causal direction, and repair owner. Location-match evaluation cannot tell those apart. Chronological tool histories also fragment and repeat evidence.

Critical strengths
  1. Evaluation goes beyond localization. Distinguishing correct location from correct mechanism and repair responsibility captures failures that conventional location matching misses. This could materially improve how debugging agents are evaluated.

  2. The fixed-evidence comparison isolates a meaningful effect. Holding observations constant shows that improvement can arise from representation rather than collecting more evidence. Blinded expert reassessment provides additional support for the direction of the gains.

Critical weaknesses
  1. The distinctive ordering contribution has weaker evidence than the overall compiler. The main comparison bundles typing, consolidation, ordering, and memory reconstruction. Table 4 isolates ordering, but reports only point estimates for its smaller gains.

  2. The supplied PDF references Appendices A–J that are unavailable in the folder. This limits my assessment; it should not automatically count against the submission if those materials were supplied separately to ICLR.

Questions for authors

Are the referenced supplements / appendix available, and are the ordering-only improvements statistically reliable?

1 comment

Dear Reviewer,

Thank you for your careful review and for recognizing the value of the FSH diagnostic evaluation framework and the fixed-evidence experiment. We particularly appreciate your question about how much of the observed improvement can be attributed to ordering. Because EviCompile combines several processing steps, an evaluation of the full system alone cannot isolate that contribution. We therefore extended the analysis of the original 60 cases with paired comparisons, confidence intervals, corrections for multiple testing, and a breakdown across all six diagnostic dimensions.

1. Independent contribution of ordering and statistical reliability

To isolate ordering, we held the evidence set, type assignments, merging of evidence across observations, and retention of source information fixed. Each strategy received the same evidence for a given case. Only the order in which that evidence was presented to the diagnostic model changed.

  • Chronological orders evidence by the time of its first appearance.
  • Simple Relevance orders evidence solely by its relevance to the diagnostic task.
  • EviCompile (Full Ordering) ranks evidence by cost-normalized marginal utility. It considers relevance, the complementary coverage an item adds to evidence already selected, and representativeness, while accounting for the item’s cost in characters.

The table reports within-case differences. A win means that the first strategy in a row achieved a higher diagnostic score than the second on the same case.

ComparisonMean difference95% CIWins/ties/lossesWilcoxon pHolm-adjusted p
EviCompile − Chronological+2.78[1.20, 4.36]38/14/80.00080.0024
EviCompile − Simple Relevance+2.08[0.65, 3.55]34/15/110.00600.0120
Simple Relevance − Chronological+0.69[-0.30, 1.70]26/20/140.18000.1800

Full Ordering significantly outperformed both alternatives after Holm correction. Simple Relevance showed a positive mean difference over Chronological, but its confidence interval included zero. These cases therefore provide insufficient evidence that moving the most textually relevant items earlier reliably improves diagnosis. The comparison with Simple Relevance isolates the benefit of the full ordering rule on identical evidence. It measures the combined contribution of complementary coverage, representativeness, and presentation cost beyond relevance alone.

To identify which aspects of diagnosis benefited most, we examined paired differences on each of the six dimensions.

DimensionMean difference95% CIWins/ties/lossesHolm-adjusted p
Failure mechanism+0.0444[0.0000, 0.0889]14/40/61.0000
Causal direction+0.0667[0.0167, 0.1167]18/36/60.2945
Component responsible for repair+0.1333[0.0833, 0.1833]28/28/40.000290
Repair action direction+0.1667[0.1167, 0.2167]34/22/40.0000109
Completeness+0.0444[-0.0056, 0.0944]14/40/61.0000
Evidential support+0.0444[-0.0056, 0.0944]14/40/61.0000

The largest gains were in identifying the component responsible for repair and determining the direction of the repair action. Both confidence intervals lay above zero, and both gains remained significant after Holm correction across 18 dimension-level comparisons. Causal direction also had a confidence interval above zero, although its gain did not remain significant after correction. This pattern aligns with EviCompile’s design goal. Once a failure manifests, a diagnosis must draw on complementary evidence to determine which component should be changed and what kind of change would address the failure.

Statistical methods. We used each case as a paired unit. Using random seed 8501, we drew 10,000 bootstrap samples with replacement, keeping each case’s scores under all three conditions together. We constructed each 95% CI from the 2.5 and 97.5 percentiles using linear interpolation. For the Wilcoxon signed-rank tests, we used two-sided exact sign permutations, excluded zero differences, and assigned average ranks to tied absolute differences. We applied Holm correction separately to the three overall comparisons and the 18 dimension-level tests.

2. Access to Appendices A–J

Thank you for drawing our attention to the difficulty of locating the appendices. We submitted Appendices A–J with the original manuscript as Supplementary Material. We apologize for the inconvenience of having to access the paper and its appendices separately. Download Appendices A–J from OpenReview.

Thank you for prompting us to clarify where ordering improves diagnosis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.