- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces FailureSnapshotHub, a 60-case interactive software-failure diagnosis benchmark spanning 49 repositories, with expert-adjudicated references and six-dimensional semantic evaluation. It also proposes EviCompile, a deterministic pipeline that compresses recoverable text, constructs typed source-linked evidence, consolidates compatible observations, and orders the complete evidence memory using relevance, coverage, representativeness, and character cost. Experiments report improved diagnosis scores in two interactive agent settings and a fixed-evidence setting, with positive mean gains retained in a selected blinded human reassessment.
Critical strengths
-
The benchmark distinguishes failure mechanism, causal direction, repair responsibility, and evidence grounding rather than treating localization as sufficient diagnosis. Expert reference construction and failed-side access restrictions are useful aspects of the evaluation design.
-
The fixed-evidence comparison helps separate representation effects from differences in evidence acquisition. The reported diagnosis-score improvements and blinded human reassessment provide evidence that the complete pipeline is useful within the evaluated setting.
Critical weaknesses
-
The paper does not sufficiently distill the practical need for evidence organization into a distinctive technical problem or methodological contribution. EviCompile largely combines recoverable compression, typed records, deduplication, provenance retention, and coverage-aware ordering. Exceptions, source locations, test outcomes, and path inventories adapt this general approach to software data, but the paper does not clearly establish what technical obstacle these domain characteristics introduce that existing evidence-organization methods cannot address, or what new mechanism resolves it. Applying a general idea to software diagnosis can be valuable, but the current formulation and evidence do not sufficiently establish that independent technical contribution.
-
The comparisons and evaluation scope do not adequately resolve this novelty concern. Raw histories, local text coding, and internal ordering variants show the utility of the combined pipeline, but do not establish an advantage over strong conventional structured, provenance-preserving, deduplicated evidence memories with comparable ordering. Typing, sharing, and reconstruction are not fully separated, and the ordering ablation provides a relatively small part of the total gain without reported uncertainty in that table. The study also uses one diagnostic model on 60 selected cases. Human reassessment supports diagnosis quality on its selected subset, but the broader transfer and practical consequences of the improvement remain uncertain. Actual repair success is not evaluated; this limits the demonstrated downstream utility rather than constituting a requirement that every diagnosis paper perform repair.
Questions for authors
-
What precisely is the technical obstacle, beyond heterogeneous formatting and redundant observations, that distinguishes this problem from general evidence organization? Which component introduces a new solution to that obstacle rather than a software-specific data-schema adaptation?
-
How does EviCompile compare with a strong structured and deduplicated evidence-memory baseline using the same source attribution and comparable relevance/diversity ordering? Component-level comparisons with paired uncertainty would help establish what independently contributes beyond conventional organization.
2 comments
Dear Reviewer,
Response 2 of 2. Benchmark contribution, multi-model results, and scope of the evaluation
Thank you for recognizing FSH’s six-dimensional evaluation and the work involved in constructing expert reference diagnoses. We appreciate your questions about how broadly the findings apply and what they establish about repair.
1. What does FSH evaluate, and how were its cases selected?
FSH contains 60 real CI failures from 49 repositories. Through expert adjudication and access restricted to failure-side evidence, FSH evaluates six dimensions: failure mechanism, causal direction, repair responsibility, repair action, completeness, and evidential support. These dimensions distinguish diagnostic errors that traditional localization metrics can miss. For example, a diagnosis may identify where a failure surfaced yet give the wrong repair direction.
Experts selected cases for which the failure-side evidence was sufficient to infer the mechanism and determine where the fix belonged without inspecting the fix diff. They excluded cases with multiple independently viable repair locations. This selection makes it possible to assess a diagnosis using the evidence available to the agent. FSH was not designed as a representative sample of all CI failures.
2. How consistent are the gains across models and workflows?
To address the limitation of evaluating a single diagnostic model, we tested three models in three settings. Each entry is EviCompile’s mean improvement over Raw.
| Model | Tool Agent | OpenRCA-style | Fixed-evidence |
|---|---|---|---|
| Qwen 3.5 Flash | +2.78 | +4.03 | +1.81 |
| DeepSeek V4 Flash | +9.72 | +10.42 | +5.55 |
| Claude Opus 4.8 | +14.24 | +18.75 | +10.00 |
All nine combinations showed positive gains. In table order, the models’ mean Raw scores across settings were 53.66, 60.19, and 79.17, with corresponding mean gains of +2.87, +8.56, and +14.33. The models with stronger Raw diagnostic performance also gained more from EviCompile, suggesting that evidence organization and model reasoning ability may be complementary.
For all three models, the gains followed the same order: OpenRCA-style > Tool Agent > Fixed-evidence. This pattern suggests that the benefit of evidence organization may also depend on the agent workflow.
3. Do the gains extend to an independent RCA benchmark?
We evaluated EviCompile on the original OpenRCA Benchmark using DeepSeek V4. Strict accuracy rose from 23.53% to 31.57%, partial accuracy rose from 36.63% to 52.71%, and total token consumption fell by 35.3%. Accuracy changes in the table are expressed in percentage points (pp).
| Metric | Base | +EviCompile | Change |
|---|---|---|---|
| Strict accuracy | 23.53% | 31.57% | +8.04 pp |
| Partial accuracy | 36.63% | 52.71% | +16.08 pp |
| Mean steps | 11.00 | 9.04 | -17.8% |
| Input tokens/query | 769,812 | 473,711 | -38.5% |
| Output tokens/query | 85,972 | 79,940 | -7.0% |
| Total tokens/query | 855,784 | 553,651 | -35.3% |
| Model calls/query | 38.2 | 31.9 | -16.5% |
| Mean runtime | 907.1s | 847.6s | -6.6% |
| Diagnoses exceeding 10 minutes | 46 | 36 | -10 |
| Sandbox aborts | 6 | 5 | -1 |
These results extend the findings beyond FSH’s semantic scores. On an independent benchmark, EviCompile improved both strict and partial diagnostic accuracy while reducing token use, model calls, and mean runtime.
4. What do these results establish about repair success?
Our evaluation concerns diagnosis before patch execution, specifically whether the diagnosis identifies where the fix belongs and what change would address the failure. End-to-end repair success also depends on the ability to generate a working patch and requires a separate evaluation.
Thank you again for your thoughtful review. Your comments helped us clarify why FSH’s cases were selected, how the findings extend across models and benchmarks, and where the evidence about diagnosis ends.
Dear Reviewer,
Response 1 of 2. Methodological contribution, strong baselines, and experimental attribution
Thank you for your detailed review and for recognizing the value of the fixed-evidence experiment. Your questions prompted us to examine what EviCompile adds to conventional evidence organization and which gains can be attributed specifically to ordering.
1. What technical problem does EviCompile address?
Evidence that determines where a fix belongs may have little textual similarity to the visible failure symptom. An exception and stack trace can show where a failure surfaced while leaving open whether to change the test, the caller, the callee, or the configuration. The evidence that resolves this question may come from the implementation on the other side of an interface or from an upstream producer.
EviCompile incorporates diagnostic roles and candidate repair responsibilities into its evidence coverage objective. Its ordering rule considers relevance, complementary coverage, representativeness, and character cost. An item can receive priority because it helps distinguish between candidate repair locations, even when its wording differs from the failure description. Evidence retention is independent of presentation order, merged records preserve source attribution, and the diagnostic agent makes the final causal judgment.
2. How does EviCompile compare with conventional evidence organization?
To address your request for a structured evidence memory with deduplication and provenance, we implemented a Conventional Evidence Memory baseline. In the fixed-evidence setting, it stores each tool response as a record containing the tool name, source path, line numbers, content, and source list. It merges duplicates using TF-IDF cosine similarity and retains all sources. It scores relevance to the initial failure description using BM25, then uses MMR to balance relevance and textual diversity, with an optional length cost. It uses none of EviCompile’s evidence types, role hints, or specialized priors.
The table lists the configurations for which the paired comparisons showed a statistically supported gain. Differences are EviCompile minus baseline. Wins, ties, and losses are counted from EviCompile’s perspective. θ is the TF-IDF deduplication threshold, and α controls the balance between relevance and textual diversity in MMR.
We also compared EviCompile with MMR, evidence truncation, and LLMLingua-2 using the same paired comparison.
3. Which gains can be attributed to Ordering?
The baseline comparisons evaluate the method as a whole. To isolate Ordering, we held the typed and shared records fixed, so every strategy presented the same evidence. Full Ordering improved the mean score by +2.78 over chronological ordering (95% CI [1.20,4.36], Holm-adjusted p=0.0024) and by +2.08 over simple relevance ordering (95% CI [0.65,3.55], Holm-adjusted p=0.012).
In the six-dimensional analysis, scores for identifying where the fix belongs and determining the direction of the repair action improved by +0.1333 and +0.1667, respectively. Both gains remained significant after Holm correction across 18 comparisons. Typing cannot be removed in isolation because both Sharing and Coverage depend on typed records.
Thank you again for asking us to examine attribution more closely. Your suggestion led us to test a baseline that preserves provenance and to quantify what changes when only the presentation order varies.